SAN FRANCISCO, October 16 — Databricks’ AI Research team shared how they use the company’s own AI Runtime to power their model development work, revealing an engineering approach that keeps hundreds of parallel research streams running smoothly. The team develops models that power Databricks AI Functions such as ai_parse_document and ai_extract, which deliver state-of-the-art document parsing quality.
Their research spans Genie, Smart Routing, and AI Functions, and includes recent advances like making search three times faster, developing new techniques for offline reinforcement learning, and pushing the frontier of complex document extraction. To manage hundreds of parallel research workstreams involving models ranging from a few billion to nearly a trillion parameters, the team relies on AI Runtime for its GPU infrastructure and integration with existing tools.
They use supervised fine-tuning, offline reinforcement learning, and online reinforcement learning to train models, depending on workload requirements, utilizing NVIDIA H100 or B300 GPUs available through AI Runtime. Largest training runs require hundreds of GPUs working together over days or weeks with high-bandwidth networking.
The team identified three challenges in supporting these workloads, which AI Runtime helps address through built-in GPU resiliency: AI Runtime performs preflight checks on GPU nodes to ensure health and communication before training, monitors node health during execution for issues like NCCL communication failures, thermal or power throttling, and GPU driver problems, and if a node becomes unhealthy and causes a job to fail, AI Runtime detects the issue and cordons off the affected node so subsequent jobs are not scheduled on the same problematic infrastructure.
The research team has been customer zero of AI Runtime since the product’s inception the previous year. This early adoption has allowed them to shape the platform’s capabilities based on real-world demands from large-scale AI research.
Sources
Background and why it matters
Databricks was founded in 2013 by the original creators of Apache Spark, aiming to simplify big data processing and machine learning workflows. Over the years, the company expanded its platform to include collaborative notebooks, data engineering tools, and machine learning lifecycle management. As demand grew for deploying large-scale AI models in enterprise settings, Databricks began integrating deeper AI capabilities into its ecosystem, leading to the formation of its dedicated AI Research team to push the boundaries of model performance and efficiency.
The AI Research team focuses on advancing core technologies that power Databricks’ AI Functions, such as natural language processing and intelligent routing systems. Their work involves developing and refining models that handle complex tasks like document understanding, semantic search, and decision-making under uncertainty. These efforts are grounded in cutting-edge techniques including supervised fine-tuning and various forms of reinforcement learning, which require substantial computational resources and stable infrastructure to iterate at scale.
To support this research, Databricks introduced AI Runtime (AIR) as a specialized layer of its platform designed to optimize GPU-based AI workloads. AIR provides automated health checks, fault tolerance, and seamless integration with tools like MLflow and Apache Spark, allowing researchers to focus on model innovation rather than infrastructure management. By serving as the first internal user—or “customer zero”—of AIR, the research team has directly influenced its development through real-world feedback from large-scale training experiments.
This approach matters because it illustrates how AI research and platform engineering can evolve in tandem, with internal teams driving both innovation and product improvement. For readers, it highlights the growing importance of resilient, integrated systems in enabling reliable AI development—not just breakthrough algorithms, but also the underlying mechanisms that make sustained progress possible at scale. Understanding this context helps clarify how companies like Databricks bridge the gap between research experimentation and production-ready AI services.
