Senior Staff Applied AI Engineer

About the Role

Databricks is seeking a Senior Staff Applied AI Engineer to lead context retrieval for Databricks agents across SaaS providers. This is a foundational, zero-to-one role with a dual focus: building a robust retrieval stack and developing intelligent search subagents. The successful candidate will have deep expertise in Information Retrieval (IR) and experience in shipping retrieval systems for RAG and agentic workloads.

What You Will Do

• Build the full retrieval stack from scratch: Own the end-to-end system including query understanding, content understanding and indexing, hybrid retrieval, ranking, and evaluation, making critical architectural decisions.
• Retrieve across heterogeneous data: Index and rank both structured assets (tables, columns, SQL queries, dashboards, code, notebooks, jobs) and unstructured content (docs, wikis, tickets, chat, images, video, audio), leveraging unique signals from each modality.
• Connect to the SaaS surface area customers actually use: Develop connectors and retrieval adapters for enterprise knowledge systems, managing freshness, permissions, and ranking for each source.
• Optimize for two consumers at once: Design retrieval systems that effectively serve both LLMs (grounded, token-efficient, hallucination-resistant context) and humans (intuitive, explainable discovery).
• Crack query understanding for agents: Implement query rewriting, decomposition, intent classification, and entity resolution specifically tuned for multi-turn agentic workflows.
• Crack content understanding at scale: Build pipelines for extracting structure, entities, embeddings, summaries, and metadata from diverse asset types, ensuring data freshness.
• Build search subagents that reason about retrieval: Design an agentic layer to determine needed context, identify sources to query, decompose and route searches, and critically, verify retrieval sufficiency. These subagents will plan multi-hop searches, issue follow-up queries, ground claims, and manage context delivery.
• Build the evaluation flywheel for both retrieval and subagents: Establish offline evals (nDCG, MRR, Recall@K, Precision@K), LLM-as-judge harnesses, human-in-the-loop labeling, and online experimentation. Extend evaluation to measure subagent decision quality.
• Set technical direction and grow the team: Define the multi-year roadmap, mentor senior engineers, and collaborate with Research, Product, and Platform leaders to elevate technical standards.

What We're Looking For

• 10+ years of software engineering experience, with significant time spent building production retrieval, search, or RAG systems at scale.
• Deep Information Retrieval (IR) expertise: proficiency in lexical retrieval (BM25, Lucene/Elasticsearch/OpenSearch), dense retrieval (embeddings, ANN indexes — FAISS, ScaNN, HNSW), hybrid retrieval, and learning-to-rank.
• Hands-on experience with modern LLM-era retrieval: RAG architectures, query rewriting, re-ranking with cross-encoders, long-context strategies, and grounding techniques that reduce hallucination.
• Experience designing agentic systems on top of retrieval — search planners, multi-hop / iterative retrieval, self-reflection and sufficiency checks, tool-using agents that decide what to fetch and verify what came back.
• Strong grasp of relevance evaluation: nDCG, MRR, Precision@K, Recall@K; offline/online experimentation; LLM-as-judge frameworks; building human labeling pipelines.
• Experience working across structured and unstructured data — you've indexed and ranked over tables, code, and documents in the same system, and have opinions about how to do it well.
• Track record of building 0→1: you've stood up a retrieval system from an empty repo, made the foundational architectural decisions, and grown it into something that customers depend on.
• Demonstrated ability to operate as a technical leader: setting direction across teams, mentoring senior engineers, and influencing roadmap with research, product, and platform partners.

Nice to Have

• Experience building retrieval over enterprise SaaS sources (permissions, freshness, multi-tenancy, ACL-aware indexing).
• Background in agentic systems, tool use, or multi-turn retrieval for LLM agents.
• Contributions to open-source IR/search projects, or publications at SIGIR, KDD, WWW, EMNLP, or similar venues.
• Experience training or fine-tuning embedding models, rerankers, or query understanding models.

Why This Role

• Foundational impact: The retrieval stack built will be a critical component for every Databricks agent and customer-built agent.
• Greenfield with scale: This role offers the unique opportunity to build from scratch with immediate access to massive enterprise scale, real customer data, and a world-class research organization.
• The right team: Work alongside engineers and researchers behind industry-leading technologies like Lakehouse, Apache Spark™, Delta Lake, MLflow, MosaicML, and DBRX.