About the Role
The Artificial General Intelligence (AGI) team is seeking a dedicated, skilled, and innovative Applied Scientist with a robust background in machine learning, statistics, quality assurance, auditing methodologies, and automated evaluation systems. This role focuses on ensuring the highest standards of data quality to build industry-leading technology with Large Language Models (LLMs) and multimodal systems.
Key Job Responsibilities
- Collaborate closely with the core scientist team developing Amazon Nova models.
- Lead the development of comprehensive quality strategies and auditing frameworks to safeguard the integrity of data collection workflows.
- Design auditing strategies with detailed SOPs, quality metrics, and sampling methodologies to help Nova improve performance on benchmarks.
- Perform expert-level manual audits and conduct meta-audits to evaluate auditor performance.
- Provide targeted coaching to uplift overall quality capabilities.
- Develop and maintain LLM-as-a-Judge systems, including designing judge architectures, creating evaluation rubrics, and building machine learning models for automated quality assessment.
- Set up the configuration of data collection workflows and communicate quality feedback to stakeholders.
- Enhance customer experiences through high-quality training and evaluation data that powers state-of-the-art LLM products and services.
A Day in the Life
An Applied Scientist with the AGI team will support quality solution design, conduct root cause analysis on data quality issues, research new auditing methodologies, and find innovative ways of optimizing data quality. This role involves setting examples for the team on quality assurance best practices and standards. Besides theoretical analysis and quality framework development, an Applied Scientist will also work closely with talented engineers, domain experts, and vendor teams to put quality strategies and automated judging systems into practice.
Basic Qualifications
- Master's degree in computer science, mathematics, statistics, machine learning or equivalent quantitative field.
- Experience programming in Java, C++, Python or related language.
- Experience with SQL and an RDBMS (e.g., Oracle) or Data Warehouse.
Preferred Qualifications
- Experience implementing algorithms using both toolkits and self-developed code.
- Have publications at top-tier peer-reviewed conferences or journals.