Key Responsibilities
- Design and evaluate autonomous AI agents across multiple LLMs for real-world domains like health, education, and daily life
- Develop evaluation rubrics with objective pass/fail criteria for agent performance
- Debug agent traces to identify failure patterns and stress test against edge cases, prompt injection, and tool misuse
- Assess production-grade modular software architecture and analyze multi-turn system interactions
- Provide high-density technical feedback for LLM training and complex architectural workflows
Requirements
- Backend engineering experience or expertise in AI automation and complex systems integration
- Proficiency in building and maintaining production-grade software with modular separation (e.g., distinct services for data parsing, logic processing, and reporting)
- Strong command of Python and at least one other major language (e.g., JavaScript, Go, or Java) with experience in SQL databases
- Experience working with live, non-mocked environments and handling multi-turn system interactions
- Familiarity with persistent state, session-tracking patterns, and security vulnerabilities like privacy leaks or prompt injection