Senior Software Engineer

Sentry is an application monitoring platform trusted by 200,000+ organizations, helping developers fix errors and performance issues before users notice. The team is building its AI-native future.

As a Senior Software Engineer on Sentry's AI/ML team, you will build evaluation infrastructure that measures the accuracy, reliability, and real-world performance of AI systems. You will design datasets, benchmarks, and test harnesses to turn ambiguous AI behavior into measurable signals.

• Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
• Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
• Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
• Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
• Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

• Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
• Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
• Comfort writing production-quality code (we use Python and TypeScript)
• Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
• Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
• Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

Base salary range: $155,000 to $400,000 USD. Eligible for incentive compensation, equity grants, paid time off, and group health insurance coverage. See Sentry Benefits for details.

Sentry is committed to equal employment opportunity and provides reasonable accommodations for disabilities. Contact accommodations@sentry.io for assistance.