Staff Research Engineer, Post-training & Evaluation

Reddit is a community of communities, built on shared interests, passion, and trust. With 100,000+ active communities and approximately 130 million daily active unique visitors, Reddit is one of the internet’s largest sources of information. The AI Engineering team at Reddit is building Reddit-native foundational Large Language Models (LLMs). This team sits at the intersection of applied research and massive-scale infrastructure, training models that understand the unique culture, language, and structure of Reddit communities. As a Staff Research Engineer for Post-Training & Evaluation Science, you will own the science of the model development feedback loop, defining how we measure whether models are safe, smart, and Reddit-native, and setting the post-training methodology that turns base checkpoints into high-performing endpoints.

• Define the Reddit Benchmark evaluation standard, owning the methodology for rigorously measuring model quality across Safety, Reasoning, representation/retrieval, and Reddit-specific knowledge.
• Own evaluation reliability and statistical rigor, including judge variance, multi-sample scoring, inter-rater/inter-sample agreement, sampling and temperature effects, and calibration of automated judges.
• Drive evaluation as a release gate, offline against frozen datasets and pre-merge in CI/CD.
• Design model-as-a-judge methodology, including judge selection, prompt design, calibration, and reliability for automated evaluation using frontier external models.
• Set post-training recipes and strategy, designing SFT recipes (data mixtures, curriculum, ablation strategy) and partnering with engineering to scale them.
• Evaluate base and CPT checkpoints, designing checkpoint-selection methodology across CPT experiments and LR studies.
• Drive synthetic data generation strategy, defining and curating high-quality instruction and evaluation sets.
• Partner with Safety Engineering to translate high-level safety policy into concrete classification metrics, probe sets, and CI/CD unit tests.
• Diagnose post-training instability, diving into loss curves and eval logs to identify alignment tax and capability degradation.
• Lead research direction, setting technical direction for evaluation and post-training across the team, mentoring engineers and scientists.

• 6+ years of professional ML experience (or PhD + 4+) with a direct focus on LLM post-training and evaluation.
• PhD or MS in CS, ML, NLP, IR, or a related quantitative field — or equivalent industry research experience.
• Deep expertise in evaluation reliability: judge/sample variance, multi-sample scoring, calibration, statistical significance, and failure modes of automated evaluation.
• Strong experience building custom, domain-specific evaluation harnesses (e.g., lm-eval-harness, Inspect AI, LightEval), understanding strengths and limits of benchmarks like MMLU and GSM8K.
• Experience evaluating both generation and representation/classification: model-as-a-judge for generative quality and precision/recall, PR-AUC, retrieval/MTEB-style metrics, gold-label denoising, and label-noise handling.
• Deep understanding of Continuous Pre-training (CPT), Instruction Tuning (SFT), and how data quality shapes model behavior.
• Fluency in Python; strong data-pipeline and eval-harness engineering (e.g., Hugging Face Transformers, vLLM, lm-eval-harness).
• Working knowledge of PyTorch and distributed training (FSDP2, DeepSpeed ZeRO-3) sufficient to direct and debug post-training runs.

• Experience with MLflow or similar experiment-tracking frameworks.
• Familiarity with modern fine-tuning frameworks (Axolotl, TorchTune) and PyTorch-native training stacks (TorchTitan).
• Synthetic data generation techniques (e.g., Self-Instruct).
• Experience with preference optimization (DPO, RLHF, RLAIF, GRPO).
• Publications in NLP/ML/FAccT or related venues, or other evidence of research leadership.
• Experience evaluating multimodal models (embeddings, hateful-memes-style classification).

The base salary range for this position is $230,000—$322,000 USD. In addition to base salary, this job is eligible to receive equity in the form of restricted stock units, and depending on the position offered, it may also be eligible to receive a commission. Reddit offers a wide range of benefits to U.S.-based employees, including medical, dental, and vision insurance, 401(k) program with employer match, generous time off for vacation, and parental leave. Additional benefits include comprehensive healthcare benefits and income replacement programs, 401k with employer match, global benefit programs, family planning support, gender-affirming care, mental health & coaching benefits, flexible vacation & paid volunteer time off, and generous paid parental leave.

To apply, please use the provided application link.