Software Engineer - Model Products

Baseten powers mission-critical inference for AI companies like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. They unite applied AI research, flexible infrastructure, and developer tooling to help companies bring cutting-edge models into production. They recently raised a $1.5B Series F led by Altimeter Capital, Conviction Partners, and Spark Capital.

As part of the Model Performance (MP) team, you'll focus on Model APIs—the infrastructure powering hosted API endpoints for the latest open-source models. This work spans distributed systems, model serving, and developer experience. You'll join a small, high-impact team at the intersection of product, model performance, and infra.

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling, and multi-modal serving.
• Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns, and optimize communication patterns across multi-GPU setups.
• Productionize performance improvements across runtimes (e.g., TensorRT, TensorRT-LLM): speculative decoding, quantization, batching, and KV-cache reuse.
• Build comprehensive benchmarking frameworks measuring real-world performance across model architectures, batch sizes, sequence lengths, and hardware configurations.
• Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks for speed, reliability, and quality.
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication.
• Collaborate with other teams to deliver robust, developer-friendly model serving experiences.

• 3+ years experience building and operating distributed systems or large-scale APIs.
• Prior ML or LLM experience not required; strong systems engineers with performance instincts are sought.
• Proven track record of owning low-latency, reliable backend services (rate-limiting, auth, quotas, metering, migrations).
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management.
• Comfortable debugging performance and reliability issues across complex systems.
• Strong written communication; able to produce clear design docs and collaborate across functions.

• Experience with LLM runtimes (vLLM, SGLang, TensorRT-LLM) or contributions to open-source inference engines.
• Knowledge of Kubernetes, service meshes, API gateways, or distributed scheduling.
• Background in developer-facing infrastructure or open-source APIs.

• Competitive compensation, including meaningful equity (U.S. only)
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company-wide Winter Break (offices closed from Christmas Eve to New Year's Day)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering learning and networking opportunities