Software Engineer - GPU Networking & Distributed Systems

Baseten powers mission-critical inference for AI companies like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. They unite applied AI research, flexible infrastructure, and developer tooling to help companies bring cutting-edge models into production. They recently raised a $1.5B Series F led by Altimeter Capital, Conviction Partners, and Spark Capital.

As a foundational engineer on the GPU Networking team, you will architect the software fabric that unifies thousands of GPUs into a cohesive operating system. You will integrate RDMA/RoCE/InfiniBand capabilities into the inference stack, optimize distributed inference for Disaggregated KV Cache Offload and Wide Expert Parallelism (WideEP), enable sub-10-second startup for trillion-parameter models, characterize networking performance on bleeding-edge clusters (H100/H200, B200/B300, GB200/300 NVL72), build observability tools for packet flow and congestion, and optimize communication kernels with NCCL and NVSHMEM.

• Deep experience with high-performance networking protocols (InfiniBand, RoCE v2)
• Fluent in C++ or Python
• Deep understanding of memory hierarchy in modern NVIDIA architectures (H100/Blackwell)
• Comfort diving into TensorRT-LLM source code, writing custom C++/Python bindings, or debugging NVLink topology issues
• Knowledge of when to use off-the-shelf solutions vs. custom builds

• Deep knowledge of NCCL, NVSHMEM, and UCX
• Experience with Rust for systems-level networking code
• Experience with GPUDirect Storage (GDS) or high-performance filesystems like Weka or 3FS
• Familiarity with TensorRT-LLM, vLLM, or SGLang
• Experience running low-level benchmarks to qualify new hardware clusters

Salary range: $165K – $330K, plus equity (U.S. only). Benefits include 100% coverage of medical, dental, and vision insurance for employee and dependents, flexible PTO with company-wide Winter Break, paid parental leave, fertility and family-building stipend through Carrot (U.S. only), and company-facilitated 401(k).