AI Research Engineer - Datadog AI Research (DAIR)

• Build multimodal data pipelines, benchmarks, and tooling for research experiments.
• Implement models and evaluate their reliability and performance at scale.
• Create simulation environments and replay infrastructure for agent training.
• Orchestrate distributed training and reinforcement learning with Ray, covering scheduling, scaling, and failure recovery.
• Develop automated regression tests for model predictions, agent performance, and simulation fidelity.
• Work with Product and Engineering to integrate prototypes into production services.
• Contribute research publications, documentation, code, and open-source artifacts.

• Depth in distributed computing, reinforcement learning infrastructure, and ML training and inference systems.
• Python proficiency, familiarity with Rust, C++, or Go, and experience with cloud and data infrastructure.
• Practical ML systems experience covering containerization, orchestration, and GPU acceleration.
• Experience with large-scale model training, fine-tuning, post-training techniques, and efficient inference.
• Ability to explain design and performance trade-offs to technical and non-technical audiences.
• Experience supporting or contributing to research publications.

Nice to have

• Experience with Ray, Slurm, or similar distributed computing frameworks.
• Software engineering experience in observability, SRE, or security.
• Experience bringing foundation models, world models, or trained agents into products.
• GPU programming with CUDA, production data pipelines, or agent simulation environments.