Senior Systems Engineer, Record & Replay

About the role

Watcher records every rank of a training job at under 5% overhead and replays the failing window exactly. You will own the engine that makes that possible, from interception at the library boundary to merging verified state back into a live run. The research behind it (iReplayer, Dthreads) is published; the production version is yours to build.

What you will do

• Extend API interception to new system calls, synchronization primitives, PyTorch operators, and CUDA runtime calls
• Enforce deterministic replay across threads, processes, and ranks, including collective operations
• Design the verified-state protocol that lets a run resume from the failing step without a checkpoint rollback
• Keep recording overhead under 5% on real pretraining workloads and measure it continuously
• Debug failures on customer clusters alongside the founder

What we are looking for

• 5+ years building systems software in C++ on Linux
• Hands-on experience with dynamic analysis, record-and-replay, debuggers, or runtime instrumentation
• Comfortable reading kernel, libc, and CUDA runtime internals
• Strong grasp of concurrency: memory models, synchronization, nondeterminism

Nice to have

• Published systems research (SOSP, OSDI, PLDI, ASPLOS, OOPSLA)
• Experience with NCCL or other collective communication libraries
• Prior work on checkpointing or fault tolerance for distributed training