Watcher · during the run
Recover without restart, or get the root cause
Watcher intercepts system calls, synchronizations, PyTorch, and CUDA APIs to keep a deterministic record of every rank. When the run fails, it rewinds and replays the failing window. Not reproduced: merge verified state and continue. Reproduced: attribute it and prescribe a fix.
Covers: crash, hang, silent data corruption, loss spike
Status: Deterministic-replay root-cause analysis runs on CPU workloads today. The CUDA/NCCL extension and automatic recovery are in development.
Deterministic Replay
Uniform recording, two tracks
Watcher records every rank at under 5% overhead with no code changes. When something fails, it rewinds and replays the failing window. Whether the failure comes back decides what happens next.
A bit flip, a network timeout, a race. Merge the verified state and continue from the failing step. No checkpoint rollback.
A logic bug, memory corruption, bad data. Attribute it to software, data, or hardware. Emit the causal chain and a fix.
Track A — Recovery
The error is not reproduced, so it was transient: a bit flip, a network timeout, a race. Merge the verified state and continue from the failing step. No checkpoint rollback.
Track B — Diagnosis
The error is reproduced, so it is persistent: a logic bug, bad data, a failing device. Attribute it to software, data, or hardware, emit the causal chain, and prescribe a fix.
Words we use
- Deterministic replay
- Re-executing a recorded window so it behaves exactly as the original run did, step for step, across every rank.
- Transient vs. persistent
- A transient fault does not come back when you replay (bit flip, network jitter). A persistent bug does (logic error, bad batch, failing GPU).
- Verified state
- A point in the run that replay has confirmed consistent across all ranks. Recovery resumes from here, not from the last checkpoint.
- Preflight
- A few minutes of profiling before launch that catches OOM-bound setups, wrong parallelism, and precision or topology mismatches.
What Watcher does
Extensible API interception
System calls, synchronization primitives, PyTorch operators, and CUDA APIs are intercepted at the library boundary. No recompilation, no custom hypervisor, no deployment changes.
Always-on selective recording
Captures what replay needs and nothing else. Under 5% overhead today; under 1% projected with the GPU extension.
Enforced replay
Triggered on failure or detection. Identical re-execution of the failing window, built on a decade of record-and-replay research (iReplayer, Dthreads).
Track A: Recovery
Error not reproduced means it was transient. Merge the verified state and resume at the failing step. No checkpoint rollback.
Track B: Diagnosis
Error reproduced means it is persistent. Attribute it to software, data, or hardware; emit the full causal chain; prescribe fix kernel, skip batch, or adjust params.
Device health
Suspicious devices are quarantined past a health threshold, before they take the run down.
Watcher · Try it
Download the trial
Leave a work email and the download starts. The package is a placeholder while the trial build is being prepared; we will follow up with setup notes.
Getting started
Start with one workload
Built for teams that run pretraining and fine-tuning. Validate compatibility and overhead on a single workload before any production rollout.