What should you capture first?
Save the error, the failing step, and the configuration needed to reproduce it. Record hardware capacity, framework versions, batch size, input shapes, precision, and distributed settings. Keep this evidence together so a later comparison refers to the same workload.
Separate observations from hypotheses. A high peak, growth across repeated steps, and a failure on one input are different patterns to investigate. Avoid treating every OOM as the same problem.
What can PyTorch memory snapshots tell you?
PyTorch’s memory visualizer shows active tensors over time and allocation stack traces. Its allocator history includes allocation and free events, memory segments, and OOM events. Inspecting the state around a failed allocation can help explain why it could not be satisfied.
The tool’s coverage matters: allocations made outside the PyTorch allocator may not appear in a PyTorch snapshot. Compare the allocator view with overall device usage when the two disagree. Consult the official documentation for the APIs available in your installed version.
How do you test an explanation?
- Choose a representative reproduction and save a baseline.
- Change one relevant variable at a time and record the exact change.
- Compare memory peaks, behavior across steps, and throughput—not only whether the job completes.
- Check the result on representative inputs before applying it to a longer run.
A successful small run is useful evidence for that configuration. It is not a guarantee for different inputs or a larger deployment.
Where does MemSage fit?
MemSage helps AI teams connect CPU and GPU memory growth and peaks to relevant allocations, tensors, and call stacks. Its three workflows are continuous monitoring, preflight checks, and on-demand diagnosis.
Bring a representative failure to a MemSage evaluation. Agree on the diagnostic question, confirm compatibility, and measure the value and overhead on your workload.