MemSage
CPU and GPU memory diagnostics for AI training and inference. Available in private beta.
Resources
Understand the workload. Investigate the failure.
Product access, practical guides, and published research from Teyon on memory diagnostics and AI infrastructure.
Download Our Products
Product demos
Short recordings of the products at work, made by the team on their own runs. Each plays here; nothing opens on another site.
MemSage
Profiling a PyTorch run with no code changes
A training script is launched under LD_PRELOAD with MemSage attached and runs unmodified. When it fails with an out-of-memory error, the recording opens the capture in Perfetto: the allocation timeline, the tensor graph, and the lines of code behind the growth. Recorded under the product's earlier name, MemScope.
Recorded
Watcher
Automatic failure diagnosis, live
An unmodified audio program crashes with a floating-point exception. Run again under Watcher, the same crash comes back with the faulting instruction, the thread, and the chain of writes that led to it, level by level, before Watcher offers a rollback.
Recorded
Engineering guides
Tool selection
Which tool should you use to investigate a training problem?
Choose evidence for the question you need to answer: operation cost, memory ownership, system behavior, or restarting a run. These workflows complement one another.
By Teyon AI ·
Read the guideGlossary
GPU memory and training diagnostics glossary
Short definitions for reading memory reports and discussing a training failure. Start here, then follow the linked guides for investigation steps.
By Teyon AI ·
Read the guideMemory diagnostics
How do you investigate a GPU out-of-memory failure?
Start with the failing allocation, the memory state around it, and the workload configuration. An OOM message identifies a failure; it does not by itself establish a leak.
By Teyon AI ·
Read the guideMemory diagnostics
Why do GPU memory measurements disagree?
Tensor memory, allocator reservations, and device-wide usage measure different things. Compare them at the same point in a run before diagnosing a leak.
By Teyon AI ·
Read the guideEvaluation checklist
What should you check before a long AI training run?
Record the configuration, exercise representative inputs, inspect memory behavior, and agree on a failure-response plan before committing to a longer job.
By Teyon AI ·
Read the guideResearch foundations
Peer-reviewed work behind our approach. Results belong to the workloads evaluated in each paper; they are not MemSage product benchmarks.
OOPSLA '23 · 2023
MemPerf: Profiling Allocator-Induced Performance Slowdowns
A research profiler that attributes performance slowdowns to memory allocation and the later use of allocated objects.
Read the summaryISMM '23 · 2023
NUMAlloc: A Faster NUMA Memory Allocator
In the paper's evaluation, this NUMA allocator ran 15.7% faster than mimalloc and 20.9% faster than glibc, scaling to 128 threads.
Read the summaryOOPSLA '20 · 2020
Watcher: In-Situ Failure Diagnosis
Diagnosed 24 evaluated program failures inside the failing process, reporting root causes in a few seconds without sharing a full memory dump.
Read the summaryUSENIX Security '18 · 2018
Guarder: A Tunable Secure Allocator
A secure allocator with tunable randomization entropy and under 3% average runtime overhead in the paper's evaluation.
Read the summaryPLDI '18 · 2018
iReplayer: In-Situ and Identical Record-and-Replay for Multithreaded Applications
Replays the last execution epoch inside the original process, with about 3% average recording overhead on the evaluated workloads.
Read the summaryCCS '17 · 2017
FreeGuard: A Faster Secure Heap Allocator
Adds defenses against several heap attacks with under 2% average runtime overhead in the paper's evaluation.
Read the summaryICSE '16 · 2016
DoubleTake: Fast and Precise Error Detection via Evidence-Based Dynamic Analysis
Checks for evidence of buffer overflows, use-after-frees, and leaks at epoch ends, with under 5% average overhead in the paper's evaluation.
Read the summarySOSP '11 · 2011
Dthreads: Efficient Deterministic Multithreading
Provides deterministic execution for supported C/C++ multithreaded programs, matching or exceeding pthreads on 8 of 14 evaluated benchmarks.
Read the summaryPatent record
Historical records of U.S. patent grants naming Tongping Liu as an inventor, with the universities and companies listed as assignees. These are inventor credentials, not a list of patents owned by Teyon. Each entry links to its public record.
- US 12,541,443Classification of Different Types of Cache MissesUniversity of Massachusetts Amherst · 2026
- US 12,314,761Memory Allocation and Management in Non-Uniform Memory Access (NUMA) Computing EnvironmentsUniversity of Massachusetts Amherst · 2025
- US 11,593,483Guarder: An Efficient Heap Allocator with Strongest and Tunable SecurityUniversity of Texas System · 2023
- US 11,599,445Watcher: Precise and Fully-Automatic On-Site Failure DiagnosisUniversity of Texas System · 2023
- US 11,294,652Low-Overhead Detection Techniques for Synchronization Problems in Parallel and Concurrent SoftwareUniversity of Texas System · 2022
- US 10,901,828FreeGuard: A Faster Secure Heap AllocatorUniversity of Texas System · 2021
- US 10,915,424Defeating Deadlocks in Production SoftwareUniversity of Texas System · 2021
- US 10,394,714System and Method for False Sharing PredictionFuturewei Technologies · 2019
- US 10,474,369Mapping Guest Pages to Disk Blocks to Improve Virtual Machine Management ProcessesVMware · 2019
- US 9,547,599System and Method for Predicting False SharingFuturewei Technologies · 2017
- US 9,678,883System and Method for Detecting False SharingFuturewei Technologies · 2017
- US 8,868,849Coupled Lock Allocation and Lookup for Shared Data Synchronization in Symmetric Multithreading EnvironmentsSamsung Electronics · 2014