About the role
We are building tools that make large GPU clusters reliable. We are launching an independent acceptance-testing service for newly delivered GPU clusters: before a customer puts hundreds of 8-GPU servers into production, we test every server and every GPU, find the faulty hardware, and give the customer evidence they can use for warranty replacement. Our tests include detection of silent data corruption, where a GPU computes wrong results without reporting any error. You will work directly with the founder. The founder leads the design and core development of the test platform. Your job is to run the testing on site at customer data centers and to contribute to the platform itself. You will split your time between the data center floor and the codebase, and what you learn in the field will shape what we build.
What you will do
- On site: Bring up our test platform on the customer's cluster: verify access, check that servers, networks, drivers and firmware are ready, and confirm that power and cooling can support all machines running at full load at the same time.
- On site: Run the test campaign across hundreds of servers: version and configuration checks, per-GPU diagnostics and performance, node and network tests, storage and thermal checks, multi-day burn-in, and silent-data-corruption screening.
- On site: Triage failures. Reproduce them, separate real hardware faults from configuration or software problems, and isolate the faulty GPU, cable, NIC or server.
- On site: Do hands-on work when needed: reseat components, swap cables and optics, read BMC logs, and work with data center staff.
- On site: Collect the evidence for each faulty part so the customer can file warranty claims, and help prepare the final report.
- Back at the office: Develop and harden the test platform: test orchestration across many nodes, automation, result collection, and report generation.
- Back at the office: Add new tests and failure checks based on what you saw in the field.
- Back at the office: Write runbooks so each engagement runs faster and needs less manual work than the last.
What we are looking for
- 3+ years working with Linux servers in production, including debugging at the hardware, driver and OS level.
- Hands-on experience with NVIDIA data center GPUs (HGX/DGX class) and their tooling: nvidia-smi, DCGM, NVIDIA drivers and CUDA, and reading Xid and ECC errors.
- Working knowledge of cluster networking, including InfiniBand or RoCE, and of collective communication tests such as nccl-tests.
- Experience with server management interfaces: BMC, IPMI or Redfish.
- Solid scripting and programming in Python and Bash. You can write maintainable tools, not just one-off scripts.
- Experience running jobs across many machines with tools such as Slurm, Kubernetes, Ansible or plain SSH automation.
- Comfort working on a data center floor and handling server hardware.
- Clear written communication. You will write findings that customers and hardware vendors rely on.
Nice to have
- Experience bringing up or burning in large GPU clusters at a cloud provider, OEM, system integrator or AI lab.
- Familiarity with benchmarks such as HPL, MLPerf, nvbandwidth or fio.
- Experience with hardware RMA processes and working with server or GPU vendors.
- CUDA or PyTorch programming experience, especially around numerical reproducibility and deterministic execution.
- Knowledge of hardware reliability topics such as ECC, memory errors and silent data corruption.
- Ability to communicate with customers and partners in Mandarin as well as English.