My RTX PRO 6000 hard-resets the box under real inference, never under benchmarks

This is the story of two failure modes that look identical, the clock lock that fixed one of them, what 1 Hz power telemetry hides, my measured attempt to get the performance back (it hard-reset the box 16 hours in), and what I still can't prove.

Several things I believed in May turned out to be wrong. They're in here too.

TL;DR

• Two modes, one signature. Instant reset, nothing logged, unclean filesystem on the next boot. Mode A happened at idle; Mode B under live inference serving.
• Mode A is the old AMD "Power Supply Idle Control" problem, re-exposed because installing the GPU reset the BIOS. Fixed with three settings.
• Mode B only happened under real serving, never under benchmarks at three times the power. Fixed by locking the GPU clocks at min = max (nvidia-smi -lgc 1500 -lmc 14001).
• nvidia-smi can't see the problem. power.draw is a ~1 s average. NVML has a 50 Hz buffer: at the same locked clock a benchmark is flat to within a few watts, while 20 hours of real serving made 10,925 steps of ≥100 W, up to 262 W within 20 ms. A 1 Hz log of the same 20 hours shows 88 transitions.
• Raising the lock to 1950 MHz gave +22% prefill, +15% decode, +23% power, and a hard reset ~16 h later, after surviving 548 idle → load steps.
• What I can't prove: which rail sags, or whether the PSU is the weak link. The PSU passes ATX 3.1 transient tests in an independent review. There's also one reset in 4.5 months I can't explain at all.

The box

It's the same workstation as my Gemma 4 on a 5070 Ti post, with a PRO 6000 added. The PRO 6000 serves models around the clock: vLLM for a production batch LLM pipeline, llama.cpp for coding agents and evals. It is a workstation, not a lab bench, which matters later: every failed experiment is a hard reset with PostgreSQL, a git forge and inference servers live.

What a hard reset looks like

The day I installed the card, the machine started rebooting itself. Every event had the same signature:

• The next boot found unclean filesystems: an ext4 dirty bit (Fs was not properly unmounted) or a BTRFS tree-log replay.
• The previous boot's journal just stops mid-stream. No shutdown sequence, no panic, no oops.
• No MCE, no Xid, no NVRM, no PCIe AER. rasdaemon's persistent database stayed empty.
• pstore empty.
• Thermals were nowhere near limits (CPU ≤ 79 °C, VRM ≤ 59 °C, GPU ≤ 81 °C).
• The box came back by itself 15–36 seconds later. A tripped PSU protection normally latches off until you cycle power, so this was a reset, not a power-off.

The kernel never got a chance to say anything. That rules out most software explanations and points at power. It doesn't say which part of the power path.

Mode A: resets at idle

The first resets happened with the box idle or just waking up: once ~20 seconds after a request finished, as the next one arrived, and during S3 suspend attempts (the deepest low-current state there is).

This is a known AMD problem. When the CPU package drops into deep C-states, current draw on the 12 V rail falls very low, and some high-efficiency PSUs don't regulate well there. AMD's firmware has a guard for it, "Power Supply Idle Control", and installing the GPU had reset the BIOS to defaults, which turned the guard off and the deep C-states back on. I didn't measure the mechanism; I applied the known fix and the idle resets stopped.

One trap: tuned-adm profile latency-performance also caps C-states, but GNOME's power slider (via tuned-ppd) silently switches it back to throughput-performance. Put the cap on the kernel command line.

Mode B: the benchmark paradox

With Mode A fixed, the box kept resetting, now under real work: llama-server answering live requests from an LLM-judge pipeline, typically 2–3 minutes in. In my words at the time: "two minutes into this light inference where the combined power never went above 300 watts, the machine crashed!"

Meanwhile, synthetic load was rock solid: dual-GPU llama-bench at 887 W sustained for an hour (BIOS OC profile) and for 10+ minutes on stock settings with the Mode A fixes, and the PRO 6000 alone at its full 600 W for 10 minutes with no power-brake slowdown. So it wasn't capacity. What I ruled out, and how:

• Power connectors. Inspected end to end: native PSU cables, one 12V-2x6 per GPU straight from the PSU's two 600 W ports, no adapters, no browning, no bent pins.
• Thermals. Never close.
• CPU, memory, fabric. rasdaemon ran through every later crash: no MCE, no ECC errors, no AER.
• A GPU-side fault. No Xid in the journal; nvidia-smi -q -d PERFORMANCE showed no HW power-brake or thermal slowdown.
• The tensor-parallel split. The first serving crash was a model split across both cards. I pinned it to the PRO 6000 alone (CUDA_VISIBLE_DEVICES=0; the model fit seven times over). It still reset.

What survived all that: steady synthetic load at 887 W was fine; bursty real serving at ~300 W was not.

A benchmark ramps once and stays there. Serving churns: requests arrive and finish, prefill and decode alternate, and with stock clocks the GPU's boost governor moves P-states, voltages and memory clocks on every one of those edges. Each edge is a current step.

The fix: lock the clocks, min = max

nvidia-smi -i 0 -pm 1          # persistence mode
nvidia-smi -i 0 -lgc 1500      # graphics clock: min = max = 1500 MHz
nvidia-smi -i 0 -lmc 14001     # memory clock

The box stopped resetting. I made it persistent with a oneshot systemd unit that runs before multi-user.target (in the appendix), and applied the same lock to the 5070 Ti. A few things I've learned about the lock since:

• Min = max is the point. My first instinct was a floor: -lgc 1500,3090. That hands control back to the boost governor anywhere between 1500 MHz and ~2.8 GHz, which brings back exactly the voltage and frequency churn we're trying to remove.
• It doesn't remove load steps. At the lock, GPU 0 still goes from ~65 W idle to 200–360 W busy and back: 1,205 such steps in 76 hours of my 1 Hz log. What the lock fixes is the voltage behind each step.
• It isn't exact. At -lgc 1500 the graphics clock reads 1447–1485 MHz with no throttle reason active, and memory reads 13,365 MHz in P1, not the 14,001 I asked for.
• It costs idle power. The card can't drop to P8 any more. It idles at ~65 W instead of ~15 W.
• It costs throughput, more than I thought. I assumed inference was memory-bound so the lock was nearly free. That holds for single-stream decode. Prefill, batched serving and diffusion are compute-bound and scale with the core clock.

What 1 Hz telemetry hides

My forensic logger, powerwatch, wrote one line per second with GPU power and temperatures, fdatasync'd per line so the last sample survives a hard cut. It turned out to be blind in a specific way.

nvidia-smi's power.draw is the same value as power.draw.average, a ~1 s average that the driver refreshes about every 500 ms. Poll it at 10 Hz and you get the same number five times. power.draw.instant exists too, at the same refresh rate.

NVML has one more source: a raw ring buffer of power samples per GPU (nvmlDeviceGetSamples with NVML_TOTAL_POWER_SAMPLES). On driver 595 it holds ~2.4 s of samples at 20 ms spacing. Drain it every second and you have a 50 Hz power log. I rewrote powerwatch to do that (stdlib Python, ctypes straight into libnvidia-ml.so.1, nothing to install; it's in the appendix). Same card, same 1500 MHz lock:

• The benchmark steps up once, 74 → 252 W within 20 ms, then sits flat: prefill ~325 W, decode 305–312 W in every sample apart from single dips between reps. Five steps of ≥100 W in the whole run.
• Real serving, captured for 20 hours, made 10,925 steps of ≥100 W between consecutive 20 ms samples, 1,611 of them ≥200 W, the largest 262 W (66 → 328 W). The GPU was busy for only ~3.2 of those hours. A 1 Hz log of the same 20 hours counts 88 idle → load transitions.

The serving window is vLLM handling one to three requests from a batch pipeline. About twice a second the GPU runs a ~60 ms burst at ~310 W (most likely prefill), holds ~200 W for ~0.3 s, then sits idle at 67 W for 0.1–0.2 s. The 1 Hz average reads a steady ~180 W.

Even 50 Hz isn't the bottom. The GPU's own SW Power Capping counter (nvidia-smi -q -d PERFORMANCE) logged 25.5 seconds of capping over five days in which no 1 Hz sample went above 356 W, against a 600 W limit. During the 20-hour capture it rose by another second while no 20 ms sample exceeded 390 W. The card's power controller is reacting to excursions none of the exposed telemetry shows. I don't know whether that's the board power limit or a current limit. Microsecond-scale behaviour on the 12 V rail needs a scope, and I don't have one on it.

Trying to get the speed back

Four months later the lock looked expensive. I'd been running a 27B dense model with speculative decoding next to vLLM, and decode was clearly capped. So I asked the obvious question, "am I gimping my rtx6k?", and raised the lock live, keeping min = max: nvidia-smi -i 0 -lgc 1950.

Measured on the same real workload (a batch pipeline driving llama-server, ~170 requests per side, ~1,050 prompt + ~320 generated tokens each, speculative decoding at 94–95% acceptance):

Over the full windows, at 1 Hz:

The box hard-reset at 04:10:54 the next morning, ~16 hours in. Same signature: journal stops mid-line, BTRFS log replay on the next boot, nothing in pstore, no Xid, no MCE. I found out because I woke up at 4:30 and the machine had a 16-minute uptime.

Two things in that trace go against my first reading of it. The box didn't die on the idle → load step; it died about 30 seconds into the burst. And it had already survived 548 of those steps at 1950. Whatever kills it is rare per event: zero Mode B resets in ~4 months at 1500, one in 16 hours at 1950, and minutes at stock clocks. The soak also ran during heavier traffic (twice the step rate), so the voltage point and the number of steps are confounded. n = 1. I put the lock back to 1500.

What I got wrong

The May postmortem ended with "the PSU's transient response is insufficient, buy a bigger one". Going back through the evidence for this post, that conclusion stood on less than I thought:

• "NVIDIA specs 1000 W for the PRO 6000 alone." I never found a source for that number. It stays out.
• The PSU is not obviously weak. The Dark Power 14 1200 W is ATX 3.1 and PCIe CEM 5.1 compliant. HWBusters' review has it passing every ATX v3.1 transient test, with good 12 V transient response. Their one weak spot is 3.3 V. I run it with OCK on, the single-rail mode they recommend for spiky GPUs.
• "With clocks locked, power is steady." No. The lock removes V/F transitions, not load steps.
• "Stable since the lock." Not quite. On 12 June the box reset once more, at idle, after ~10 days of uptime, with every mitigation verified in place. powerwatch happened to be disabled that week, so there's no trace. I don't know what it was. A mains event isn't ruled out.

One new piece of evidence does point somewhere. Since Linux 6.16, AMD systems print the FCH's recorded reason for the previous reset at boot (x86/amd: Previous system reset reason). After a planned reboot this box logs software wrote 0x6 to reset control register 0xCF9, so the decode works. After the 10-08 reset it logged nothing: none of the causes the kernel knows was recorded. No thermal trip, no watchdog, no CPU shutdown, and no data-fabric sync flood, which is what a fatal MCE or PCIe error escalates to.

My best explanation now: a rail sags far enough to drop PWR_OK, the board resets, and the PSU (which never tripped) brings it straight back up. That fits every observation. It is still a hypothesis.

What's next

1. A logging UPS (pure sine, monitored with NUT). It settles the mains question, which 12 June raises, and logs input dips.
2. A 1600 W PSU with native 12V-2x6. That's the decisive test: if a raised lock holds on it, the PSU was the weak link. If it doesn't, the board is next.

Both will be a follow-up post, with 50 Hz traces either way.

If you run one of these

• Check the BIOS after installing the card. A reset to defaults re-enables deep C-states.
• Put processor.max_cstate=1 on the kernel command line and mask the sleep targets.
• Lock the clocks min = max with a boot-time unit. Never a floor-only range.
• Log power at 50 Hz from NVML, not nvidia-smi at 1 Hz, and fsync every line.
• Don't trust a benchmark to validate stability. Validate with the bursty workload you'll actually run, for days.

Appendix

The triage runbook, the clock-lock unit, the full powerwatch source, and how to arm crash capture on Fedora. Click any to expand.