AI · How-to
nvidia-smi, and how to power-cap your GPU for AI
Updated 19 July 2026 · about an 8 minute read
If you run anything on an NVIDIA card, nvidia-smi is the one command worth knowing. It came free with your driver, it needs no install, and typing those ten letters tells you everything: how hot the card is, how much power it's pulling, how full the VRAM is, and exactly what's using it. And the trick most people miss - you can also tell it to draw less power, which for local AI means a cooler, quieter, cheaper card that runs your models almost as fast.
I keep a terminal with this running whenever I've got a model loaded.
What nvidia-smi is
It stands for NVIDIA System Management Interface, and it's a little command-line tool that ships with every NVIDIA driver. Install the driver, you've got it - Windows, Linux, doesn't matter. It reads straight from the card, so what it tells you is the truth, not an estimate. It monitors, and it manages: you can watch the GPU, and you can change a few things about how it runs. The official manual lists every last flag, but you'll use about six of them.
Reading the dashboard
Open a terminal and type nvidia-smi. You get two tables: the card, then a process list. Here's the top table from my own 48GB 4090, captured while vLLM was serving a model (I've written up that whole setup separately):
Mon Jul 20 11:33:58 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 610.74 KMD Version: 610.74 CUDA UMD Version: 13.3 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4090 WDDM | 00000000:01:00.0 On | Off | | 39% 47C P2 108W / 370W | 47774MiB / 49140MiB | 100% Default | +-----------------------------------------+------------------------+----------------------+
Top strip first: the driver version and the CUDA version the driver supports - worth a glance, because a lot of AI tooling is fussy about CUDA. Then the card itself, row by row:
- Temp - 47C here, which is a water-cooled card being spoiled. On air under load you want it under about 80; over 83 and the card starts throttling itself.
- Memory-Usage - 47774 of 49140 MiB. The model plus its context cache has taken nearly all 48GB, which is deliberate (vLLM grabs what you let it). If a model won't fit at all, things spill to system RAM and crawl.
- GPU-Util - 100%, so the card's flat out, not sat waiting on the CPU.
- Pwr: Usage/Cap - the pair this whole article builds to. Look at it next to the util figure: the card is at 100% utilisation and drawing 108 watts of a 370-watt cap. Flat out, and using less than a third of its power budget - because LLM inference is limited by memory bandwidth, not raw compute. That's the entire case for power-capping, made by the card itself.
- Perf - the performance state, P0 (flat out) down to P12 (idle). P2 is the normal compute state under load.
The second table is the process list: which programs are on the GPU and how much memory each has taken. A C marks a compute process (your AI runner); a G is graphics. One wrinkle from my setup that confused me at first: because my vLLM runs inside WSL2, the Windows-side process list shows only desktop apps - the actual model server doesn't appear there at all, even while it's clearly the thing using 47GB. Run nvidia-smi inside the WSL2 terminal instead and the compute process shows up properly. As I understand it that's a WDDM quirk rather than anything being wrong.
Watching it during inference
A single snapshot is fine, but the interesting bit is watching the numbers move while a model answers. Refresh it once a second:
nvidia-smi -l 1
On Linux watch -n1 nvidia-smi does the same and redraws in place. For something you can actually log or graph, ask for just the columns you care about, as CSV:
nvidia-smi --query-gpu=timestamp,power.draw,temperature.gpu,utilization.gpu,memory.used --format=csv -l 1
timestamp, power.draw [W], temperature.gpu, utilization.gpu [%], memory.used [MiB] 2026/07/20 11:35:36.506, 23.75 W, 42, 6 %, 990 MiB 2026/07/20 11:35:37.519, 24.91 W, 42, 6 %, 995 MiB 2026/07/20 11:35:38.528, 24.49 W, 42, 7 %, 1003 MiB
That's my card a few minutes after the snapshot above, ticking over between jobs at about 24 watts. Worth knowing your idle number - it's what the card costs you all day when nothing's running. Fire a prompt at the model and you'll see the power line jump and utilisation head for 100; that's how you confirm the GPU's actually doing the work, and how you spot the classic bug where everything's running on the CPU by mistake and the GPU sits near zero.
Power-capping for AI (the useful bit)
Here's the part the tutorials skip. For running LLMs, a graphics card's flat-out power limit is mostly wasted. Inference is limited by memory bandwidth, not raw compute, so once the model's loaded the card spends a lot of its wattage generating heat rather than tokens. Which means you can cap the power hard and lose almost nothing - you saw it in my snapshot above, flat out at 108 of 370 watts. Owner reports on the 5090 tell the same story: pulled from 575 watts down to 400, a 32B model reportedly barely slows. What a cap does change is the heat, the fan noise, and the number on your electricity bill.
Three commands. First, turn on persistence mode so your settings stick (Linux):
sudo nvidia-smi -pm 1
Check what your card will allow - it'll show a min and max enforced limit:
nvidia-smi -q -d POWER
Then set the cap, in watts:
sudo nvidia-smi -pl 300
That's it. The card now won't draw more than 300 watts. (Why capping works so well for inference, cap versus undervolt, and how to find your own card's knee is now its own article.) One catch: this resets when you reboot, so if you want it permanent, drop it into a startup script or a systemd service. And a platform note - -pl works cleanly on Linux with sudo, but on Windows the GeForce driver often blocks it, in which case set the power limit slider in MSI Afterburner instead. Same result, different door.
Sensible caps per card
The 4090 row reflects what my own card does; the rest are collated from owner reports and reviewer testing rather than my bench, so treat them as places to start. Every card's a little different - cap it, run your usual model, and check the tokens per second barely moved. If it dropped more than a few percent, give it back 25 watts.
| Card | Stock | Inference cap | Why |
|---|---|---|---|
| RTX 3090 | 350W | 250-300W | GDDR6X runs hot; capping keeps the memory junction sane. |
| RTX 4090 | 450W | 300-350W | The efficiency sweet spot. ~300W loses only a few tokens/s. |
| RTX 5090 | 575W | 400-450W | Bandwidth-bound; 575 to 400W barely moves a 32B model. |
| RTX 3080 | 320W | 240-260W | Small card, big saving, negligible speed loss. |
The cheat sheet
nvidia-smi The dashboard - one snapshot of everything. nvidia-smi -L List the GPUs in the machine. nvidia-smi -l 1 Refresh the dashboard every second (watch -n1 nvidia-smi on Linux). nvidia-smi dmon A scrolling one-line-per-second monitor - power, temp, util. nvidia-smi -q -d POWER Full power detail, including the min/max limits your card allows. nvidia-smi -pm 1 Persistence mode on, so settings and the driver stay put (Linux). sudo nvidia-smi -pl 300 Set the power cap to 300W. Resets on reboot. The walk-through that got me started is below if you'd rather watch someone do it. And once you've got a feel for what your card's actually doing, the GPU comparison is where I'd look next - every card's page has the power-limit and undervolt settings people run for AI.
Video: Liv4IT - How To Monitor and Manage GPUs with Nvidia-smi Command. Command reference: NVIDIA's official nvidia-smi manual.