GPU Utilization Monitoring: Finding the 40% You Are Wasting

GPU utilization monitoring only finds wasted spend when it measures what the GPU actually does, not whether a kernel happens to be running. Track allocation, chip activity, and model FLOPs utilization together, tie every GPU-hour to a job, and the gap between what you pay for and what you use becomes visible and fixable.

• Idle capacity is the norm: Cast AI’s 2026 State of Kubernetes Optimization Report measured average GPU utilization of 5% across roughly 23,000 Kubernetes clusters, and its authors describe about 50% as a realistic benchmark for optimized production systems.

• The default metric overstates work: The GPU-Util figure in nvidia-smi only records whether a kernel was running, while NVIDIA’s DCGM documentation says SM activity of 0.8 or more is necessary but not sufficient for effective use, and values below 0.5 likely indicate waste.

• Even excellent training leaves headroom: Meta reported 38 to 43% BF16 model FLOPs utilization while pretraining Llama 3 405B on up to 16,384 H100 GPUs, so a cluster that looks fully busy can still leave more than half of its peak compute unused.

• Attribution drives the savings: By joining DCGM telemetry with Slurm job data and automatically ending jobs that sat idle for an hour, NVIDIA cut idle GPU waste on its research clusters from about 5.5% to about 1%.

• Utilization sets your real price: On GMI Cloud, H200 capacity starts at $3.20 per GPU-hour (as of October 2026), which works out to about $5.33 per productive GPU-hour at 60% utilization and $3.76 at 85%. Closing that gap is usually cheaper than buying more GPUs.

Teams fight hard for GPU capacity and then lose a large share of it quietly. In April 2026, Cast AI published its 2026 State of Kubernetes Optimization Report, which found GPU utilization averaging 5% across the clusters it analysed, with organizations provisioning about 20 times more GPU capacity than they actively used. The sample covers Kubernetes clusters on the large public clouds rather than dedicated training labs, and the report’s authors put the realistic target for optimized production environments at around 50%.

The 40% in the title is a working number, not a law. A team that averages 60% utilization pays for 40% of its GPU-hours without getting work back, and many teams cannot say where they sit because their dashboards show only GPU-Util, a metric that reads close to 100% as long as any kernel is running. GPU utilization monitoring that finds real waste needs three views: Whether a GPU is allocated and doing anything at all, how much of the chip is working when it is, and how much useful model work comes out.

This guide covers the metrics that separate busy from productive, a practical DCGM and Prometheus setup, the patterns behind common waste, and how to price the gap, with notes on running these workloads on GMI Cloud.

Why GPU-Util Overstates the Work Your GPU Does

A Duty Cycle, Not a Measure of Work

The GPU-Util column in nvidia-smi, exported by DCGM as DCGM_FI_DEV_GPU_UTIL, is a duty cycle. It records the share of each sample window in which at least one kernel was executing. It says nothing about how many streaming multiprocessors (SMs) that kernel occupied or whether the Tensor Cores did anything.

NVIDIA’s DCGM profiling documentation shows why this matters. If a kernel launches only enough thread blocks for one fifth of the SMs and runs for the entire interval, SM activity reads 20%. A duty-cycle metric would show the same GPU as fully busy. The difference between those two numbers is pure waste that never appears on a GPU-Util dashboard.

GPU-Util is still useful for spotting GPUs that do nothing at all. NVIDIA’s own fleet team used it as its primary idle signal, while planning a move to DCGM_FI_PROF_GR_ENGINE_ACTIVE for a more precise view. The mistake is treating it as a measure of efficiency.

Three Layers of GPU Waste

In its write-up on making GPU clusters more efficient, NVIDIA groups waste into four categories: Hardware that has failed, healthy GPUs that nobody has scheduled, jobs that hold GPUs but never use them, and jobs that use GPUs inefficiently. NVIDIA observed the last category most often and idle-but-allocated jobs at a moderate rate. For monitoring, it helps to collapse these into three layers, because each has its own metric and usually its own owner.

The GPU Utilization Metrics That Matter

NVIDIA’s Data Center GPU Manager (DCGM) reads hardware counters on data-center GPUs at low overhead. Its profiling fields are sampled at 1 Hz by default and can be queried as often as every 100 ms. They are interval averages rather than a kernel trace, so they tell you which GPU or job is underperforming, not which line of code is responsible. NVIDIA recommends using DCGM to locate an underused device and then switching to Nsight Systems or Nsight Compute for application-level detail.

Graphics Engine Activity

Field 1001, exported as DCGM_FI_PROF_GR_ENGINE_ACTIVE, measures the fraction of time any part of the graphics or compute engine was busy. It is a better duty-cycle signal than GPU-Util and the natural input for idle detection, but it shares the same weakness: A trickle of work keeps it high.

SM Activity and Occupancy

Field 1002, DCGM_FI_PROF_SM_ACTIVE, is the fraction of time at least one warp was active on an SM, averaged across all SMs. This is the first metric that reflects how much of the chip is in use. NVIDIA gives a clear rule of thumb: 0.8 or higher is necessary but not sufficient for effective use, and anything under 0.5 likely means the GPU is being used ineffectively. “Active” does not mean computing, because warps waiting on memory still count.

Field 1003, DCGM_FI_PROF_SM_OCCUPANCY, measures resident warps relative to the maximum an SM supports. Higher occupancy helps memory-bandwidth-bound kernels hide latency, but NVIDIA notes it does not necessarily improve compute-bound workloads, so treat it as a diagnostic rather than a target.

Tensor and DRAM Activity

Field 1004, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, is the fraction of cycles the Tensor Core pipe was active. For transformer training and inference, this is closest to “the work you are paying for”. NVIDIA warns that 20% tensor activity is ambiguous on its own: It could mean a fifth of the SMs running flat out, every SM at a fifth of capacity, or full activity for a fifth of the time. Read it alongside SM activity to tell these apart.

Field 1005, DCGM_FI_PROF_DRAM_ACTIVE, is the fraction of cycles the memory interface was sending or receiving data. NVIDIA notes that about 0.8 is the practical maximum, so a value near that level means the workload is limited by memory bandwidth, not by compute.

Note that the current DCGM documentation lists some of these fields under newer RATIO names, while DCGM Exporter still uses the *ACTIVE names shown here; the field IDs are the stable reference.

Model FLOPs Utilization

Hardware counters show activity; model FLOPs utilization (MFU) shows output. MFU compares the floating point operations your model mathematically requires with what the GPUs could deliver at peak. NVIDIA’s DGX Cloud benchmarking recipes define it as follows.

MFU = (global batch size × model FLOPs per sample) ÷ (step time × number of GPUs × peak FLOPS per GPU)

The same recipes put the H100’s peak at 989 TFLOPS for dense BF16 and 1,979 TFLOPS for dense FP8, and report roughly 43 to 45% MFU for Llama 3.1 405B in FP8 across 32 to 2,304 H100 GPUs. Meta’s Llama 3 paper reports 38 to 43% BF16 MFU on up to 16,384 H100s. An MFU near 40% on a large training job is therefore good, while a single-digit MFU on a job whose GPU-Util reads 95% is a strong sign of activity waste. For inference, vLLM can report MFU through its --enable-mfu-metrics flag.

Setting Up GPU Utilization Monitoring with DCGM

Spot Checks with dcgmi

Before building dashboards, confirm which profiling fields your GPU supports and watch them while a real job runs. Field support varies by GPU generation, so NVIDIA recommends asking each GPU for its own catalogue rather than assuming. Profiling counters require administrator privileges, which means nv-hostengine must run as root.

# List the profiling fields this GPU supports
dcgmi profile --list --entity-id gpu:0

# Stream engine, SM, tensor and DRAM activity once per second
dcgmi dmon --entity-id gpu:0 --field-id 1001,1002,1004,1005 --delay 1000

DCGM profiling and Nsight tools compete for the same counters, so run dcgmi profile --pause before an Nsight session and dcgmi profile --resume afterwards.

Fleet Telemetry with DCGM Exporter

DCGM Exporter turns DCGM fields into Prometheus metrics, served on port 9400 at /metrics. Its default counter file leaves DCGM_FI_PROF_SM_ACTIVE and DCGM_FI_PROF_SM_OCCUPANCY commented out, so the most useful efficiency metrics are missing unless you supply your own list. Keep that list short: DCGM multiplexes counters that cannot be read together, and requesting many profiling fields lowers the accuracy of each one.

# dcgm-metrics.csv: DCGM field, Prometheus type, help text
DCGM_FI_DEV_GPU_UTIL,            gauge, GPU utilization (%).
DCGM_FI_DEV_FB_USED,             gauge, Memory used (MiB).
DCGM_FI_DEV_POWER_USAGE,         gauge, Power draw (W).
DCGM_FI_PROF_GR_ENGINE_ACTIVE,   gauge, Engine activity.
DCGM_FI_PROF_SM_ACTIVE,          gauge, SM activity.
DCGM_FI_PROF_SM_OCCUPANCY,       gauge, SM occupancy.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Tensor pipe activity.
DCGM_FI_PROF_DRAM_ACTIVE,        gauge, Memory interface activity.

Deploying DCGM Exporter with Custom Metrics

To deploy DCGM Exporter, use NVIDIA's official container image with the custom dcgm-metrics.csv configuration file. Select a current image version from NVIDIA NGC, enable GPU access and the required profiling permissions, and make the configuration file available inside the container.

Configure the exporter to expose Prometheus metrics on port 9400. Once it is running, check the /metrics endpoint and confirm that DCGM_FI_PROF_SM_ACTIVE appears in the collected metrics.

Attribute Every GPU-Hour to a Job

A fleet-wide average tells you that waste exists, not who can fix it. NVIDIA’s cluster team built its pipeline by joining DCGM telemetry with Slurm job metadata, using the exporter’s HPC job-mapping feature to tag GPU activity with job context. Slurm data arrived at five-minute granularity, which was enough to align with the higher-resolution DCGM fields and compute an idle-waste metric per job.

The team’s main lesson was organizational. Once utilization was visible per user and per job, better habits followed, especially when the data appeared at job submission time and inside experiment-tracking tools.

Dashboard Queries Worth Keeping

Three queries cover most weekly reviews: A fleet efficiency trend, idle GPUs, and GPUs that look busy without doing much tensor work.

# Fleet-wide SM activity over the last day (1.0 = 100%)
avg(avg_over_time(DCGM_FI_PROF_SM_ACTIVE[1d]))

# GPUs that have been allocated but idle for the past hour
max_over_time(DCGM_FI_PROF_GR_ENGINE_ACTIVE[1h]) == 0

# Busy but not productive: engine active, Tensor Cores mostly idle
avg_over_time(DCGM_FI_PROF_GR_ENGINE_ACTIVE[15m]) > 0.8
  and avg_over_time(DCGM_FI_PROF_PIPE_TENSOR_ACTIVE[15m]) < 0.2

Reading the Signals: Where the 40% Goes

Allocated but Idle

NVIDIA’s list of recurring idle patterns will look familiar to anyone who runs a shared cluster. It includes CPU-only jobs on GPU nodes, jobs that request a whole node but use part of it, stalled jobs, slow container downloads, and forgotten interactive sessions. The fixes were mostly operational. An idle-job reaper ended workloads after an hour of continuous GPU inactivity, with thresholds users could tune for jobs that have predictable quiet phases. A job linter flagged exclusive-node requests that left GPUs unused, and a separate tool cancelled queued follow-on jobs that were no longer needed.

Starved by the Input Pipeline

When engine activity saw-tooths between high and low while host CPUs run hot, the GPU is usually waiting for data. PyTorch’s performance tuning guide points out that DataLoader defaults to num_workers=0, which loads data synchronously in the main process, and that pin_memory defaults to False. Raising num_workers overlaps loading with compute, and pinned memory enables faster, asynchronous host-to-GPU copies. Data stored far from the GPUs makes it worse, so local NVMe or a nearby cache often helps.

Waiting on Communication and Checkpoints

Distributed jobs show a different pattern: Activity drops on every rank at the same moment. NVIDIA lists checkpoint reads and writes, data loading, and initialization among the expected idle phases of training jobs. Synchronized dips usually point to blocking checkpoints or collectives waiting on the network; a single trailing rank points to a straggler or a degraded node. Compare per-GPU series rather than averages, watch the NVLink and PCIe byte counters during collectives, and run dcgmi diag on any node that keeps falling behind.

Memory-Bound Inference

Inference servers often show high DRAM activity with modest tensor activity, especially at small batch sizes, because token generation reads model weights and KV cache far more than it computes. That is normal behaviour, not waste, but it changes what “underused” means. The useful questions are whether requests queue while the GPU has room, and whether the GPU sits loaded with a model while almost no requests arrive. vLLM’s production metrics answer both through its waiting-requests gauge (vllm:num_requests_waiting) and its KV cache usage gauge (vllm:kv_cache_usage_perc). Long periods with an empty queue and a nearly empty cache mean the deployment has more replicas than traffic needs.

Putting a Price on Wasted GPU Hours

Percentages rarely move budgets; dollars do. Two formulas turn GPU utilization monitoring into a cost conversation.

cost per productive GPU-hour = hourly rate ÷ average utilization

monthly idle spend = (1 - average utilization) × GPUs × hours per month × hourly rate

The example below uses eight H200 GPUs running all month (730 hours) at GMI Cloud’s on-demand starting rate of $3.20 per GPU-hour (as of October 2026) from the pricing page, which puts the monthly bill at $18,688. Committed and reserved rates are quoted per customer, so substitute YOUR_COMMITTED_RATE if you have one.

Be explicit about which utilization you plug in. An allocation-level figure (engine activity above zero) measures what you pay for idle time and is the right input for these formulas. SM activity and MFU measure how well the busy hours are used; they explain why a job takes longer than it should, but multiplying them into the same formula double-counts. Moving the same eight GPUs from 60% to 85% allocation-level utilization saves about $4,672 a month, which is often more than a GPU upgrade would.

Running Utilization-Aware Workloads on GMI Cloud

The best fix for idle GPUs is not to pay for them when traffic does not justify it. GMI Cloud’s serverless inference scales to zero with no idle cost and includes built-in batching and latency-aware scheduling, so spiky or early-stage workloads hand the utilization problem to the platform. When traffic becomes steady, dedicated endpoints give you your own GPUs for a model, and GMI’s documentation lists a Stopped state for deployments you halt manually, which suits endpoints that serve nobody overnight.

For training and self-managed inference, GMI Cloud offers dedicated, single-tenant NVIDIA GPUs priced per GPU-hour through its Cluster Engine. Bare metal servers give root access, an isolated VPC, and RDMA-ready networking, which also means you can run nv-hostengine with the privileges DCGM profiling counters require. On the Kubernetes-based Container Service, confirm with GMI which profiling counters are exposed before you build dashboards around them. GMI’s guide to comparing container, bare metal, and managed cluster quotes explains how the three options differ in scope.

As of October 2026, the GMI Cloud pricing page lists H100 from $2.60 per GPU-hour, H200 from $3.20, B200 from $5.00 with limited availability, and GB200 from $8.00, with lower unit costs available through reserved capacity and commitment-based pricing. GMI reports that Higgsfield runs real-time generative video on the platform with 45% lower compute cost and 65% lower inference latency. See the GMI Cloud GPU page for hardware options.

Conclusion

GPU utilization monitoring pays for itself when it stops asking whether a GPU is busy and starts asking whether it is productive. GPU-Util catches GPUs that do nothing; SM, tensor, and DRAM activity from DCGM show how much of the chip a busy job actually uses; and MFU ties that activity to model progress, with roughly 40% MFU a strong result even for the largest training runs. Attribute every GPU-hour to a job, automate obvious fixes such as reaping idle jobs, and express utilization as cost per productive GPU-hour so the business case is clear. For workloads with variable demand, pairing that discipline with scale-to-zero serverless inference and dedicated GPUs priced per GPU-hour on GMI Cloud keeps the 40% from coming back.