Executive Summary

Key Figures

All four come from the Princeton team's own count across 7,399 jobs over ten days. The first three say how much capacity goes unused; the last says how long the line for that capacity is.

Source: arXiv 2610.36329 (2026-10-26)

On October 12 the Korea Economic Daily, citing the Wall Street Journal, reported how much harder GPUs have become to obtain. By the data firm SemiAnalysis's tally, the hourly cost of renting Nvidia H100-based compute under a one-year contract had risen 60% over the previous year. One qualification is worth keeping. That 60% applies to one-year contracts, and on-demand pricing, bought by the hour, moved in a different direction over the same stretch. What does not change is that whoever is locking in long-term capacity is paying more for it.

Two weeks later, on October 26, Andrew Sheinberg, Samyak Gupta, and Kai Li of Princeton's computer science department posted a paper to arXiv. Its title is "Plenty of Room in the Middle." It reports ten days of watching their own university's shared GPU cluster at 100 ms intervals.

The middle in that title is not the middle of the cluster but the middle of the computing stack. Work on making computation faster usually happens at the two ends: building faster hardware at the bottom, writing better-optimized applications at the top. The authors cite Feynman's 1960 lecture on the room at the bottom alongside a 2020 paper on the room at the top, then write that there is also plenty of room in the middle. That middle is the systems layer that assigns GPUs to jobs, sets their order, and multiplexes several jobs onto the same hardware. Much of the capacity today's clusters leave idle, they argue, can be recovered in that layer.

The two numbers do not contradict each other, and nothing here says the shortage is overblown. The paper's opening paragraph takes the shortage as given: high cost and growing demand make for long queueing delays on GPU clusters. That delay was measured on this cluster too. A job requesting a single GPU waited a median of nearly 22 hours, and jobs requesting several waited two days.

The machine under measurement is a research cluster shared by 150 people from 12 departments. Each of its 20 high-performance nodes carries a 128-core AMD EPYC Rome CPU, 768GB of RAM, and two Nvidia A100 PCIe 40GB cards. Each of its 69 standard nodes carries a 48-core Intel Ice Lake CPU, 100GB of RAM, and four A100 SXM4 80GB cards. The nodes are tied together over 100Gb/s InfiniBand. Together that comes to 89 nodes and 319 GPUs.

The team gathered several months of data but narrowed the analysis to ten days, the stretch where the mapping between job IDs and GPUs survived intact. Inside that window, 7,399 jobs ran.

Two reasons are given for the 100 ms interval. One is that 100 ms is the finest resolution Nvidia's DCGM library supports. The other is that it matches the length of Linux's default round-robin time slice. It is a decision to look at the GPU on the same ruler the operating system uses to share out a CPU.

Running at that interval for months takes a light tool. The standalone daemon the team wrote in C consumes about 0.9% of a single CPU core and takes roughly 800 microseconds per sampling round on a host with four GPUs. It produces about 1.5GB per host per day, and its local buffer holds 600 samples in under 2MB. Each host keeps its own SQLite database, so no central synchronization is required.

Six things get written down every 100 ms: compute-core (SM) utilization, GPU memory utilization, SM occupancy, PCIe traffic, NVLink traffic, and Tensor Core usage. The idle figures in Section 3 come from the first two, and the communication periods in Section 5 are read out of the PCIe and NVLink records.

2.1What a Minute-Level Ruler Could Not See

Earlier studies worked with much coarser rulers. Microsoft's Philly and Helios sampled at 100-second intervals, and AntMan at around one second. MIT SuperCloud did sample at 100 ms, but read its numbers from nvidia-smi, which changes what the number means.

The utilization nvidia-smi hands back is the fraction of time a kernel was running. It does not ask how many compute cores ran. The paper's example is sharp: a job using just one SM out of 108 on an A100 could still be reported as 100% utilized. DCGM's SM Activity divides that fraction across the cores, so it records the same instant far lower.

Memory first. More than 70% of single-GPU jobs used less than 20% of memory capacity. Only 5% of jobs ever reached 75%, and among jobs running longer than eight hours, just 10% went past 90%. Many jobs hold an 80GB A100 and use something like 16GB of it.

Compute is emptier still. Half of all jobs sustained an average SM utilization below 5%. For 75% of jobs, peak memory usage and average SM usage both stayed under 20%. Only 10% of single-GPU jobs averaged above 50% SM utilization.

Averaging across the whole cluster instead of job by job gives the same picture: 29% GPU compute utilization and 16% GPU memory utilization. It is not only the GPUs, either. CPU utilization sat at 21% and host RAM usage at 17%. Resources across the board are reserved and empty.

Submitting a job to a shared cluster means writing down how long you think it will take, and the scheduler places it on the strength of that number. On this cluster the number was wrong about eight times out of ten. Only around 20% of requested durations landed within a factor of two of the actual runtime. The misses were not small ones either, since a substantial share were off by more than an order of magnitude.

The errors do not all lean one way. Of the jobs, 60.7% completed normally, while 34.2% used up their requested time without finishing and were killed, and the remaining 5.1% failed through user cancellations, errors, or host memory limits. Roughly one job in three asked for too little time and was cut off mid-work, and a large share of the rest asked for too much and tied up seats they never used.

How much was tied up and nothing more has been counted separately. The paper classifies a job whose GPU never ran for the whole of its execution as fully idle, and breaks those jobs out by duration.

The bucket holding the most jobs and the bucket holding the most time are not the same one. Fully idle jobs that ended in under a minute account for 14.1% of all jobs but only 0.02% of reserved time, and they look like jobs that died on a bad configuration. Fully idle jobs lasting more than eight hours are 0.6% of jobs yet took 2.15% of reserved GPU time. Fully idle jobs as a whole took 2.82% of reserved GPU time, and most of that comes from that last row alone.

For the scheduler this is not an algorithm problem. The input arriving is wrong, so no placement strategy can make the output better. That is why the authors' first proposal is to learn from past runs and predict runtimes automatically. Rather than asking users to write down better numbers, let the system estimate them.

The third finding has a different character from the first two. The multi-GPU jobs the team examined showed highly periodic communication patterns. The sample is small. Across the ten days, 343 jobs used more than one GPU and only 7 spanned more than one node. The authors set this down as a limitation.

The share those few take is not small, though. By count, 95% of jobs requested a single GPU, but by allocated GPU time, 26% went to parallel workloads. One job in twenty takes a quarter of the cluster's time. A small sample and a small share are different things here.

A neuroscience lab's four-GPU job showed large communication phases roughly every 17 minutes with smaller bursts spaced a minute apart. A psychology lab's four-GPU job ran 10-second and 60-second periods on top of each other. In both, the same shape repeated until the job finished.

The clearest case came from electrical engineering. Four nodes and sixteen GPUs ran for twelve hours, and throughout, the network saturated at close to 100Gb/s in repeating two-second windows. Every seven seconds, one second arrived in which the compute cores stopped entirely, the whole job apparently waiting on synchronization. The authors calculated that if other work could be packed into those troughs, around 21 hours of GPU computation could be recovered out of those twelve hours.

Regularity is the point. Knowing in advance when a gap will open means another job can be pushed into it. That is where the authors' second proposal comes from. Rather than buy more GPUs, share the ones already installed, by means Nvidia has already built.

• MIG — splits one GPU into several pieces at the hardware level and hands each piece to a different job. The isolation is strongest here
• MPS — runs kernels from several processes on one GPU at the same time. Isolation is weaker, but it is good at filling empty cores
• UVM — allows allocating more memory than physically exists, so jobs with memory to spare can be stacked together

An order of application is written down too. Single-GPU jobs come first, since they do not need several GPUs brought up at once. Jobs with steady memory usage come next, and jobs with predictable communication periods come last. For the slice of time, the paper suggests 100 to 200 ms, the same rhythm an operating system uses to divide a CPU and the same interval this paper measured with.

The remaining limits are ones the authors state themselves. DCGM caps out at ten samples a second, so kernel behavior at the microsecond scale stays invisible. The analysis window is ten clean days out of several months, and the multi-GPU sample is small. Because the two A100 models were analyzed together as percentages, the effect of the heterogeneous hardware is not separated out. Above all, this is one university research cluster. Nothing in the paper promises that a commercial data center's numbers look like these.

What makes this paper interesting is that it brought no new hardware and no new algorithm. One thing changed, the ruler. Shrinking the interval from 100 seconds to 100 ms turned the same cluster from a packed house into an empty one. Measurement produced the conclusion.

Anyone working on data quality knows this shape. Before asking whether a value is true or false, you have to ask what it was counting. A utilization of 92% may be perfectly correct. Whether it is the fraction of time a kernel ran or the fraction of cores that ran, whether it is a one-minute average or a 100 ms sample, decides what that identical number points at. Read a dashboard by the name of the metric and the difference stays hidden.

Before filing to add GPUs, or before raising a cloud GPU budget, three things are worth checking.

• Which tool produced the utilization we look at, and at what interval. A minute-level aggregate built on nvidia-smi may be reporting partial use as 100%
• Whether reserved resources and actually running resources are recorded apart from each other. Fold them into one number and seats that were merely held never surface
• Whether the runtime a user requested is kept next to the runtime that actually elapsed. Without that record, the route to fixing the scheduler through prediction is closed

The room in the middle this paper names is not won by swapping hardware or rewriting applications. It comes out of the layer that decides who gets which GPU, when, and how jobs are stacked on top of one another. To work on that layer, though, you first have to show what the layer has been missing, and on this cluster what made it visible was a finer ruler. Between adding resources and counting resources properly, the one needed first was the second. The scarcer GPUs become, the clearer that order gets.

Thank you for reading this far. The cluster configuration and the monitoring tool's specifications, the memory and SM utilization distributions, the mismatch between requested and actual durations, the fully idle job table, the three cases of periodic communication, and the proposals and limitations were all checked against paper 2610.36329, posted to arXiv on October 26, 2026. The tally that H100 compute on a one-year contract rose 60% in hourly cost over a year is a SemiAnalysis figure reported by the Korea Economic Daily on October 12, citing the Wall Street Journal. Section 6, which holds that the sampling interval should be asked about first, is this article's interpretation. If you run a cluster, we would be glad to hear at what interval you measure its utilization.

Pebblous Data Communication Team
October 30, 2026

• 1.Sheinberg, A., Gupta, S., Li, K. (2026). "Plenty of Room in the Middle: Revealing Opportunities on a GPU Cluster via Fine-Grained Monitoring." arXiv:2610.36329 (Oct 26, 2026).
• 2.als244 (Andrew Sheinberg). "gpu_server_monitoring." GitHub. The open-source repository for the lightweight DCGM monitoring tool used in the paper.
• 3.The Korea Economic Daily. (2026-10-12). "돈 있어도 못 구해…AI 열풍에 GPU 임대료 60% 껑충." In Korean. Citing the Wall Street Journal's report of SemiAnalysis's tally.