Back to Resources
Blog

Who Ordered the GPU?

Mike Langdon·November 17, 2026

The Kubecost Unallocated cost dashboard in Kubecost for Grafana. A row of stat panels shows an unallocated share of 9.8%, $95.07 with no team label, $65.82 with no owning controller, $128.70 in pods with no resource requests and $0.00 in unmounted volumes. Below, an Unallocated cost over time chart plots the no team label and no owning controller costs from 11/07 to 11/13, between about $9 and $15 a day.

After three weeks of late nights, the words "death march" leaving the realm of jokes, and more knocking on wood than usual, you've finally shipped the release. The whole team is huddled around a back room table at the steakhouse down the street, sneaking glances for notifications the whole first hour, but by 6:30, everyone is breathing easier. The code works! Someone orders appetizers for the table. Someone else orders a third round before the second one's gone. Your tech lead tells the story about the time staging went down during a customer demo, and this time everybody laughs. Only the SRE checks their phone again.

Dana from QA has brought a plus-one, a friend who's in town for the week. The friend fits right in. They ask good questions about the release and laugh at the right parts of the staging story. At some point they lean over to the server and say, "Something nice for the table." A bottle shows up, and glasses go around. The SRE makes a toast to zero rollbacks. Around eight, the friend ducks out for an early flight.

Then the check comes. It's one check for the whole table. You can account for most of it: the barely touched appetizers, the rounds, the fries nobody finished. And then there's the $300 Napa Valley Cabernet blend. Nobody at the table ordered it. The person who did is on their way to the airport.

Your Kubernetes bill has the same line. On a GPU cluster, the most expensive pods are usually training jobs. Kubecost puts a lot of those jobs in a bucket called __unallocated__. They have an owner. But it's not a kind of owner Kubecost understands.



It's not a new problem

Kubecost created OpenCost, an open-source CNCF project, and the two share most of the code that decides who owns each pod's cost. People have filed the same complaint against OpenCost for four years running. In June 2022, someone reported missing data when aggregating by deployment and controller. In February 2024, it was __unallocated__ resources for Deployments. In August 2025, someone filed an issue with nearly the same title as the 2022 one. In March 2026, deployment-level breakdowns showed costs "only as unallocated or idle". The last two are still open.

The July 2026 version gets specific about where the money is:

"On GPU clusters, the most expensive pods are usually owned by workload CRDs: PyTorchJob (Kubeflow), RayCluster/RayJob (KubeRay), JobSet, or vendor operators."

Those pods "get no controller attribution and aggregate as __unallocated__." According to the issue, the allocation code queries "a fixed set of kinds, plus two hand-written special cases."

Let's go easy on the maintainers. Each new operator defines its own custom resource, such as PyTorchJob or RayJob, and someone has to write code before Kubecost can support it via OpenCost. A RayJob doesn't even own its pods directly. It owns a RayCluster, which owns the pods. The reporter's proposal is to make the list of custom resources OpenCost supports a matter of configuration, so that the next operator doesn't take a code change. That's a reasonable fix, but it'll take time.

Kubernetes already records which custom resource owns each pod, though. kube-state-metrics publishes a kube_pod_owner metric with an owner_kind label for every pod, and OpenCost's own metric emitter writes the same metric. If you scrape either one, the answer to "who ordered the GPU?" is sitting in your metrics today.

Idle and shared aren't the same problem

Two other buckets sit near unallocated cost on most cost dashboards, and they're easy to confuse with it.

Idle is capacity you paid for that no pod requested or used.

Shared is the appetizers for the table: kube-system, your monitoring stack and other services that every team leans on. Cost tools usually spread it across the other namespaces in proportion to what they spend.

Both have an owner by agreement. You decide where idle goes, and you decide how shared cost gets split. Everybody agreed to split the appetizers. Nobody agreed to the wine. Unallocated cost has no owner until somebody fixes the workload.

Two ways a cost ends up owned by nobody

Cost allocation works when every pod answers two questions: which team it belongs to, and what owns it. A pod that can't answer one of them drops a cost on the table that nobody picks up.

No team label. Showback, the report that tells each team what it spent, groups cost by a label, usually team. A workload without that label has nowhere to go in a cost-by-team report.

No owning controller. Bare pods have no owner at all. Pods owned by a custom resource have an owner Kubecost doesn't support. That's where a PyTorchJob's pods end up.

Missing resource requests cause a different problem. Kubecost charges each pod for whichever is larger: what it requested or what it used. A pod that requests nothing is charged only for what it happened to use. Nothing reserves capacity for it, and its cost looks smaller than the room it takes.

All three problems come down to how workloads get deployed. Make some friends in finance: put each fix in a manifest or a Helm values file.

Seeing it in Grafana

Integration Plumbers built Kubecost for Grafana to read Kubecost's API directly, and we gave unallocated cost a dashboard of its own. The Kubecost Unallocated cost dashboard gives each ownership gap its own panel: "No team label" and "No owning controller". Next to them, it totals the pods without resource requests.

Below the totals, each cause is broken down by namespace, and a table lists the pods without resource requests, most expensive first, with the CPU and memory they used.

The lower half of the Kubecost Unallocated cost dashboard. Two bar gauges break unallocated cost down by namespace: No team label, led by monitoring at $33.77 and kube-system at $25.09, and No owning controller, led by shop at $8.60 and analytics at $7.12. Below them, a Workloads without resource requests table lists pods by cost, from nightly-rollup-lb-h95 in analytics at $38.25, using 0.821 CPU cores and 4.41 GiB of memory, down to metrics-agent-83-u6k in kube-system at $4.43.

The plugin doesn't fix attribution. Kubecost still can't tell you which team a RayJob belongs to. What the dashboard gives you is the list of things to fix, sorted by cost, in the Grafana you already use. It also includes an alert rule, "Unallocated share high," that fires when more than 25% of a cluster's cost has no team label.

Everyone gets their own tab

The fix for the dinner check isn't a better way to split it. The fix is for the server to open a tab for each person when they sit down, so the wine lands on the plus-one's tab when it's ordered.

The cluster works the same way. Before a job runs, give it a team label and resource requests. Even when Kubecost doesn't know the job's owner, the team label puts its cost on the right tab in a cost-by-team report.


Is unallocated cost the same as idle cost?+

No. Idle is capacity you paid for that no pod requested or used, and you decide where it goes. Unallocated cost comes from pods that did run, but that Kubecost can't attach to a team label or an owning controller. It has no owner until somebody fixes the workload.

Why do Kubeflow and Ray jobs show up as unallocated?+

Their pods are owned by custom resources, such as a PyTorchJob or a RayCluster. The allocation code Kubecost shares with OpenCost resolves a fixed set of owner kinds, so those pods get no owning controller. A team label on the job still puts its cost on the right team in a cost-by-team report.

Can the dashboard use a label other than team?+

Yes. The Unallocated cost dashboard has a Label key selector. The "Unallocated share high" alert rule reads the team label, and the comments in its rule file say where to change it.

Does the plugin install anything in the cluster?+

No. It reads from Kubecost's API. Nothing is installed in the cluster.


Try it

Kubecost for Grafana is in the Grafana plugin catalog. Install it and open the Unallocated cost dashboard to see who ordered the GPU.

Not Sure Where to Start?

Take our free OTEL Maturity Assessment to identify gaps and get a personalized action plan.

Take the Free Assessment