Last quarter, the cheapest option won
A procurement evaluation compared three GPU configurations for an inference workload. Configuration A had the highest throughput — and the highest price. Configuration B had moderate throughput at a moderate price. Configuration C had the lowest per-unit cost and reasonable throughput. The committee chose C. It scored well on “performance per dollar.”
Eighteen months and several operational surprises later, the team calculated total cost of ownership. Configuration C’s power draw per unit of sustained throughput was materially higher than B’s. Its thermal characteristics in the target rack density required additional cooling investment. Maintenance costs were higher. Effective throughput under production conditions — at sustained load, with thermal settling — diverged from the evaluation benchmark by a wider margin than the other options.
Configuration B, the moderate option, would have delivered lower total cost over the deployment horizon. The procurement evaluation captured acquisition cost and peak throughput. It missed the rest.
That pattern is common enough to be worth naming precisely. Cost, efficiency, and value are three separate measurements that happen to share a currency symbol, and they routinely rank the same hardware in three different orders. What follows is not a replacement for an organisation’s financial evaluation process — it is a way of making sure the right dimensions enter the decision before the spreadsheet gets locked.
Three distinct dimensions, routinely conflated
When people say “cost-effective infrastructure,” they could mean any of three things:
Cost: direct financial expenditure — acquisition price, cloud instance price, power, cooling, floor space, maintenance contracts, staffing. Cost answers: how much money does this require?
Efficiency: the ratio of useful output to resource consumed — throughput per device, tokens per watt, inferences per dollar-hour. Efficiency answers: how much work do we get per unit of resource?
Value: the business outcome delivered per total investment — SLA achievement, time-to-model, competitive capability, risk reduction. Value answers: was the money well spent relative to what the organisation needed?
You can minimise cost by buying the cheapest hardware and destroy efficiency in the process, if that hardware is power-hungry and underperforms. You can maximise efficiency by buying the best throughput-per-watt device and still miss on value, if it cannot run the target workload at the required SLA. And you can optimise for value — infrastructure that serves the business need exactly — and find it is neither the cheapest nor the most efficient option on the shortlist. Each dimension needs its own measurement, because each produces a different ranking of the same candidate list.
Performance per dollar is context-dependent
“Performance per dollar” is the most commonly cited efficiency metric in hardware evaluation, and among the most misleading when applied naively.
The numerator depends entirely on what was measured. Peak throughput, sustained throughput, throughput at a target latency, throughput at a target precision — each produces a different number for the same card. A device with excellent peak throughput per dollar may have mediocre sustained throughput per dollar if it throttles under continuous load.
There is a second, quieter ambiguity in that numerator. Whatever sits above the division line is not a card; it is an AI Executor — a device plus the software stack driving it. Two physically identical GPUs on different runtimes, driver versions, or graph compilers are two different cost cases, because they do different amounts of work for the same purchase price. A CUDA-and-TensorRT path and a stock PyTorch eager path on the same silicon will not divide into the same dollar figure. If the numerator does not name the stack, the ratio is not reproducible.
The denominator varies just as much. Acquisition cost only? Acquisition plus three years of power? Acquisition plus power plus cooling plus maintenance plus the staffing to keep the thing running? Each scope produces a different cost-per-performance ranking.
So performance per dollar is not a metric. It is a family of metrics, and which member of the family matters depends on deployment duration, cost structure, and the performance dimension the workload actually demands. As explored in how hardware evaluation should match deployment reality, the evaluation frame has to reflect real operating conditions. A ratio that omits power for a three-year deployment in a power-constrained facility is measuring the wrong thing confidently.
Split the denominator by the work being paid for
One structural fix costs nothing and removes a lot of confusion: report Training, Inference, and Compute separately rather than collapsing them into one undifferentiated rating.
An inference-serving budget and a fine-tuning budget do not share a denominator. Serving is a latency-and-concurrency problem billed continuously; fine-tuning is a burst problem billed against a deadline. A single blended score divided by a single price yields a figure that describes neither. When the three classes of work are kept apart, spend can be divided by the work being paid for — and the two budgets can move independently, which is how they behave in reality anyway.
The same logic explains why CPU belongs on the same scale as every accelerator rather than being assumed out of the comparison. If the CPU and the accelerator are scored under the same conditions, then “is the GPU worth its cost on this workload at all” becomes a question with a measured answer. We see teams skip that check routinely, and it is the cheapest one available: on small-batch, latency-bound, or heavily pre/post-processing-dominated pipelines, the accelerator’s margin over the host CPU is sometimes thin enough that the accelerator’s cost is not earned.
Power and operational costs matter over time
For short-term deployments or cloud burst capacity, acquisition cost dominates. For owned infrastructure running three to five years, operational cost — primarily power and cooling — often exceeds it.
A device drawing 700W versus one drawing 400W produces 2,400 watts of difference across an eight-device node. Over three years of continuous operation at $0.10/kWh, that is roughly $63,000 in power cost per node. Across a hundred nodes the differential runs to millions, dwarfing any plausible acquisition-price gap between the two options. (Arithmetic on stated wattages and a stated tariff — not a measurement of either device.)
The arithmetic is trivial. It is routinely excluded anyway, because most benchmarks measure throughput and not power. The result is a ranking that reflects one cost dimension — compute throughput per acquisition dollar — while ignoring another that is larger over the horizon.
Value emerges from sustained, usable performance
Performance the organisation can actually use is worth more than performance that exists on paper.
A device that benchmarks at 1,500 tokens/second but needs software optimisations the team cannot deploy — framework incompatibility, deployment constraints, an expertise gap — delivers nothing from those 1,500 tokens. A device that benchmarks at 1,000 tokens/second and works with the existing stack delivers 1,000 tokens/second of real value. Likewise, a system with high throughput that cannot hold P99 latency under production traffic fails the value test regardless of its efficiency numbers.
This is where the word sustained has to be defined rather than gestured at, because it is the measured half of the sentence. Sustained means: a warm-up phase is discarded, completed iterations are counted inside a continuous timed measurement window, and the workload has already been scaled until throughput stops improving. What the machine holds under load, in other words, rather than its best instant. That definition is narrow on purpose — and it is also the boundary. A 15–30 minute measurement window says what happened inside that window. It does not license a claim about behaviour over months of duty cycle; that part is modelling, and it belongs to the buyer.
Two systems can post nearly identical performance-per-dollar on a spec sheet and diverge sharply in practice, because the paper number freezes a single operating point while deployments live across many. Utilisation is the first divergence: a configuration averaging 40% utilisation wastes most of the dollars its ratio assumed. Power draw at the sustained operating point is the second. Lifecycle is the third — driver and framework support windows, resale or redeployment value, and the headroom to absorb a workload shift. None of these appear in the headline ratio, which is why two options that look tied on cost efficiency rarely stay tied once they run.
Which half of the cost equation can a benchmark actually supply?
A measured result supplies exactly one side, and supplies it honestly: sustained, workload-bound performance for a named device and stack, split by class of work, under a stated release name. Everything else comes from the buyer.
| Input | Comes from | Why |
|---|---|---|
| Sustained throughput per Executor, by work class | Benchmark result (named release) | Measured inside a timed window at a saturating workload; device plus stack identified |
| CPU baseline on the same scale | Benchmark result | Makes “does the accelerator earn its cost here” answerable rather than assumed |
| Purchase or instance price | Buyer | Varies by vendor, contract, region, timing |
| Power tariff and cooling overhead | Buyer | Site-specific; often the largest multi-year term |
| Duty cycle and expected utilisation | Buyer | Determines whether an efficiency advantage is ever realised |
| Depreciation and support horizon | Buyer | Finance policy, not a hardware property |
Read the left column as two blocks: the top two rows are measurements of a machine, the bottom four are facts about a business. A score is not a price, a cost, or an efficiency figure — it measures performance, and every cost term in the ratio comes from the reader’s own numbers. Nor is an aggregate rating a physical quantity that can be divided by a price to produce a value index; it is an ordinal aggregate with no reference-device normalisation, so treat it as an ordering, not a magnitude. One further discipline: a figure carrying one release name does not belong in the same cost model as a figure carrying another.
The cost of getting the measured half is close to negligible, which removes the usual excuse for skipping it: 15–30 minutes on a machine already in hand, roughly 15 GB of disk, and a one-time 2 GB model download.
Cost efficiency, cost effectiveness, and how to align them with the decision
The two phrases get used interchangeably and should not be. Cost efficiency asks whether you are getting the most output per unit of spend. Cost effectiveness asks whether the spend achieved the objective. A configuration can be highly cost-efficient and entirely ineffective — excellent throughput per dollar on a workload that misses its latency target is a well-priced failure.
The practical remedy is to declare, before evaluation begins, which dimensions matter for this decision and how they are weighted:
| Decision type | Primary metric | Secondary metric | What to watch for |
|---|---|---|---|
| Cost-constrained, flexible SLAs | Acquisition + operational cost per unit of sustained throughput | Efficiency floor (minimum acceptable throughput/watt) | Hidden operational costs — power, cooling, maintenance — that shift the ranking over the horizon |
| Latency-critical production | P99 latency at target request rate, thermally settled | Cost ceiling (maximum acceptable $/request) | Throughput figures that look strong in a benchmark but mask tail-latency failures under production traffic |
| Long-lived infrastructure investment | Total cost of ownership over the deployment horizon | Workload evolution headroom | Optimising for today’s workload at the expense of flexibility across three to five years |
Each framing produces a different methodology, a different metric set, and potentially a different recommendation. Making the weighting explicit is the point; leaving it implicit in the choice of benchmark is how a decision ends up defensible only in retrospect.
That connects to the broader practice of treating benchmarks as traceable evidence in institutional decisions. As explored in how benchmarks function in governance and risk management, the metrics admitted to an evaluation are never neutral — they encode assumptions about what matters. Where these three dimensions meet an actual budget line, they become the production cost of inference, the applied primer that puts cost, efficiency, and value on one page.
A last caution about scope. A published result covers the fixed catalogue of one named release on one Executor; it has not priced anyone’s workload. If the workload in question is not in that catalogue, the result is a reference point for the shape of the answer, not the answer.
LynxBenchAI reports performance under declared conditions — sustained, per precision, per class of work, with the device and stack named — so that the measured half of a cost model can be compared like for like rather than assembled from incompatible sources. The free Personal Edition is what has shipped; the other editions are contact-gated. Which leaves the question a buyer can answer and a benchmark cannot: over your duty cycle, at your tariff, does the accelerator earn its cost on the work you actually run?
Frequently Asked Questions
Why are performance, cost efficiency, and business value three different metrics that get conflated in AI hardware decisions?
Cost measures direct expenditure — acquisition, power, cooling, maintenance, staffing. Efficiency measures useful output per unit of resource consumed. Value measures the business outcome delivered per total investment: SLA achievement, time-to-model, risk reduction. They get conflated because each can be written as a ratio with money in it, but they rank the same shortlist differently — the cheapest option is often not the most efficient, and the most efficient is often not the most valuable.
Why is performance-per-dollar always context-dependent rather than a universal score?
Both halves of the ratio are ambiguous. “Performance” can mean peak throughput, sustained throughput, throughput at a target latency, or throughput at a given precision — each a different number for the same card. “Dollar” can mean acquisition only, acquisition plus power, or full operational cost over the horizon. The right member of that family depends on deployment duration, cost structure, and the performance dimension the workload demands.
Why does “value” in AI infrastructure emerge from sustained, usable performance rather than from any headline number?
Headline numbers describe best-case conditions that often do not survive contact with the team’s stack, the workload’s tail-latency requirement, or the rack’s thermal envelope. Value is the work the infrastructure actually does over its life — usable performance, met SLAs, manageable operating cost, headroom for change. It does not reduce to one ratio, which is exactly why it is the dimension most often omitted and the one most likely to decide whether the money was well spent.
Which part of a total-cost-of-ownership picture can a benchmark result actually supply, and which numbers have to come from the buyer’s own situation?
A result supplies sustained, workload-bound performance for a named device and stack, split by class of work, under a stated release — plus a CPU baseline on the same scale. Price, power tariff, cooling overhead, duty cycle, and depreciation come from the buyer. The measurement is one side of the equation, honestly bounded to its timed window; the cost side is entirely the reader’s own numbers.
What is the difference between cost efficiency and cost effectiveness, and why does that distinction change which AI hardware you buy?
Cost efficiency asks whether you are getting maximum output per unit of spend. Cost effectiveness asks whether the spend achieved the objective. A system can be highly efficient and wholly ineffective — best-in-class throughput per dollar on a configuration that misses its P99 target is a well-priced failure. Efficiency picks among options that already clear the requirement; it cannot substitute for clearing it.
If the CPU and the accelerator are scored on the same scale, how do you read the pair of numbers to decide whether the accelerator earns its cost on this particular workload?
Read the gap, then price it. Take the sustained-throughput ratio between accelerator and CPU baseline on the specific class of work, and compare it against the ratio of their fully scoped costs over your duty cycle. If the performance gap is smaller than the cost gap, the accelerator has not earned its place on that workload. This tends to matter most on small-batch, latency-bound, or preprocessing-dominated pipelines.
Why can the same physical card produce two different cost cases depending on the runtime and stack driving it?
Because the numerator of a per-dollar figure is an AI Executor — a device plus the software driving it — not the silicon alone. Runtime, driver version, graph compilation, and precision handling change how much work the same card completes for the same purchase price. Two identical cards on different stacks therefore divide into different figures, which is why a ratio that does not name the stack is not reproducible.
