We have been sizing infrastructure for decades
Infrastructure sizing is not new.
For most of the last few decades, infrastructure sizing has been a reasonably mature discipline. It has never been perfect, and there has always been an element of judgement involved, but in most traditional workloads we have had a fairly good idea of what we are trying to measure and how those measurements translate into hardware.
Take SAP (one of the world’s leading enterprise resource planning (ERP) platforms) as an example. SAP developed SAPS, or SAP Application Performance Standard, as a benchmark that gives a standardised measure of how much SAP workload a system can process. Rather than simply saying that one server has more CPU cores or a faster processor than another, SAPS gives us a common performance reference that can be related back to transaction volumes and application demand.
That was useful because it gave customers, software vendors and infrastructure suppliers a more consistent language for sizing. CPU and memory requirements could be assessed against a known workload, and the storage side was not simply about how many terabytes were required. We could also look at IOPS, latency, throughput and the behaviour of the application under load. Over time, all of this became quite well understood.
Virtualisation is similar. If we are replacing or refreshing a VMware, Hyper-V or Nutanix environment, we normally begin with the environment that already exists. We can monitor CPU utilisation, memory consumption, storage capacity, IOPS, read/write ratios, latency and network throughput over a reasonable period of time. We can look at peaks, averages, growth and any existing bottlenecks, then apply headroom for resilience, future capacity and maintenance. The monitoring tools exist, the methodology is mature across hypervisors, and we now have decades of collective experience to draw on.
It does not mean the resulting design is mathematically perfect, but it is generally a fairly deterministic process. We have a known workload, we can measure it, and we can design around what we have observed.
That word, deterministic, is important.
In a traditional environment, if a workload consumed a certain amount of CPU, memory and storage I/O yesterday, there is a reasonable chance that under similar conditions it will behave in a broadly similar way tomorrow. There will be spikes, growth and exceptions, but the relationship between workload and infrastructure is usually understandable and repeatable enough that we can design around it.
AI is rather different.
When we start with AI, we are dealing with a much less deterministic sizing problem. We can still measure the infrastructure, but the workload driving it is far more variable. The same broad use case can behave very differently depending on the model, prompt, context length, concurrency, workflow and application design.
If somebody tells us they have 100 VMware virtual machines, that immediately gives us a useful starting point. If somebody tells us they want to run a private LLM for 50 users, it actually tells us surprisingly little.
What model are they planning to use? How large is it? What level of quantisation will they accept? How long will their prompts and context windows be? Are the users asking relatively simple questions, or are they working through large documents? Is this a straightforward private LLM, or will it include RAG, agents, document processing, vision or several different models working together?
Even the number of users can be misleading. Fifty occasional users asking short questions is very different from ten people running complex agentic workflows that may generate multiple model calls in the background. On paper, the second environment has fewer users. In practice, it may require considerably more infrastructure.
This is the non-deterministic part that makes AI sizing difficult. The relationship between “one user” and infrastructure demand is not fixed. One request may consume relatively little resource, while another may involve a long context window, several retrieval steps, multiple model calls and significantly more GPU memory and compute.
Two users can therefore generate completely different loads, even when they appear to be using the same application.
This is one of the reasons I am wary of simple statements such as “this GPU supports 50 users” or “this server will run this size model”. They can be useful as a very rough starting point, but they can also give a false sense of certainty.
The GPU itself is obviously important, but AI sizing cannot be reduced to a GPU specification. GPU memory, memory bandwidth, model size, context length and concurrency all influence performance, while CPU, system memory, storage and networking can become important depending on how the application is built. If the workload includes a large vector database, document ingestion, retrieval or several supporting services, we very quickly move beyond the idea that this is simply a GPU sizing exercise.
This is where I think our approach to infrastructure sizing needs to change.
Rather than trying to calculate the final answer entirely on paper, AI lends itself much more naturally to a process where theoretical sizing and practical testing work together. We can still begin with assumptions around model size, number of users, expected concurrency, context length and response time, but we should be prepared to validate those assumptions on real hardware.
That makes the proof of concept much more important.
Traditionally, a POC was often used to answer a fairly simple question: does the technology work? With AI, I think the POC also becomes part of the infrastructure sizing process.
We are not just trying to prove that a model will load and respond. We want to know how much GPU memory it consumes, how quickly it generates a response, what happens as the context grows, and how performance changes when several users begin working at the same time. Depending on the workload, we may also want to look at CPU utilisation, system memory, storage throughput and IOPS, particularly where RAG, document processing or data-heavy workflows are involved.
From there, the process becomes much more useful. We can start with a representative model and workload, run it on a known platform, record the results and then change one variable at a time. Increase the number of concurrent users. Try a larger model. Increase the context window. Add retrieval. Introduce an agent. At each step, we can see where the bottleneck begins to move.
That gives us something very important: evidence.
Instead of saying that we think a particular GPU or server should be enough, we can begin to show how the workload actually behaves. Once we have that baseline, the next sizing decision becomes much more grounded. It may point to a larger GPU, several GPUs in one server, multiple systems working together, or it may show that the original platform is already sufficient.
Over time, AI sizing will become more mature. We will develop better rules of thumb, vendors will publish more validated designs, software will provide better telemetry and we will get better at relating model size, tokens, concurrency and response time back to infrastructure.
I do not think we are quite there yet.
For IT managers looking at on-premises AI today, the important point is not that AI infrastructure cannot be sized. It is that the method needs to be more empirical. With SAP, virtualisation and traditional storage, we have spent years building the benchmarks, tooling and operational knowledge that allow us to size systems with a high degree of confidence. With AI, we are still building that knowledge.
For now, the most practical approach is to start with a sensible platform, run a representative workload, measure it properly and use those results to guide the next step.
The first AI system, therefore, does not always need to be the final infrastructure design. It can be the platform that gives us the evidence we need to design that final environment properly.
