Why AI Infrastructure Matters
Artificial intelligence is moving quickly from experimentation into practical business use. For many mid-market organisations, the first experience of AI has been through cloud services such as Microsoft Copilot, ChatGPT or other SaaS platforms. As use cases become more specific, however, organisations are increasingly looking at where workloads should actually run and whether some AI capability should sit within their own infrastructure.
Mid-market organisations, typically around 100 to 1,000 users in Australia, occupy an interesting position when it comes to AI infrastructure. Smaller businesses will often consume AI almost entirely as a cloud service, while large enterprises may have dedicated AI, data science, infrastructure and facilities teams to design and operate their own platforms.
Mid-market organisations often have significant data, existing on-premises infrastructure and a genuine requirement to run some AI workloads locally, but with relatively small IT teams responsible for the entire environment. This makes getting the infrastructure design right particularly important. The platform needs to be capable, supportable and scalable without becoming unnecessarily complex.
There are a number of reasons to consider on-premises AI. These may include data privacy, sovereignty, predictable operating costs, performance, integration with local applications or the need to process large datasets without continually moving them into and out of the cloud.
This does not mean that every AI workload should be brought on-premises. In many cases, the right answer will be a mixture of cloud and local infrastructure. The important point is that AI workloads introduce different infrastructure requirements, and these need to be understood before selecting hardware.
An AI platform is more than simply adding a GPU to a server. Compute, GPU memory, storage, networking, power, cooling, backup and operational support all need to be considered together.
Start With the Workload
The most important part of designing an AI environment is understanding what it will actually be used for.
We have decades of experience sizing infrastructure for traditional workloads such as databases, ERP systems and websites. There is a huge amount of documentation, benchmarking and proven guidance around how to size these applications based on user numbers, transaction volumes and application complexity. Some experienced infrastructure architects can almost size these environments in their sleep.
AI workloads are different.
At one end of the scale, you may be running a simple LLM query and returning a few lines of text. At the other, you may be performing multi-step agent orchestration, analysing high-resolution images or processing hundreds of video feeds in real time.
The infrastructure required for these workloads can vary enormously, and in many cases the real requirements are not fully understood until the workload has been tested.
AI infrastructure requirements can vary significantly depending on whether the organisation is running an internal chatbot, document search platform, vision application, AI agent, local language model or model training workload.
Some of the initial questions should include:
- What applications or models will be run?
- Is the workload primarily inference, training or fine-tuning?
- How many users will access the system?
- How many users are likely to be active at the same time?
- What performance or response time is expected?
- What data will the system need to access?
- How quickly is the workload expected to grow?
It is easy to start by selecting a GPU and working backwards. In most cases it is better to understand the workload first and then size the infrastructure around it.
Compute, GPUs and Memory
GPUs have become the main compute platform for modern AI because they can perform the large number of parallel mathematical operations required by AI models much more efficiently than conventional CPUs.
However, GPU selection is not simply about processing performance. GPU memory, usually referred to as VRAM, is often equally important.
The size of the model being used, the numerical precision it operates at and the number of concurrent requests all influence the amount of GPU memory required. Techniques such as quantisation can reduce memory requirements and allow larger models to run on smaller GPUs, although this can involve trade-offs.
For smaller workloads, a single GPU workstation or server may be sufficient. Larger environments may require multiple GPUs within a server or several GPU servers working together.
There is also an increasing range of systems using unified or shared memory architectures. These can offer large amounts of accessible memory at relatively low cost and may suit development, experimentation and some production workloads. Traditional GPU servers, however, generally provide greater expansion capability, hardware redundancy, remote management and scalability.
The right platform therefore depends on both the application and the operational expectations of the business.
The Software Platform
The physical infrastructure is only one part of an AI environment. Organisations also need to consider how applications will access and share the available GPU resources.
Smaller environments may run workloads directly on a workstation or server, while larger platforms will commonly use virtualisation, containers or orchestration platforms to manage multiple applications and users. The choice can affect GPU utilisation, scalability, security and how easily new AI workloads can be introduced.
As with the hardware, there is no single architecture that suits every organisation. The software platform should reflect the workloads being run and the skills available to operate it.
Storage and Networking
AI can also change the way storage and networking infrastructure is used.
Language models themselves may range from a few gigabytes to hundreds of gigabytes, while AI applications can also operate against large collections of documents, images, video or other business data.
Fast local NVMe storage is commonly used within AI servers, particularly where models or datasets need to be loaded quickly. Existing SAN, NAS or object storage platforms may also continue to play an important role, particularly for shared datasets, long-term storage and backup.
As implementations scale, traditional storage architectures can become a bottleneck. Larger AI environments may therefore require highly parallel storage platforms, often software-defined, to deliver the throughput and concurrent access needed by multiple GPUs, servers and workloads.
Networking requirements depend heavily on scale. A small inference server may operate perfectly well on existing 10 or 25Gb Ethernet infrastructure. Larger multi-server AI environments can generate considerably more network traffic, particularly when GPUs, storage and compute nodes need to exchange large amounts of data.
For larger GPU platforms, 100Gb Ethernet and above is becoming increasingly common, with higher-speed Ethernet and InfiniBand used as environments scale further. This does not mean every AI deployment requires a new high-speed network; the requirement should be driven by the workload, data movement and architecture.
The objective should be to understand where the data resides, how it moves and whether the existing network is appropriate before introducing new infrastructure unnecessarily.
Power, Cooling and the Data Centre
One of the biggest differences between conventional infrastructure and AI infrastructure is often found outside the server itself.
Modern GPU servers can consume significantly more power than traditional virtualisation hosts. A server that physically fits into a rack does not necessarily fit within the rack’s available power and cooling capacity.
Before deploying higher-density AI systems, organisations should consider:
- Available rack power
- UPS capacity
- PDU and outlet capacity
- Power redundancy
- Cooling capacity
- Rack space and physical installation requirements
- The effect of additional load on the wider server room
This becomes particularly important in smaller data centres and server rooms where the original facility may have been designed around relatively modest server densities. Over the last decade, many mid-market server rooms have operated comfortably with rack loads of only a few kilowatts. A modern GPU server can consume 2kW or more on its own, and placing several of these systems into a rack can quickly change the power and cooling requirements of the entire room.
For many mid-market customers, reviewing the physical environment is therefore just as important as selecting the GPU.
Security, Availability and Operations
AI infrastructure still needs to be managed as production IT infrastructure.
On-premises AI can provide greater control over where organisational data is stored and processed, but running a workload locally does not automatically make it secure. Authentication, network segmentation, access control, logging, patching and software supply chain management remain important.
The operational model should also match the importance of the workload.
Some AI workloads may require redundant servers, storage and power. Others may be perfectly capable of restarting a job or moving a workload to another system if hardware fails. Not every AI environment requires the same high-availability architecture used for critical transactional systems.
Monitoring should include the underlying servers as well as GPU utilisation, temperature, memory usage and workload performance. Organisations should also consider how models, configurations and supporting data will be backed up and recovered.
Cloud, On-Premises or Hybrid?
There is no single correct location for AI workloads.
Cloud AI is well suited to experimentation, rapidly changing requirements and workloads that occasionally require very large amounts of compute. On-premises infrastructure can make sense where workloads are predictable, data needs to remain local, latency is important or the organisation wants greater control over cost and infrastructure.
In practice, many organisations will use both.
The objective should not be to move AI on-premises for its own sake. It should be to determine which workloads make sense locally and which are better consumed as cloud services.
Where to Start
For most mid-market organisations, the best approach is to start with a defined use case rather than attempting to build an enterprise-wide AI platform immediately.
The first step is to identify the workload, users, models and data involved. From there it is possible to estimate GPU, memory, storage and network requirements and then assess whether the existing data centre or server room can support the proposed platform.
This is where an infrastructure assessment becomes particularly useful. We look at the AI workload alongside the existing compute, storage, networking, rack, power and cooling environment rather than treating each area independently.
The result is usually a much clearer view of what can be reused, what needs to change and what level of infrastructure is actually required.
For many organisations, the initial AI platform may be relatively modest. The important part is to design it with enough understanding of the workload and the underlying infrastructure that it can be expanded when there is a genuine requirement to do so.
