What a Private AI Platform May Look Like. From Small Teams to Mid-Market Organisations
Private AI does not describe a single product or server configuration. It is a platform made up of compute, GPU resources, storage, networking, model-serving software, retrieval systems, identity, security, monitoring and operational controls.
The right design depends heavily on how the platform will be used. A small team experimenting with internal document search may need only a single GPU system. A larger organisation supporting hundreds of staff may require multiple inference nodes, shared storage, higher-speed networking, redundancy, workload separation and more formal governance.
This guide provides a practical vendor-agnostic view of how private AI platforms can scale from small departmental deployments through to shared enterprise environments supporting several hundred users. It is intended as a design reference rather than a fixed architecture. Actual sizing should be based on workload testing, model requirements, concurrency, data volumes, latency expectations and availability requirements.
1. What Is a Private AI Platform?
A private AI platform is an environment used to run AI workloads within infrastructure controlled by the organisation.
The platform may support:
- Local large language models
- Retrieval-Augmented Generation (RAG) – integration of your company documents
- Internal document search
- Summarisation and content generation
- Coding or engineering assistants
- Agentic workflows
- Vision and image-analysis workloads
- Department-specific AI applications
- Integration with internal systems and data sources
The key distinction is that the AI workload, organisational data and supporting services can remain within infrastructure selected and controlled by the organisation.
A private AI platform should therefore be considered as more than a GPU server. It is a combination of infrastructure and software services designed to provide secure, repeatable and supportable AI capability.
2. The Core Platform
A typical private AI environment can be viewed as several layers.
Users and Applications
Staff access the platform through web interfaces, business applications, APIs or specialist AI tools. API Access can also be for custom applications written within the organisation.
AI Services
These may include LLM inference, embedding services, RAG, agent frameworks, model routing and supporting application services.
Platform and Orchestration
Containers, virtual machines or Kubernetes may be used to deploy and manage workloads depending on scale and complexity.
Compute and GPU Infrastructure
GPU-enabled workstations, servers or dedicated AI platforms provide the processing capability required to run models.
Data and Storage
Storage is required for models, vector databases, source documents, application data, logs and backups.
Networking
The network connects users, AI services, storage and compute resources. Higher-performance environments may require 25GbE, 100GbE or faster interconnects.
Security, Identity and Governance
Authentication, authorisation, logging, segmentation, data controls and human approval processes should span the entire platform.
Typical Infrastructure Progression
3. Small Deployment — Individual or Small Team
A small private AI environment may support approximately 1–10 active users.
Typical use cases include:
- AI experimentation
- Internal document search
- Coding assistants
- Small departmental knowledge bases
- Model evaluation
- Proof-of-concept agent workflows
A simple architecture may consist of:
- One GPU workstation or compact GPU server
- Local NVMe storage
- One or more local models
- Containerised inference services
- Basic RAG capability
- Browser-based user interface
- Integration with Active Directory or Entra ID where required
- Hosting and facilities: Requirements are generally modest. A workstation or compact server may be installed in an existing office or server environment, provided suitable power, cooling, noise and physical security are available
At this scale, simplicity is generally more valuable than redundancy.
The main objective is to validate the use case and understand actual performance before investing in a larger platform. We internally often use this as a first stage for sizing unknown or new workloads.
4. Departmental Platform — Approximately 10–50 Users
As usage grows, the design typically begins to separate the AI platform from individual users and applications.
A departmental platform may include:
- One or two dedicated GPU servers
- Larger GPU memory capacity
- Centralised model hosting
- Shared storage, potentially reusing existing SAN or NAS infrastructure where appropriate
- Centralised RAG services
- Model and application containers
- User authentication and access controls
- GPU and system monitoring
- Backup of configuration and business data
- Hosting and facilities: Rack space, electrical load, cooling capacity and UPS provision should be reviewed. In many cases, an existing server room or data centre can accommodate the platform without significant changes
This environment may support several models simultaneously, for example:
- General-purpose LLM
- Coding model
- Embedding model
- Specialist departmental model
At this level, concurrency becomes an important design consideration rather than pure grunt.
A GPU that performs very well for a single user may behave very differently when 20 or 30 users submit requests at the same time.
5. Shared Mid-Market Platform — Approximately 50–200 Users
A private AI environment serving a wider organisation usually requires more formal platform design.
A typical architecture may include:
- Multiple GPU compute nodes
- Shared high-performance storage – potentially dedicated
- 25GbE or 100GbE networking
- Centralised inference services
- Model routing
- RAG and vector database services
- Identity integration
- Role-based access
- Monitoring and logging
- Backup and recovery
- Development, test and production separation
- Defined software and model update processes
- Hosting and facilities: Power and cooling become a more significant design consideration as GPU density increases. Existing rack capacity, UPS load, electrical distribution and cooling should be validated before deployment.
Workloads may also begin to diverge.
For example:
- General staff may use a shared AI assistant
- Developers may use a coding model
- Finance may use a restricted RAG dataset
- Engineering may require a specialist model
- Automated agents may consume AI resources independently of human users
This is where capacity planning should move away from simple user-count assumptions.
Important measurements include:
- Time to first token
- Tokens per second
- Concurrent sessions
- Model memory usage
- GPU utilisation
- Queue depth
- Storage performance
- Network utilisation
- Power consumption
6. Larger Mid-Market Platform — Approximately 200–500 Users
A platform supporting several hundred users should normally be treated as shared infrastructure rather than as a single AI server.
The architecture may include:
- Multiple inference nodes
- Workload or model separation
- Multiple GPU types or node sizes
- Load balancing
- Model routing
- Shared enterprise storage
- Redundant networking
- High-speed east-west connectivity
- Central identity and access control
- Audit logging
- Observability and alerting
- Backup and recovery
- Production change control
- Security and governance policies
- Hosting and facilities: Facilities planning becomes a critical part of the platform design. Higher-density GPU systems can place substantial demands on rack power, UPS capacity and cooling, and existing server rooms may require upgrades or alternative data-centre capacity.
The platform may also use different infrastructure pools.
For example:
General AI Pool
Runs smaller, efficient models for broad employee use.
High-Performance Pool
Runs larger models or workloads requiring more GPU memory.
Specialist AI Pool
Hosts department-specific models or sensitive workloads.
Agent / Automation Pool
Supports workloads that may generate high request volumes without direct user interaction.
This approach can be significantly more efficient than attempting to run every workload on the largest available GPU.
7. Why User Count Alone Is Not Enough
It is tempting to size AI infrastructure based on employee numbers.
For example:
100 users = one GPU
500 users = four GPUs
In practice, this is rarely reliable.
AI infrastructure sizing is affected by:
- Model size
- Quantisation
- Context length
- Prompt size
- Output length
- Number of concurrent users
- Batch size
- Agent activity
- RAG workload
- Vision or multimodal processing
- Required response time
- Availability targets
Two organisations with 500 employees may therefore require very different infrastructure.
One may have 20 active AI users during the day.
Another may have automated agents generating thousands of requests.
The correct approach is to establish an initial architecture and then validate it using representative workloads.
8. Design for Growth
Private AI platforms should ideally scale incrementally.
A practical approach is:
Stage 1 — Prove the Workload
Run representative models and applications on a small platform.
Stage 2 — Measure
Capture latency, throughput, concurrency, GPU utilisation, memory usage, storage and power.
Stage 3 — Size
Determine the infrastructure required for expected production usage.
Stage 4 — Deploy
Implement the production platform with appropriate security, identity and operational controls.
Stage 5 — Optimise
Monitor real usage and add compute capacity or change models as requirements develop.
This avoids both under-sizing and unnecessary capital expenditure.
9. Key Design Principle
A private AI platform should not be designed by starting with a GPU model.
The design should begin with:
What workloads need to run, who needs to use them, what data is involved, what performance is required and how important the service is to the organisation.
The infrastructure can then be designed around those requirements.
For many organisations, the final environment will include a combination of smaller models, larger models, RAG, specialist services and different tiers of GPU infrastructure.
The objective is not simply to run an AI model.
It is to create a private AI platform that is secure, measurable, supportable and able to grow with the organisation.
