A server you control
An x86 CPU server, physical or in your own cloud account. No GPU. Our measured server had 22 vCPUs and 86.4 GB of RAM, and the whole stack peaked at 23.7 GB. We size the server to your documents and users during scoping.
On-premise LLM
Run an LLM on-premise, on CPU servers you already own.
Most guides to on-premise LLMs start with buying GPUs. BAI runs a private LLM and document search on standard CPU servers, so documents, questions and answers never leave your network. This page covers what you need, what it costs to run, and where the limits are today.
Requirements
What an on-premise LLM deployment needs.
Four things: a server, a model, your documents, and a network boundary you already control.
An open-weight model
Models in GGUF format, run locally through llama.cpp. The standard configuration is Llama 3.2 3B Instruct, 4-bit quantised, with bge-small-en-v1.5 for search. You can change the model; larger models need more memory and answer more slowly on a CPU.
Your documents
PDFs, scans, images, recordings, video, Word, PowerPoint and web pages. Each is extracted on the server and indexed for keyword and semantic search, and every answer returns the passages it used.
No outside AI service
Documents, questions and answers are processed on your server. Models are copied in from local media or your own network, so answering needs no internet connection. Web URLs are the one input that fetches from the internet.
Measured, no GPU
How an LLM performs on a CPU server.
Figures from one Google Cloud c3-standard-22: Intel Xeon Platinum 8481C @ 2.70 GHz (x86-64), no GPU, $1.17/hour, on-demand list price. Measured 29–30 October 2026; they apply to that server only.
- Median time to a full answer
- 9.3 s
- 1 user, 93 questions
- Search latency, 1 user
- 9.2 ms
- Median, top 5 passages. p95 15.7 ms.
- Compute cost per answer
- $0.0035
- 1 user (5.63 answers/min). $0.0025 at 8 users (7.69 answers/min). On-demand list price, compute only.
- Peak memory, whole stack
- 23.7 GB
- Search and answer server alone: about 230 MiB.
On-premise or cloud API
What changes when the LLM runs on your server.
| Aspect | Metered cloud API | BAI on your server |
|---|---|---|
| Where prompts and documents go | To the provider’s servers | Stay on your server |
| How you pay | Per token, growing with usage | Fixed setup and a flat maintenance fee |
| Hardware | None of your own | An x86 CPU server you already run |
| Internet connection | Required for every request | Not needed to answer questions |
| Model choice | The provider’s models | Any open-weight GGUF model you choose |
| Model size and speed | Large models, fast GPUs | Smaller models; answers take seconds on a CPU |
Moving to GPUs later is the intended path: llama.cpp supports GPUs, and the same models and search index carry over. BAI’s GPU configuration hasn’t been benchmarked yet.
Deployment
From scoping to first answer.
Scope
We look at your documents, users and servers, and size the deployment with you.
Install
BAI’s services install on your server: Windows Server through the BAI installer, or Linux with Docker.
Load models
Models are copied onto the server from local media or your own network.
Ingest and ask
Your team adds documents, recordings and slides, then starts asking questions.
Current limits
- User accounts and role-based access control aren’t built yet. Run BAI inside your network, behind your existing access controls.
- BAI doesn’t handle TLS itself. Put it behind a TLS reverse proxy.
- There is no enforced offline switch yet. For a fully offline environment, don’t add web URL sources and block outbound traffic at your firewall.
- On the measured server, 1 user stayed within the response-time targets. At 4 users, time to first word p95 was 14.4 s against a 10 s target. Targets were fixed before the run.
Planning an on-premise LLM?
Tell us about your documents, users and servers, and we’ll size a deployment on your hardware.