Measured
A. Standard x86 CPU servers
Results on this page come from a Google Cloud c3-standard-22 (22 vCPU Intel Xeon Platinum 8481C, no GPU). Earlier AWS x86 runs are being verified and will be published as a separate result.
BAI Performance Benchmarks
Private AI performance, measured on real hardware.
BAI benchmarks retrieval, ingestion, inference and resource use on defined workloads, each tied to its hardware, model and evaluation methodology.
No GPUMeasured 29–30 October 2026Methodology v1.0
Results · x86
Google Cloud CPU server
- Instance
- Google Cloud c3-standard-22
- CPU
- Intel Xeon Platinum 8481C @ 2.70 GHz (x86-64)
- vCPUs
- 22
- Memory
- 86.4 GB
- GPU
- None (0 model layers offloaded)
- Price
- $1.17/hour, on-demand list price
Workload
- Public Apollo 11 archive from NASA: 259 files (PDF, JPEG, MP3 and MP4), each pinned by SHA-256 so anyone can download and check it.
- A set of Kenyan public-sector documents with 93 test questions, used for answer timings.
Run c3-standard-22_20261029T115431Z. These results apply to this server only. They aren’t averaged or combined with any other environment.
Answering
- Median time to a full answer
- 8.5 s
- 1 user, Apollo 11 archive, 57 questions. p95 11.7 s. Earlier ranking configuration: 9.7 s (p95 20.9 s).
- Median time to a full answer
- 9.3 s
- 1 user, Kenyan public-sector documents, 93 questions. p95 17.1 s. Earlier ranking configuration: 10.8 s (p95 23.6 s).
- Median time to first word
- 7.3 s
- 1 user, load test, earlier configuration. Full answer in that test: median 9.3 s, p95 18.5 s.
Search
- Search latency, 1 user
- 9.2 ms
- Median, top 5 passages. p95 15.7 ms.
- Search latency, 8 users
- 25.8 ms
- Median, 295.6 searches per second. An earlier full-harness test mixing 1, 4 and 8 users measured a 42 ms median.
Capacity, cost and memory
- Users within response-time targets
- 1
- At 4 users, time to first word p95 was 14.4 s against a 10 s target. Targets were fixed before the run.
- Compute cost per answer
- $0.0035
- 1 user (5.63 answers/min). $0.0025 at 8 users (7.69 answers/min). On-demand list price, compute only.
- Peak memory, whole stack
- 23.7 GB
- Search and answer server alone: about 230 MiB.
- PDF pages indexed
- 4,881
- Text and scanned PDFs, plus images, audio and video from the same archive.
Ingestion by input type
| Input | Files ingested | Volume | Rate |
|---|---|---|---|
| Text PDFs | 19 / 19 | 3,236 pages | 124.6 pages/min |
| Scanned PDFs (OCR) | 3 / 3 | 1,645 pages | 54.6 pages/min |
| JPEG images (OCR) | 204 / 211 | 204 images | 76.7 images/min |
| Audio (MP3) | 11 / 11 | 2.22 hours | 6.4× faster than real time |
| Video (MP4) | 15 / 15 | 0.08 hours (small sample) | 4.1× faster than real time |
| Word documents | Supported · not yet benchmarked | ||
| PowerPoint presentations | Supported · not yet benchmarked | ||
| Web URLs | Supported · not yet benchmarked | ||
Throughput is the total ingestion time per type as recorded by the benchmark harness. Upload, extraction, OCR or transcription and indexing were not timed separately in this run. Word, PowerPoint and web URLs were not part of this run.
Ingestion benchmark roadmap
Timing every format, stage by stage.
The next ingestion benchmark covers all seven input types and times each stage separately. Per-type timings will appear here once they are measured. Until then, there are no per-stage numbers to show.
Recorded for every file
- Input file size, with page count, image count or media length
- Upload duration, where a file is uploaded
- Extraction duration
- OCR or transcription duration
- Chunking and indexing duration
- Total time until the content is searchable
- Hardware and workload configuration
Hardware roadmap
Three classes of hardware.
Each class gets its own results, published with its configuration. Untested hardware is shown as planned, never as a result.
Planned
B. AI-oriented CPU servers
Next: newer compute-oriented CPUs, starting with an AWS Graviton (Arm64) instance. BAI’s search and answer server builds for 64-bit Arm and passes its functional tests under emulation. The inference runtime and ingestion service still have to be built and checked on Arm before a test is scheduled.
Planned
C. Compact single-GPU systems
First candidate: AWS g4dn.xlarge (NVIDIA T4), subject to regional availability and budget, with g5.xlarge (NVIDIA A10G) as a later comparison. NVIDIA DGX Spark is the on-premises reference for future physical-hardware tests. Its CPU and GPU share one pool of unified memory, unlike cloud instances where the GPU has its own separate memory, so cloud GPU results won’t stand in for DGX Spark.
Technical note
Intel AMX acceleration
Intel Advanced Matrix Extensions (AMX), available on recent Xeon processors including the one tested here, can speed up the prompt-processing step of inference. A same-machine prompt-processing test is being checked against its original outputs. If confirmed, it will be published as a prompt-processing result on supported Intel CPUs, not a general inference speed-up. Answer-generation speed is a separate measurement and can behave differently.
Answer citations and accuracy
Traceable, not infallible.
BAI is designed to provide source references so users can trace answers back to their documents. Citation correctness and answer quality are evaluated separately. Every answer returns the passages it used, with their document and page. In-text citations aren’t yet present in every answer, and no AI system gets every answer right.
Not published yet
What we’re still verifying.
These results exist in draft or haven’t been measured. They stay off this page until they meet the same standard as the figures above.
- Being verified
AWS x86 results
Earlier AWS runs are not in the evidence log yet. They will be published separately from the Google Cloud results once their run artefacts are checked.
- Draft
Retrieval and answer accuracy
The question sets are still being reviewed. Accuracy will be published with its confidence intervals once the review is complete.
- Draft
Citation coverage
Being re-measured after recent changes, and published together with answer quality.
- Being verified
Intel AMX acceleration
A same-machine prompt-processing test is being checked against its original outputs.
- Not measured
Cost on your own hardware
On your own servers there are no per-question fees to any AI provider. Running costs depend on your hardware and are not measured.
Methodology
How these numbers were measured.
- Hardware and CPU architecture
- Google Cloud c3-standard-22: 22 vCPU Intel Xeon Platinum 8481C @ 2.70 GHz, x86-64.
- GPU and memory
- No GPU; no model layers offloaded. 86.4 GB of RAM.
- Software
- BAI’s runtime services with llama.cpp’s server for inference and ONNX Runtime for search embeddings. Exact build and configuration are recorded with each run.
- Model and quantisation
- Llama 3.2 3B Instruct, 4-bit (Q4_K_M), the standard configuration, run by llama.cpp; bge-small-en-v1.5 embeddings. The model file’s checksum will be added to the published run record. Search returns the top 5 passages; answers are streamed.
- Datasets and questions
- The public Apollo 11 archive from NASA (259 files, each pinned by SHA-256), with 57 test questions; and a set of Kenyan public-sector documents with 93 test questions.
- Held-out questions
- Timed questions come from test splits kept separate from the questions used during development and tuning.
- Warm-up and workload
- Untimed warm-up requests run first and are excluded by rule. Answer times are sequential, one user at a time. Search was measured at 1 and 8 concurrent users for 20 seconds per level. The capacity test ramps users against response-time targets fixed before the run.
- Latency definitions
- Search latency: one request returning the top 5 passages. Time to first word: from sending a question to the first answer token the client receives. Time to a full answer: from sending to the last byte of the answer. Prompt processing and generation speed are separate measurements and aren’t reported here yet.
- Percentiles, intervals and sample size
- Medians and 95th percentiles use the nearest-rank method. Accuracy will be published with 95% confidence intervals. Small samples are flagged where they occur, such as the 0.08 hours of video.
- Ingestion boundaries
- Ingestion rates are total time per input type as recorded by the harness. Upload, extraction, OCR or transcription, and indexing were not timed separately in this run.
- Date and version
- Run on 29–30 October 2026. Methodology version 1.0.
Want numbers for your own hardware?
Tell us about your documents, users and servers, and we’ll scope a test on your workload.