# Private LLMs for confidential documents: how we chose the models and the GPU server

> A US financial services business needed AI that reads its documents without any cloud AI service. Here is how we tested open-weight models and sized a private LLM server before they bought a single GPU.

**Published:** November 09, 2026
**Updated:** November 09, 2026
**Author:** Matthew Clarkson, Osher Digital
**Category:** Artificial Intelligence
**Reading time:** 16 min read
**Source:** https://osher.com.au/blog/private-llm-confidential-documents/

---

A US-based financial services business asked us to build a private AI system that reads, sorts and extracts data from its documents. The project is highly confidential, so I can't name them. I can tell you the rule that shaped every decision: **none of their documents may ever reach a cloud AI service.** No OpenAI, no Anthropic, no Google, no Bedrock. Every model runs on servers they own, and the main server is air-gapped. It has no internet connection at all.

So this was a self-hosted LLM project with no shortcuts. You can't send a scanned legal agreement to a cloud API and get clean JSON back. You have to choose open-weight models (models whose files you download and run on your own hardware), size a GPU server for them, and prove the whole thing works before anyone signs a purchase order.

Their hardware supplier had already proposed a server. Our job was to test that proposal against the real workload and either confirm it or tell them what to buy instead. We had two weeks.

This post covers how we ran that evaluation, what we found, and what surprised us.

## The shape of the problem

The client planned two servers:

1. **An air-gapped server** for the document pipeline. It reads every page, sorts documents by type, splits bundled PDFs into their parts, extracts structured data into records, matches people and entities across documents, and builds a search index.
2. **An internet-connected server** for a later phase: a chat assistant over the same records, for a small number of users.

The document pipeline is really six jobs, and each one can use a different model:

| Job | What it does |
|---|---|
| Read pages | Turn every page (typed or scanned) into text, with tables intact |
| Sort | Label each document: tax filing, insurance policy, operating agreement and so on |
| Split | Find where one document ends and the next begins inside a bundled PDF |
| Extract | Pull names, dates, amounts and relationships into structured records |
| Match | Decide whether "J. Smith" in one document is the "John Smith" in another |
| Search | Embed and rerank pages so people can find things |

A person approves every proposed record before it reaches the system of record. The AI proposes and a person decides. Keep that in mind for the accuracy section.

## Rule one: synthetic data only

We could not test on the client's documents, even on rented GPUs. So we built a synthetic set: 166 documents, 736 pages. It covers the kinds of documents a financial services business holds: legal agreements, financial statements, tax forms, insurance policies, property records and bundled PDFs. About 30% of the pages are scans, some deliberately skewed, faded or badly photocopied, because real archives look like that.

Every synthetic document has an answer key: the true text, the true document type, the true page boundaries and the true records. The answer key is what lets you score a model with numbers instead of impressions.

Synthetic data has a cost. It's cleaner than reality, and the answer key is only as good as the person who wrote it. (That bit us on extraction, as you'll see.) We say this plainly in the report: the accuracy figures come from synthetic data, and the pilot on the client's own hardware is where they get confirmed.

## Rule two: no licence surprises

The client wants to keep using, adapting and possibly reselling what we build. So they set a hard rule: recommend only models whose licence puts no restriction on use, white-labelling or resale. In practice that means Apache 2.0, MIT or BSD.

That removes some very good models. Several popular model families ship under custom licences with usage caps, naming requirements or acceptable-use clauses. We didn't test those at all. It also means checking the licence on each model card, for each size, because licences change between versions in the same family. We pinned every model to an exact revision and recorded the licence we saw at that revision.

We did not filter by the developer's country. The client wanted the best model for each job and the facts to decide for themselves. The final stack has models from France, China and the US. The report lists US-only alternatives and what they cost in accuracy, so the client can make that call with the numbers in front of them.

## Rule three: measure, don't guess

A lot of GPU sizing for private LLMs is done from parameter counts and back-of-envelope VRAM maths. That gets you within a factor of two, which isn't good enough when the purchase runs to six figures.

We rented the same GPU the supplier proposed (NVIDIA RTX PRO 6000 Blackwell, 96 GB) in 1, 2 and 4 card configurations and ran everything on it. Same card, same driver, and the same version of the inference server (vLLM) that will later be installed offline on the air-gapped machine.

## The test harness

The evaluation is a Python harness. It runs each model against the synthetic set and writes everything to disk: raw outputs, scores, timings, GPU memory and power samples, and a record of exactly what ran. Each job has its own scoring:

- Page reading gets character accuracy, plus two stricter checks that matter more for financial documents: are the numbers right, and are the table cells right? A model can get 99% of characters correct and still put a figure in the wrong column.
- Sorting is plain accuracy against the answer key.
- Splitting gets a boundary score (did it find the first page of each document?) and the share of exact page ranges.
- Extraction gets a field-level F1 score against the answer key's records.
- Matching gets F1 on entity pairs, plus a separate count of wrong merges. Merging two different people is worse than missing a match.
- Search counts how often the right document lands in the top 1, 5 and 10 results.

We also ran one commercial frontier model on the synthetic documents as a reference point. It answers a simple question: how far behind the best available model are we?

### The question nobody asks: does it know when it's wrong?

The client asked something I wish more buyers asked. When the system makes a mistake, how often does it flag the mistake itself?

So every job reports two more numbers: the share of its own errors it flags for review, and its false-alarm rate (how often it flags an answer that was actually right). A self-check that flags everything catches every error and is useless. We report the best catch rate at a false-alarm rate of 10% or less.

The results were mixed, and we reported them that way:

| Job | Errors flagged (false alarms 10% or less) |
|---|---|
| Match | 100% |
| Sort and split | 83% |
| Read pages | 29% to 37% |
| Extract | 20% |

Matching and splitting know when they're unsure. Page reading and extraction mostly don't: their confidence scores don't separate right answers from wrong ones well enough. The client's target was 95%, and we missed it on those two jobs. That's the main reason a person approves every record, and it's why we're planning a second read for unclear text.

I'd rather a client hear this from us in a table than find out in production.

## Round one: about 20 models on a small sample

We started with about 20 candidate open-weight models across the six jobs and ran each one on a small slice of the set. This round was a cheap filter.

Three models didn't load at all in the current inference server. One had an unsupported architecture, one an unsupported layer, one a tokenizer problem. A large chat model we wanted to try didn't run on this GPU generation because a GPU kernel it needs doesn't exist yet. That's normal with brand-new models, and it's exactly what you want to find on a rented card rather than on hardware you've already paid for.

The best two or three per job went through to the full set.

## Round two: the full set

The full-set results were better than I expected. On most jobs, the open-weight models matched or beat the commercial reference:

| Job | Chosen model | Result | Commercial reference |
|---|---|---|---|
| Read pages | LightOnOCR-3 (4B) | 98.8% characters, 99.4% numbers, 95.5% table cells | not tested |
| Read scans and hard pages | Qwen3.8 (27B) | 98.1% characters, 100% numbers, 95.8% table cells | 98.2%, table cells 96.2% |
| Sort | Gemma 4 (31B) | 94% | 98% |
| Split | Gemma 4 (31B) | boundary score 1.00 | 0.98 |
| Extract | Gemma 4 (31B) | field score 0.71 | 0.63 |
| Match | Gemma 4 (31B) | 0.98, no wrong merges | 0.98 |
| Search | Qwen3-VL embedder and reranker (8B each) | right document in the top 5 for 93% of questions | not tested |

All of these are Apache 2.0. Three things stood out.

**One model does four jobs.** Gemma 4 (31B) sorts, splits, extracts and matches. On an air-gapped server that's a big simplification: fewer models to install offline, update and watch, and less GPU memory.

**A small model reads most pages.** A 4B specialist reads typed pages quickly and accurately. The 27B vision model only gets the hard pages: scans, handwriting and dense tables. Sending pages to a model by difficulty saves a lot of GPU time.

**Sorting is one point under target.** 94% against a 95% target, and the commercial model gets 98%. A person approves every record, so a wrong label doesn't reach the system of record. We still put the gap in the report.

### The extraction score that looked worse than it was

Extraction was the hardest job to score fairly. The first numbers looked mediocre. When we read the outputs, many of the "errors" were real facts the model had found and our answer key hadn't listed. If a balance sheet lists twelve assets and the answer key records four, a model that extracts all twelve gets marked wrong eight times for being right.

We kept the strict score as the headline: 0.71, against 0.63 for the commercial reference on the same 91 documents. We also reported the score with those genuine extra findings left out (0.82), and explained both. Reporting only one of them would have misled the client, whichever one we picked.

If you build an evaluation set, remember that your answer key can be wrong too. Read the failures before you trust the score.

### Compression costs accuracy on scans

Running a model at lower precision (FP8 instead of BF16) halves its memory and speeds it up. For extraction it made almost no difference (0.705 against 0.711). For scanned pages it hurt: one model's character error rate on scans went from 2.5% to 6.4%. So the page readers run at full precision and the extraction model runs compressed. You only learn that by testing each job on its own.

### A rare failure worth knowing about

On 3 of 538 pages, the page reader got stuck repeating the same line until it hit its output limit. That sounds rare until you scale it: on a 200,000-page archive it's more than a thousand pages. The production pipeline caps the output length and sends those pages to review.

## The test that mattered most: every model on one GPU server

Benchmarks usually give each model a whole server. Production doesn't. On the air-gapped server, all five models share two GPUs and run their jobs at the same time.

So the last test loaded the full stack onto one rented 2-GPU server, laid out exactly as it will be in production:

| GPU | Models | Memory, loaded | Memory, peak |
|---|---|---|---|
| GPU 0 | Qwen3.8 (27B) and LightOnOCR-3 (4B) | 80.3 GB | 84.2 GB |
| GPU 1 | Gemma 4 (31B, FP8), embedder, reranker | 89.3 GB | 95.0 GB of 97.9 GB |

Then we ran the whole synthetic set through every job at once. It finished in 51.5 minutes, with the same accuracy as when each model ran alone.

Two things came out of this test that solo benchmarks hide:

1. **Throughput drops a lot when models share.** Each model ran 2 to 4 times slower than it did with a server to itself. Page reading fell from about 4,900 pages an hour alone to about 1,270 in the shared stack. Sized from solo numbers, we'd have told the client their archive would take days. It takes weeks.
2. **The second GPU is nearly full.** 95.0 of 97.9 GB at peak. The stack fits, but there's very little room to grow. Another model, or a bigger one, means more GPUs. That's in the report as a limit, not a footnote.

The two cards in the proposed server have no NVLink between them. That sounds like a problem, but here it isn't: each model sits on one card, so the cards never need to talk to each other.

## Speed: what it means in days

Accuracy mattered most to this client, but speed decides when they can start using the system. From the measured shared-stack rates:

| Archive | Pages | Scanned pages | Time, running day and night |
|---|---|---|---|
| 10,000 documents | 200,000 | 35,000 | about 7 to 11 days |
| 20,000 documents | 400,000 | 45,000 | about 2 to 3 weeks |

After the archive, new documents run at about 1,000 pages an hour, so a normal day's intake finishes the same day. Tuning in production will probably beat these numbers, but we didn't count on it.

Long documents have to be split. With this memory layout, the extraction model reads up to about 32,000 tokens at once, roughly 40 to 60 typical pages. Anything longer is processed in parts.

## The chat server: a quick check

The second server is for a later phase, so the brief was a light check: three people at once asking 20 questions about an 11,000-token set of documents.

| Model | GPUs | Correct | Time to first word (typical / slowest 1%) | Speed per user |
|---|---|---|---|---|
| Gemma 4 (31B) | 2 | 85% | 0.10 s / 4.5 s | 35 tokens/s |
| Gemma 4 (31B) | 1 | 85% | 0.16 s / 6.3 s | 22 tokens/s |
| gpt-oss (120B) | 1 | 70% | 0.06 s / 1.0 s | 117 tokens/s |

The target was a first word in under 1.5 seconds and 30 tokens a second per user. Gemma 4 on two cards meets it for typical questions and gave the best answers, so that's what we recommended. gpt-oss is more than three times faster but got noticeably more answers wrong. For a financial services business, I'd take slower and right.

The largest open-weight models (300B parameters and up) don't fit on two of these cards, even at 4-bit. If the client wants them later, the chat server needs four GPUs. That's their business decision, so we asked it as a direct question instead of burying it in an appendix.

## The recommendation

The supplier's proposal was right: one 2-GPU server for each side, with the same card in both. We confirmed it by measurement instead of taking it on trust. We also found its limits (memory headroom, no failover, chart reading and the self-check rates), so nobody gets a surprise later.

Power was modest. The two GPUs in the air-gapped server averaged 716 W and peaked at 977 W with every job running. Storage needs were small too: model files, page images and the search index for 20,000 documents come to under 0.5 TB per server. 10 TB is plenty.

## Keeping the cloud spend under control

We tracked every GPU hour carefully. Every session had a written estimate and a hard cap, rented servers deleted themselves when a run ended or hit its time limit, and every hour went into a ledger. Things still went wrong. Some rented servers never became ready and had to be deleted and replaced. The provider's billing lags behind real usage, which briefly put our ledger out until we corrected it from our own logs. None of it got expensive, because the caps were there.

## What I'd tell anyone doing this

1. **Score each job separately.** "Which model is best?" has no answer. "Which model is best at reading scanned tables?" does.
2. **Test the full stack together.** When models share GPUs, solo benchmarks overstate production speed by 2 to 4 times.
3. **Read the failures.** Our extraction score was partly measuring our own answer key.
4. **Ask whether the system knows when it's wrong.** Then report the answer, even when it's 20%.
5. **Check licences per size and per version.** Pin the revision you checked.
6. **Rent the card you plan to buy.** New models break on new hardware in ways no spec sheet will tell you.
7. **Cap every GPU hour.** Things go wrong in the cloud, and a cap keeps the mistakes cheap.

## Common questions

**What is a private LLM?**
A large language model that runs on hardware you control, so your data never leaves your network. In this project the main server is also air-gapped, so it has no internet connection at all.

**Are open-weight models the same as OpenAI's models?**
No. "Open-weight" means the model files are published, so you can download them and run them yourself. Most of the models we tested come from other developers (Google, Alibaba and LightOn among them). OpenAI has published one open-weight family, gpt-oss, which we tested for chat.

**How much GPU memory does a self-hosted LLM need?**
It depends on the models, the precision and how many documents run at once. Our five-model document stack used 84 GB and 95 GB at peak on two 96 GB cards. Measure it with your own workload before you buy.

**Can a self-hosted LLM match a commercial model on documents?**
On our synthetic set, yes for most jobs. The open-weight models matched or beat the commercial reference on page reading, splitting, extraction and matching. Sorting was 94% against 98%.

**Does an air-gapped AI server need NVLink?**
Not if each model fits on one card. Ours do, so the cards work independently.

## What comes next

Next is the pilot on the client's own hardware, with their real documents, inside their walls. That's the test that counts. They go into it with a server they know will hold the stack, models whose licences they've read, and a clear list of what still needs work.

*Osher Digital builds private AI systems for businesses that can't send their data to the cloud. If that's you, get in touch.*