Private AI on Your Own Hardware: What One GPU Runs
What Private Means Here
Private AI, in this post, means an assistant that runs on hardware your company controls, answers from the documents you give it, and does not send prompts, files, or embeddings to someone else’s API. It is not a replacement for every cloud model.
If the model runs through a local inference server, the prompt and retrieved document chunks are HTTP requests to your own machine rather than to OpenAI, Google, Anthropic, or a SaaS wrapper. llama.cpp documents exactly that pattern: use a local model file, start llama-server, and expose an OpenAI-compatible HTTP endpoint on your own host. Its main README also lists CUDA support and CPU+GPU hybrid inference, which is why the answer is no longer “you need a datacenter”. The trade-off is speed: you get privacy and a predictable cost, and you give up throughput.
What a Single 16 GB GPU Runs
A realistic company server today is a workstation-class box with a consumer NVIDIA card: 16 GB of VRAM, enough system RAM, fast NVMe storage, and a service that only your team can reach.
On that machine, the practical target is a mixture-of-experts model with a small active path rather than a large dense one. Qwen’s model card for Qwen3-30B-A3B states the useful numbers plainly: 30.5 billion parameters in total, 3.3 billion activated per token, native 32,768-token context, and validated long context up to 131,072 tokens with YaRN. That is a 30B-class open-weights model, with a much smaller active path.
The file size is also concrete. Unsloth’s GGUF build includes a Q3_K_M quantized file of about 14.7 GB (13.70 GiB — the unit your GPU tool reports memory in) and an UD-IQ3_XXS file of about 12.9 GB (12.00 GiB). Those two numbers decide what fits on a 16 GiB card, but the model weights are not the only thing that has to fit: llama.cpp also keeps a key-value (KV) cache in VRAM, one per token of context, and it grows with context length.
The model’s config.json gives the numbers needed to compute it: 48 layers, 4 KV heads, head dimension 128. At llama.cpp’s default KV cache type (f16, documented in the server flags), that works out to 48 × 4 × 128 × 2 (key and value) × 2 bytes = 96 KiB per token — about 3.00 GiB at the model’s native 32,768-token context, or 12.0 GiB at the validated 131,072-token long context.
| Quantization | File size | + KV cache at 32K context | Total | Fits in 16 GiB? |
|---|---|---|---|---|
| Q3_K_M | 13.70 GiB | 3.00 GiB | 16.70 GiB | No — 16.70 GiB > 16 GiB. Headroom above the file is 2.30 GiB, which the KV cache alone exhausts around 25K tokens (2.30 GiB ÷ 96 KiB/token), before any CUDA context or compute-buffer overhead |
| UD-IQ3_XXS | 12.00 GiB | 3.00 GiB | 15.00 GiB | Arithmetically yes, with about 1 GiB left for CUDA context and compute buffers — tight, not comfortable |
The practical consequence: on a 16 GB card, UD-IQ3_XXS is the quantization that leaves room for the model’s full advertised 32K context; Q3_K_M does not — its KV cache alone runs out of headroom well before 32K. With llama.cpp, you can also keep the hot path on the GPU and spill part of the work into system RAM through hybrid inference when neither fits — that works, but it costs speed, and it is not the same as having the whole model and its context on the card.
In practice that is enough for private chat over internal documents, drafting, summaries, translation, log explanation, policy Q&A, and code review help. It is not enough for hundreds of users at once, huge batch jobs, or standing in for a frontier cloud model.
What “Data Never Leaves” Means
The server is still connected to the internet. Updates and package downloads need outbound traffic; monitoring, when an integration provides it, does too.
What stays inside your boundary is the sensitive path:
- documents are indexed on your server;
- embeddings are stored on your server;
- prompts go to your local model endpoint;
- answers are generated on your hardware;
- logs and backups stay on storage you control.
That is the difference between “we have a privacy policy” and “there is no third-party AI processor in the request path”. If you ask the assistant to summarize a contract, the contract does not need to be uploaded to a cloud model first. If someone asks about a client folder, the retrieved chunks come from your own document store.
This still needs normal security work: access control, TLS, audit logs, updates, backups, and restore tests. On a neglected server the privacy does not last. It also depends on which model actually answers: if the inference server is configured to call an external model provider instead of a local model file, those requests leave the server and follow that provider’s terms and data handling — the local-only guarantee only holds for local inference.
Licensing Basics
An open-weights license still has terms. Check the model license before using it at work.
For this example, the Qwen3-30B-A3B model card declares Apache-2.0, and the linked license text grants rights for use, reproduction, and distribution. In plain business terms: commercial use is allowed, but you still keep the notices and follow the license. (That is a summary of the license, not legal advice.) Other models may have field-of-use limits, attribution rules, or separate acceptable-use policies. Check this before deployment.
When It Makes Sense
A single-GPU private assistant makes sense when the value is in your internal data: procedures, offers, contracts, support history, project notes, meeting summaries, and code repositories. It is strongest when answers can cite your own files and when the risk of sending those files to a third party is the real blocker.
It is the wrong tool when you need massive throughput, image or video generation at scale, or the absolute best reasoning score on public benchmarks. Then the cloud may still win.
For most small companies, the sensible middle ground looks like this: one server, one carefully chosen model, and local retrieval over your own documents, running on your hardware under your rules.
principale.dev installs open-source AI assistants such as OpenClaw on a server you own. Every install is free. See how it works.