Running LLMs in air-gapped networks: what it takes to reach production

Field notes from Strategus. October 2026.

Defense organizations and regulated enterprises often can't send data to external AI services. A pilot built on a cloud API rarely survives the move inside the network. Here is what changes when there is no internet connection, and the decisions that decide whether the project reaches production.

What changes without an internet connection

• No external model APIs. Every model has to run on hardware inside the network.
• No package registries. Libraries, container images and model weights arrive only through an approved transfer process.
• Limited, fixed hardware. You size the system to the GPUs that are already approved, not the other way around.
• Long security reviews. Every new component needs approval before it can run.
• Strict access control. Users may only see documents they are already cleared to see.

Decision 1: Choose the model for the hardware you have

Start from the approved hardware, then pick an open-weight model that fits it. Quantization makes larger models practical on fewer GPUs. For a narrow task, a smaller model tuned and grounded on your own documents often beats a larger general one. Decide this in week one, because it drives every later choice.

Decision 2: Bring in the whole stack, not just the model

A working system needs much more than model weights: the inference server, an embeddings model, a vector store, OS packages, container images and their dependencies. Build an offline bill of materials, verify every file with checksums, and agree on the transfer procedure with the security team early. A missing dependency discovered inside the network can cost weeks.

Decision 3: Retrieval is where the value is

Most of the value comes from answering questions over internal documents, with retrieval-augmented generation. Two rules matter most. Enforce document permissions at retrieval time, so the model never sees a document the user can't open. And show sources with every answer, so users can check them and trust the system.

Decision 4: Evaluate offline, before go-live

You can't call an external service to grade answers. Build an evaluation set with the domain experts who will use the system: real questions, expected answers and the documents that support them. Measure accuracy, citation correctness and refusals. Run the same set after every model or index update.

Decision 5: Plan operations from day one

Logging for audit, monitoring of usage and quality, and a controlled release process for model and index updates through approved media. Treat each update like a software release, with a rollback plan.

Common failure modes

• Prototyping on a cloud API model, then finding that the approved on-prem model can't match it
• Underestimating the time needed for transfers and approvals
• Ignoring permissions in retrieval
• No evaluation set, so there is no way to prove the system is ready

A realistic timeline

Once access and hardware are in place, a first workflow can be in production within weeks. Most of the schedule risk sits in approvals and data transfer, not in the AI itself. Start those in week one, in parallel with the build.

Checklist before you start

• Inventory of approved hardware
• Data sources and their permission model
• Offline bill of materials
• Agreed transfer procedure
• Evaluation set owned by domain experts
• A named owner on the client side