The Complete Guide to Air-Gapped LLM Deployment

Air-gapped LLM deployment means running a large language model entirely inside a network with no path to the public internet, so model weights, prompts, and outputs never cross a security boundary. It is the most demanding form of on-premises AI, and in defense, healthcare, nuclear, and critical-infrastructure organizations it is often the only compliant way to use LLMs at all. This guide explains what an air gap actually is, how an offline LLM deployment works in practice, what it costs, and how to decide which isolation level a workload really requires.

What an air gap actually means

NIST’s glossary defines an air gap as a separation between two systems that are not physically connected and at which any logical connection is not automated — data moves across the gap only manually, under human control. In practice, an air-gapped network has no network interface to the outside world; anything that crosses the boundary moves on physical media through a controlled, logged process.

Air gaps are not an AI invention. They predate large language models by decades and remain the standard isolation pattern for military and government computer networks, financial systems such as stock exchanges, and industrial control systems like the SCADA networks that run oil and gas operations.

The term is used loosely in AI projects; three levels are worth distinguishing:

• Physical air gap. No network hardware connects the environment to anything outside it. Isolation is enforced by the absence of a path, not by a policy that could be misconfigured.
• Logical air gap. The hardware sits on shared physical fabric, but the segment is closed off with enforced routing, firewall policy, and network controls. A connection could exist; none is permitted or automated.
• Virtual air gap. Logical separation inside a shared compute platform — segmentation, namespace isolation, and egress blocking in a virtualized or private-cloud environment. It is the cheapest to operate and the weakest to defend. The virtual air gap glossary entry covers how far that protection actually goes.

The stronger the isolation, the higher the operating cost. The job of the architecture decision is to buy the level the requirements demand — not the level that looks most impressive in a board deck.

Why regulated organizations need it

Defense and defense contractors. Federal contractors that touch Controlled Unclassified Information (CUI) must protect it under NIST SP 800-171, which sets the confidentiality requirements for CUI resident in nonfederal information systems. DoD’s CMMC final rule goes a step further: under the program, Level 2 requirements apply to all contractors that process, store, or transmit CUI, and DoD’s rule estimates 135 third-party certification assessments in the program’s first year alone. For many primes and suppliers, the practical consequence is that CUI — and any LLM that might see data containing it — runs on networks that commercial cloud endpoints cannot reach. Sovereign AI for defense covers the broader program picture.

Healthcare. Health systems face the same problem in different language. The HIPAA Security Rule requires that electronic protected health information be safeguarded against reasonably anticipated threats, hazards, and impermissible uses and disclosures, and NIST’s implementation guidance (SP 800-66) frames those obligations as concrete security safeguards. When an LLM will read clinical notes or patient data, sending that data to a commercial API is a disclosure risk unless the provider is a contracted business associate — and many health systems simply keep that traffic on isolated networks.

Nuclear, energy, and finance. Utilities and nuclear operators extend the same logic to safety and operational data, and financial firms have run air-gapped systems around exchange and trading infrastructure for decades. The pattern is identical across all four: a regulator or contract counterparty watches the data, and the LLM must work on it without it ever leaving the boundary.

This is the practical core of sovereign AI: ownership of the model, the compute, and the data path inside the security perimeter, with an air-gapped deployment as the most stringent expression of that posture. NIST’s AI Risk Management Framework, released in January 2023 and intended for voluntary use, is a useful governance frame for the program — its purpose is to help organizations incorporate trustworthiness into the design, development, use, and evaluation of AI systems.

How offline LLM deployment works

An air-gapped LLM deployment differs from a cloud deployment in exactly three loops: how the model gets there, how it is served, and how it gets updated.

Model acquisition and transfer

Model weights, embedding models, and every runtime dependency are downloaded in a connected “gold” environment, checksummed and, where available, signature-verified, then transferred into the air-gapped network on approved physical media. There is no runtime pull from a model hub. If a weight is not on the network, the model does not exist — the acquisition list has to be complete before day one, and the media-transfer process becomes part of the security control set.

Serving without cloud fallback

Inference runs on local GPUs. A self-hosted, OpenAI-compatible serving stack such as vLLM is the common pattern: Elastic’s documentation for air-gapped deployments describes running vLLM behind a reverse proxy with no outbound network access at all, and calls it a safe option for air-gapped environments. Applications point at the local endpoint and behave as if they were calling an API.

The critical difference: there is no cloud fallback. If the serving node fails, the feature stops — redundancy (spare GPUs, multi-node serving, on-prem alerting) must be designed in, not assumed from a provider’s SLA.

Sizing is arithmetic once quantization is chosen. A 70B-parameter model needs roughly 140 GB of memory for its weights alone at full precision; the same model at 4-bit quantization needs approximately 43–45 GB of total memory including context and framework overhead. A 43-45 GB footprint fits on a single 80 GB data-center GPU card, and mid-size models fit on far smaller footprints. Throughput — tokens per second at expected concurrency and context length — then determines how many GPUs, not how many gigabytes.

The update cycle

Models and software are updated in batches, not continuously. The cycle: a release lands in the gold environment, is tested in a lab replica, and is staged as a signed artifact for media transfer. The cadence — quarterly, semiannual — is a business decision, not a technical one, and it should be written into the operating agreement with the teams that consume the model. For an AI program accustomed to continuous improvement, this is the single largest cultural change.

Operational tradeoffs

Stale models. The model’s knowledge and the weights themselves freeze at the last transfer. There is no mid-cycle improvement. The trade is acceptable for most document, extraction, and analysis workloads; it is not for workloads that depend on current external knowledge.

No failover path. A cloud deployment degrades when one provider hiccups; an air-gapped deployment has nothing to degrade toward. Every failure mode is local, and availability depends entirely on the on-prem redundancy the organization built.

Offline maintenance. Every OS, driver, CUDA, and framework patch is an offline project: fetch it in the gold environment, test it, transfer it, apply it. Government and military organizations already run their core systems this way — their networks are disconnected and air-gapped with no access to the public internet, and vendors ship offline installers and internal certificate authorities to serve them. Most AI tooling was not designed for that reality, and the gap shows up as patch lag.

Staffing. The stack needs a small team covering infrastructure, MLOps, and the update process. An understaffed air-gapped environment does not fail dramatically; it quietly falls behind on models, patches, and support.

Velocity. New models and features cross the gap only when the update process carries them — budget for that lag explicitly.

The Cost in a CFO’s Frame

The cost model has a capital half, an operating half, and a risk term that cloud pricing never shows.

Capital. GPU servers sized for the chosen model and throughput — a 70B model at 4-bit fits on one 80 GB GPU, while full-precision weights of roughly 140 GB mean two or more — plus storage for the model library and retrieval data, network gear for the isolated segment, and in many cases a dedicated space with appropriate power and cooling. This is a one-time outlay amortized over the hardware refresh cycle.

Operating. Power and cooling; the one to three people who own the stack; the recurring cost of the media-transfer and lab-testing process; and the opportunity cost of a fixed update cadence. The honest comparison against cloud is not the server price — it is total cost per 1,000 tokens served at the organization’s actual volume, versus API pricing at the same volume.

Risk. This is the term that justifies the spend. The air gap is a purchase of assurance: data residency, no external dependency, and a threat surface a regulator or contract counterparty can audit. In CMMC- or 800-171-covered programs, that assurance has a direct price — non-compliance costs contract eligibility, not merely a fine. Model the decision as (capital + operating) versus (cloud spend + residual risk). In regulated environments, the risk term usually dominates.

When do you actually need an air gap

Most organizations land in the middle, not at the extremes. Work the questions in order:

1. Does a regulation, contract, or customer requirement specify the isolation level? If a contract requires physical separation, the question is closed — budget for a physical air gap and move on.
2. What is the classification of the data the model will touch? CUI, PHI, or safety-critical operational data points to at least a logical air gap; classified work points to a physical one.
3. Who is in the threat model? An air gap defends against actors outside the boundary — including commercial cloud providers. It does not defend against insiders; access control and audit do that.
4. Can the business absorb a batched update cadence? If the use case depends on weekly model releases, a strict air gap will frustrate it. Choose the strongest isolation the update tolerance allows.
5. Does an approved media-transfer process exist? If not, budget for it as a first-class workstream. It is where most air-gapped deployments quietly stall.

The mapping that follows from those answers:

• Physical air gap: classified work, contractually mandated isolation, hostile threat models.
• Logical air gap: CUI or PHI workloads where the rule requires network separation, not physical separation.
• Virtual air gap: pilots and internal tools on an on-prem platform while a dedicated environment is planned — a transitional state, not an end state.
• No air gap: low-sensitivity internal workloads. Run them on-prem or in a private cloud and keep the budget for workloads that need the isolation. The boundary case worth naming: if the requirement is simply “no public cloud” but the environment keeps normal internal connectivity, that is on-premises AI, not an air gap — and it is cheaper to run. See on-premise AI for that decision.

How to plan an air-gapped LLM deployment

1. Map the data first. Identify every data class the model will touch and the handling requirement attached to each. The isolation level follows from the data, not from the model.
2. Choose the isolation level the requirement and threat model justify — the strongest level that is required, not the strongest level available.
3. Size the serving stack for the real workload: expected concurrency, context length, and tokens per second. Use quantized weights to cut memory footprints by an order of magnitude, and size GPU count for throughput.
4. Design the update pipeline before buying hardware: gold environment, checksums, signed media, a lab replica, and a documented cadence.
5. Plan for failure without a fallback path: spare GPUs or a multi-node configuration, monitoring, and an on-prem alerting path.
6. Staff the operating model explicitly: who owns patching, media exchange, and model refresh.
7. Track the program like any other infrastructure: availability, time from vendor release to production, and cost per 1,000 tokens served locally.

When a vendor platform is on the shortlist, require an air-gap reference architecture in the RFP: model transfer, update cycle, and offline telemetry separate a deployable product from a brochure. Book a demo to see how an air-gapped deployment looks end to end.

Last verified: 2026-09-05