Updating model weights in an air-gapped environment without breaking the air gap

An air gap is not a property of a machine. It is a property of a procedure. The moment a model update becomes routine, someone will look for a shortcut, and the shortcut is always the same one: a laptop that touched both networks. This article is the procedure that removes the need for the shortcut — how to move a new set of model weights into an isolated environment, verify it, roll it out, roll it back, and leave an audit trail that survives a review.

Everything below is written for a deployment that is genuinely disconnected: no proxy, no allowlisted mirror, no “just this one host”. If you have an outbound proxy, most of this still applies and about a third of the work disappears.

The boundary is defined by what crosses it, not by what is plugged in

An air gap survives on one rule: every byte that crosses the boundary is inventoried, hashed, signed on the outside, and verified on the inside before anything executes it. A network cable is not the boundary. The boundary is the set of transfer events, and each transfer event needs a name, a manifest, a signature, and a record.

This reframing matters because model updates fail the “is it plugged in” test easily and fail the “what crossed” test constantly. A model directory is not one file. It is weights, a tokenizer, a config, a chat template, sometimes a Python module the loader will import, and — if you are careless — a pickle that runs arbitrary code the instant it is loaded. Treating it as “just weights” is how an air gap becomes a slow-motion supply-chain incident.

What actually has to cross the gap

Six categories of artefact cross an air gap for a model update, and they have different risk profiles. Bundling them together and verifying them as one unit is the only way to keep the inventory honest.

Separating the runtime from the weights is the single highest-value decision here. Weights change monthly; the inference runtime should change quarterly, under a different and slower approval path. Bundling them together forces every model swap through a full runtime review, and the predictable result is that people stop updating models at all.

Build the bundle on the connected side, and make it self-describing

The bundle is built once, on the connected side, by a script — never by hand. A hand-assembled directory has no manifest, and a bundle without a manifest cannot be verified, only trusted.

#!/usr/bin/env bash
# build-bundle.sh — CONNECTED side. huggingface_hub >= 0.24 (newer releases alias
# `huggingface-cli download` as `hf download`), gnupg >= 2.2, coreutils.
set -euo pipefail

REPO="${1:?usage: build-bundle.sh <hf-repo-id> <version-tag>}"
TAG="${2:?}"
OUT="bundle/${TAG}"
SIGNER="releases@example.internal"

mkdir -p "${OUT}/weights"

# 1. Pull tensors and metadata only. Explicitly refuse pickle formats.
huggingface-cli download "${REPO}" \
  --local-dir "${OUT}/weights" \
  --include "*.safetensors" "*.safetensors.index.json" "*.json" "*.txt" \
            "*.model" "*.jinja" "LICENSE*" \
  --exclude "*.bin" "*.pt" "*.pth" "*.pkl" "*.ckpt" "*.h5" "*.msgpack"

# 2. Record where this came from, at the commit level — not the tag level.
#    Tags move. Commit hashes do not.
REV="$(huggingface-cli scan-cache -v | awk -v r="${REPO}" '$1==r {print $NF; exit}')"
cat > "${OUT}/PROVENANCE.json" <<JSON
{
  "repo": "${REPO}",
  "revision": "${REV}",
  "bundle_tag": "${TAG}",
  "built_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
  "built_by": "$(id -un)@$(hostname -f)",
  "runtime_included": false
}
JSON

# 3. One manifest over everything, in a stable order.
( cd "${OUT}" && find . -type f ! -name 'MANIFEST.sha256*' -print0 \
    | sort -z | xargs -0 sha256sum > MANIFEST.sha256 )

# 4. Detached signature over the manifest only. Signing the manifest signs
#    every file in it, transitively, and stays a one-line verification.
gpg --detach-sign --armor --local-user "${SIGNER}" \
    --output "${OUT}/MANIFEST.sha256.asc" "${OUT}/MANIFEST.sha256"

echo "bundle ready: ${OUT}  ($(du -sh "${OUT}" | cut -f1))"

Two details in that script are load-bearing. The download pins a commit revision, not a branch or tag, because tags on public model hubs are mutable and “we pulled main in August” is not a provenance record. And the signature covers the manifest rather than each file, so verification stays a single command that either passes completely or fails completely.

Verify before the bytes reach a GPU host

Verification runs on a staging host inside the boundary that has no GPU and no inference runtime installed. Its only job is to reject bad bundles, so it should not be capable of running one.

#!/usr/bin/env bash
# verify-bundle.sh — ISOLATED side, on the quarantine host.
set -euo pipefail
BUNDLE="${1:?usage: verify-bundle.sh <bundle-dir>}"

# 1. Signature. The release public key was imported once, out of band,
#    and its fingerprint is recorded in the change management system.
gpg --verify "${BUNDLE}/MANIFEST.sha256.asc" "${BUNDLE}/MANIFEST.sha256"

# 2. Content. --check fails on any missing or altered file.
( cd "${BUNDLE}" && sha256sum --quiet --check MANIFEST.sha256 )

# 3. Format gate. Pickle-backed tensors execute Python on load; safetensors
#    cannot. This is a hard refusal, not a warning.
python3 - "${BUNDLE}" <<'PY'
import pathlib, sys
BANNED = {".bin", ".pt", ".pth", ".pkl", ".ckpt", ".h5", ".msgpack"}
bad = [p for p in pathlib.Path(sys.argv[1]).rglob("*") if p.suffix in BANNED]
if bad:
    sys.exit("REFUSED: pickle-format tensors present:\n  " +
             "\n  ".join(str(p) for p in bad))
code = [p for p in pathlib.Path(sys.argv[1]).rglob("modeling_*.py")]
if code:
    sys.exit("REFUSED: custom modelling code requires separate review:\n  " +
             "\n  ".join(str(p) for p in code))
print("format gate: clean")
PY

echo "bundle ${BUNDLE}: VERIFIED"

The pickle refusal is not theoretical. torch.load deserialises through Python’s pickle, which executes code embedded in the checkpoint during loading — the documented reason safetensors exists is that it stores raw tensors with a JSON header and executes nothing. Inside an air gap you have no upstream scanner and no vendor to page, so the format gate is your only control. The same logic applies to trust_remote_code=True in Transformers: it imports Python shipped alongside the weights. In an isolated environment that flag stays off, and any model that requires it gets its modelling code vendored and reviewed as source, on the slow path.

You can also inspect a safetensors file’s tensor inventory without allocating a byte of GPU memory, which is a cheap way to confirm the architecture matches what you approved:

# check_header.py — reads the safetensors header only. torch not required.
import json, struct, sys

with open(sys.argv[1], "rb") as f:
    (header_len,) = struct.unpack("<Q", f.read(8))
    header = json.loads(f.read(header_len))

tensors = {k: v for k, v in header.items() if k != "__metadata__"}
print(f"{len(tensors)} tensors")
for name in sorted(tensors)[:5]:
    t = tensors[name]
    print(f"  {name:60s} {t['dtype']:8s} {t['shape']}")

Mirroring the model registry inside the boundary

Inside the boundary you need a registry, not a directory of files, because the thing you roll back to has to be addressable by name. Two mechanisms cover most on-premise stacks.

Ollama stores everything under ~/.ollama/models as blobs/sha256-<hash> plus a manifest tree under manifests/<registry>/<namespace>/<model>/<tag>. The layout is content-addressed, so the blobs are already verifiable and transferring a model is a matter of copying the blobs a manifest references and the manifest itself. The more maintainable path for an air gap is to skip the upstream registry entirely and build the model locally from a GGUF file you carried across:

# Modelfile — build inside the boundary, tag with the bundle version.
FROM ./qwen2.5-14b-instruct-q4_K_M.gguf
PARAMETER num_ctx 8192
PARAMETER temperature 0.2
SYSTEM """You answer questions about the data you are given. If the data does
not contain the answer, say so."""

ollama create internal/analyst:2026-08-27 -f Modelfile
ollama list          # confirm the tag exists before any traffic is pointed at it

Tagging with the bundle date rather than latest is what makes rollback possible later. latest is a name for one thing at a time; you cannot roll back to it.

Hugging Face-format models served by vLLM or a Transformers-based microservice need the cache pinned and the network calls disabled. Both are environment settings, and both should be set at the systemd unit level so they cannot be forgotten in a shell:

# /etc/systemd/system/embeddings.service.d/offline.conf
[Service]
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1
Environment=HF_HUB_DISABLE_TELEMETRY=1
Environment=HF_HOME=/srv/models/hf
Environment=VLLM_NO_USAGE_STATS=1
Environment=DO_NOT_TRACK=1

HF_HUB_OFFLINE=1 stops HTTP calls to the Hub when loading a model, and telemetry is disabled in offline mode as well (huggingface_hub environment variables). vLLM’s usage statistics are disabled by VLLM_DO_NOT_TRACK, DO_NOT_TRACK, or VLLM_NO_USAGE_STATS, or by the presence of $HOME/.config/vllm/do_not_track (vLLM usage stats). Set them anyway — but do not rely on them as the control. In an air gap the control is that the packets have nowhere to go; the environment variables exist so your logs are not full of connection failures.

Digisky’s own embedding path is a self-hosted BGE-M3 microservice — 1024-dimension dense vectors, MIT licensed, XLM-RoBERTa based. The licence is part of the deployment decision, not a footnote: a model whose weights cannot be redistributed cannot legally be carried across an air gap on a USB drive by the integrator, and that eliminates a large fraction of the candidate list before any benchmark is run.

The rollout is a row in a settings table, not a deployment

Inside an air gap, a deployment is expensive: it means another bundle, another approval, another transfer event. So the model in use must not be a deployment artefact. In DataCopilot the active model is held in an app_settings key/value table and switched at runtime, which means an operator changes the model in use without shipping a release.

That property is what makes a staged rollout possible on the inside:

-- Stage 1: the new model exists in the registry but serves no traffic.
INSERT INTO app_settings (key, value)
VALUES ('llm.model.candidate', 'internal/analyst:2026-08-27')
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value;

-- Stage 2: route one named group to the candidate. Everyone else is untouched.
UPDATE app_settings
   SET value = 'internal/analyst:2026-08-27'
 WHERE key = 'llm.model.group.data_team';

-- Stage 3: promote, keeping the previous value written down in the same
-- statement so the rollback target is never a matter of memory.
UPDATE app_settings
   SET value = 'internal/analyst:2026-08-27'
 WHERE key = 'llm.model.default';

Run each stage as its own change, with a defined observation window between them. What you are watching for in an isolated environment is narrower than what a connected team watches: you have no upstream incident feed, no vendor status page, and no comparison against a hosted control. So watch the things that are local and unambiguous — refusal rate, tool-call parse failures, output length distribution, and the rate at which the agent hits its step ceiling.

Rollback is a row update, and it must be rehearsed

Rollback in an air gap has one requirement: the previous model must still be resident. If the rollback plan is “carry the old bundle back in”, there is no rollback plan, because the transfer takes hours and needs an approval.

Keep at minimum the current and the previous version present on every inference host, and make disk retention an explicit policy rather than a side effect of ollama rm. Then rollback is:

UPDATE app_settings SET value = 'internal/analyst:2026-06-14'
 WHERE key = 'llm.model.default';

Rehearse it during the rollout window, not during the incident. A rollback that has never been executed is a hypothesis.

The audit trail an auditor will actually ask for

An auditor reviewing an isolated environment asks four questions about any artefact on it, and a model is no exception: what is it, where did it come from, who approved it, and who moved it. Answer all four in one record, written at transfer time.

The one design choice worth copying: write the audit record in the same database transaction as the change it describes. Digisky’s legal platform does this — the audit row and the action commit together, with a database trigger that makes UPDATE and DELETE on the audit table raise. An audit log written after the fact by a separate process is a log that can disagree with reality, and in an isolated environment there is no second source to reconcile it against.

Where this advice stops applying

This procedure assumes a fully disconnected environment with a physical transfer step and a signing authority you control. It does not carry over cleanly in four situations.

If you have an approved outbound proxy to a vendor mirror, the manifest and signature work is still worth doing, but the staged transfer choreography is overhead — pull through the mirror and verify on arrival.

If your models are delivered as encrypted or licence-checked blobs by a commercial vendor, the format gate cannot inspect them and the provenance record depends on the vendor’s own signing. That is a materially weaker position, and worth knowing before you sign.

If you are updating the runtime rather than the weights, this is the wrong procedure. Runtime updates change the CUDA and driver surface, and need a rehearsal environment with the same GPU model, not a checksum and a settings flip.

And if your isolated environment has no second inference host, none of the staged-rollout advice survives: with one host, promotion and rollback are the same event, and the only real mitigation left is keeping the previous model resident and the rollback command written down where the on-call operator can find it at three in the morning.