Updates & Synchronization

The appliance ships with a baked-in data snapshot and, from then on, stays current by periodically syncing from UPDATE_URL (https://www.proofaml.com/updates/v1 by default). This page explains how that sync works, and how to run entirely without it for air-gapped deployments.

In plain language

Every SYNC_INTERVAL, the appliance:

1. Downloads a small manifest describing the current state of every dataset — not the data itself.
2. Compares it to the manifest it applied last time, and downloads only the deltas (changes) for datasets that moved — not a full re-download of everything.
3. Verifies each downloaded file's checksum before touching anything on disk.
4. Applies the deltas to a working copy, then verifies the result against the manifest's stated entity counts and content checksums.
5. Only once that verification passes does it atomically swap the working copy in as the live dataset. Until that swap happens, /v1/search keeps serving the previous, already-verified data — a sync in progress, or a sync that fails partway, never produces a window where the API serves partial or corrupted data.

The manifest

curl -s https://www.proofaml.com/updates/v1/manifest.json

{
  "manifestVersion": "2026-08-02T03:00:00Z",
  "epoch": 4,
  "datasets": [
    {
      "id": "us_ofac_sdn",
      "entityCount": 17842,
      "contentSha256": "9f2c...b71a",
      "deltaUrl": "https://www.proofaml.com/updates/v1/us_ofac_sdn/2026-08-02.jsonl.gz",
      "deltaSha256": "3ab0...e94f"
    },
    {
      "id": "un_sc_sanctions",
      "entityCount": 941,
      "contentSha256": "4d61...20cc",
      "deltaUrl": "https://www.proofaml.com/updates/v1/un_sc_sanctions/2026-08-02.jsonl.gz",
      "deltaSha256": "77e1...0b3d"
    }
  ]
}

The manifest is what dataVersion in /healthz and /v1/status identifies, and what /v1/status's datasets[].entityCount is checked against.

Delta files

Each dataset's deltaUrl points at a gzipped JSONL file — one JSON object per line, each an upsert or delete operation:

{"op": "upsert", "id": "NK-a1b2c3d4", "entity": { "…": "…" }}
{"op": "upsert", "id": "NK-b2c3d4e5", "entity": { "…": "…" }}
{"op": "delete", "id": "NK-c3d4e5f6"}

Shipping only the changed records — not a full dataset re-export — is what keeps a daily sync small: a routine day's sync for a dataset is typically a handful of upserts/deletes, not the whole list.

Checksum verification and self-healing

Every sync verifies two things, at two different points:

1. On download: each fetched delta file's SHA-256 must match deltaSha256 in the manifest before it's applied at all. A mismatch aborts that file's application (the previous data for that dataset stays live) and retries on the next sync interval.
2. After applying: once deltas are applied to a working copy, the resulting per-dataset entity count and content checksum are compared against the manifest's entityCount and contentSha256. If either doesn't match — for example, a missed delta from an earlier sync left the dataset in a state the incremental deltas can't reconcile — the appliance self-heals: it discards the incremental result and does a clean full re-download of that dataset instead of leaving it silently inconsistent.

This means an operator never needs to manually diagnose "my incremental deltas drifted" — the appliance detects and corrects it on its own on the next sync.

Epoch

epoch is a single integer in the manifest. Every dataset a given epoch covers is described relative to that epoch's full snapshot; incremental deltas only ever apply within the same epoch.

When ProofAML makes a change that incremental deltas can't safely express — a schema change or a resolver change that affects how records are matched or merged — it bumps the epoch. On seeing a new epoch, the appliance invalidates all currently-fielded data for the affected datasets and performs a fresh full snapshot download, rather than trying to apply deltas against data built under different rules. This is why /healthz and /v1/status surface epoch alongside dataVersion: a sudden epoch bump in your monitoring is the appliance telling you a full resync is happening (or is due), not an error state.

Atomic swap

The appliance never modifies the live, currently-served dataset in place. It builds the new state in a working copy, fully verifies it (checksums + counts, above), and only then swaps it in as the version /v1/search, /v1/batch, and /v1/monitor/* read from. A container restart, a crash mid-sync, or a failed verification all leave the previously-verified dataset serving traffic, untouched.

Air-gapped operation

Set OFFLINE=true and the appliance never contacts UPDATE_URL — no outbound connection is made, ever, on any schedule. This is the correct setting for a fully air-gapped network.

To update an offline appliance, side-load the manifest and delta files into its volume manually:

1. On a machine with network access (not the air-gapped appliance itself), fetch the manifest and the delta files for the datasets you want to update:

   curl -s -o manifest.json \
     https://www.proofaml.com/updates/v1/manifest.json
   curl -s -o us_ofac_sdn-2026-08-02.jsonl.gz \
     https://www.proofaml.com/updates/v1/us_ofac_sdn/2026-08-02.jsonl.gz
   

2. Transfer manifest.json and the delta files across your air gap by your organization's approved offline-media process, into a staging directory inside the appliance's volume: /var/lib/proofaml/updates/incoming/.
3. Trigger the appliance to apply them from local disk instead of the network:

   curl -s -X POST http://localhost:8400/v1/admin/sync
   

   With OFFLINE=true, /v1/admin/sync reads from updates/incoming/ on the volume instead of UPDATE_URL. The same checksum verification, count verification, and atomic swap described above apply identically to a side-loaded update — offline mode changes where the bytes come from, not how rigorously they're verified before going live.
4. Confirm the new version landed:

   curl -s http://localhost:8400/v1/status
   

Admin endpoints

See also

• Configuration reference — UPDATE_URL, SYNC_INTERVAL, OFFLINE, DATA_DIR.
• Search API reference — /v1/status's lastSyncAt / nextSyncAt / updateUrl fields.
• Troubleshooting / FAQ — stale data and checksum-mismatch scenarios.