Retrieve archives with the gateway
Serve verified local, S3, Azure Blob and GCS archives from a customer-hosted Gateway.
Separate customer-hosted service
Run coldctl-archive on your VM with read/list access to the archive namespace and local catalog/staging disk. This service is separate from the agent and Cloud metadata console. It reads completed manifests, verifies full object size/SHA-256 and Parquet metadata, then queries verified temporary files. It does not upload or delete archive objects. Use HTTPS at ingress and a dedicated read-only storage identity.
S3 role configuration
Save this as coldctl.s3.toml, replace YOUR_BUCKET and configure the named gateway token through the environment. The storage role and gateway tenant token serve different purposes. A trusted local gateway-layout.json is required. Azure uses kind = "azure", account, container and managed_identity = true instead of the S3 provider fields.
state_dir = "./gateway-state"
layout_file = "./gateway-layout.json"
[remote]
prefix = "coldctl"
[remote.provider]
kind = "s3"
bucket = "YOUR_BUCKET"
region = "ap-south-1"
role_auth = true
[server]
listen = "127.0.0.1:8080"
[[users]]
name = "operator"
token_env = "COLDCTL_ARCHIVE_TOKEN"
tenants = ["customer"]
admin = trueMap each source dataset
For a shared PostgreSQL/MongoDB sink prefix, map job UUIDs explicitly to trusted tenant and dataset names in gateway-layout.json. Replace the placeholders with completed job IDs and extend the layout for new recurring jobs. Do not map a mixed-source prefix to one dataset. Missing mappings fail catalog refresh; archive data cannot assign its own tenant permissions.
[
{"prefix":"POSTGRES_JOB_UUID","tenant":"customer","name":"orders","source":"PostgreSQL / public.orders"},
{"prefix":"MONGO_JOB_UUID","tenant":"customer","name":"events","source":"MongoDB / app.events"}
]Check, refresh and serve
Supply COLDCTL_ARCHIVE_TOKEN as a strong token of at least 32 bytes, check configuration, refresh the catalog and start the service. Use the configured token in the UI or API. Refresh again after newly completed archives and layout changes; the cloud console does not perform these steps.
coldctl-archive --config coldctl.s3.toml config check
coldctl-archive --config coldctl.s3.toml catalog refresh
coldctl-archive --config coldctl.s3.toml serveCurrent boundaries
Default limits include 128 MiB per object and a shared 2 GiB temporary staging budget. Whole-object verification precedes queries and downloads; no persistent cache or ranged reads are provided. Export status and completed files persist until their configured expiry. Previews return a bounded first page; deep pagination is not implemented. Checksums establish consistency, not signed historical provenance. Test with representative archive sizes before production use.
Durable export recovery
Export status and completed files persist in exports.sqlite and the exports directory. Startup validates checksums; interrupted work becomes INTERRUPTED for explicit resubmission, and missing/corrupt completed files become FAILED. Current owner and tenant grants still control downloads. Temporary download tickets expire on restart; request a new ticket for a retained READY job. Stop the gateway before backing up its catalog, export database and export files together.
Browser access and audit boundaries
The browser offers dataset selection, typed filters, column selection, record inspection and bounded CSV/JSONL/Parquet exports. Credentials identify named users with exact tenant grants; download authorization checks current grants and export ownership. Audit events record user, tenant, dataset, operation, fingerprint and counts in the local catalog. These events are not a tamper-proof compliance ledger. Column selection is not field-level authorization. Built-in PII masking, legal holds and dataset-level grant overrides are not implemented. MongoDB results expose the raw-BSON envelope; arbitrary nested document-field search is not supported.
Which lookups are audited?
Successful searches and record views write SEARCH and RECORD_VIEW events to the local catalog.sqlite before the API returns their results. If that audit write fails, the endpoint returns an error. Events contain a UTC timestamp, configured user name, tenant, dataset, operation, fingerprint and row count. Export creation and download access are also logged; a download event does not prove the client received every byte. Failed or denied lookups are not comprehensively recorded in this audit table. It is not a record-by-record access ledger, and it contains no full returned documents. Protect and back up the catalog and arrange separate retention and central collection if required. Give each person a separate configured credential: shared tokens cannot establish which human made a request. Tenant grants cover datasets mapped to that tenant, not individual fields.
Google Cloud Storage provider
Use kind = "gcs" with the existing bucket and matching remote prefix. Provide read/list Google credentials to the Gateway.