Give historical records a portable archive format.

Columns, row groups and metadata

Parquet organizes records into row groups with separate column chunks. A reader can choose the columns needed by a query instead of decoding every field. Footer metadata identifies the chunks; available statistics can help a reader skip groups that cannot match a filter.

Column encodings and compression can reduce storage, but the result depends on value distribution, types and writer settings. Measure your actual data. A wide text payload, repeated categorical values and already-compressed binary data have very different characteristics.

The export schema is part of the archive

Record the source schema and the chosen type mapping. Make decimal precision, timestamp units and timezone interpretation explicit. Test nulls, boundary values and binary content through the reader you expect to use. For documents, distinguish a missing field from an explicit null when the retention contract needs it.

Schema changes across files need a reader policy: compatible additions, intentional conversions or separate schema versions. Parquet files do not preserve a database’s full DDL, indexes, foreign keys, triggers or application behavior. Keep that information separately if reconstruction requires it.

Choose file boundaries around retrieval

Partition by a useful, reasonably coarse dimension such as event month. Avoid high-cardinality directory trees and a file per record. Too many tiny files increase listing, metadata and request overhead; very large files can make retries and targeted retrieval cumbersome. Benchmark layouts using representative reads.

Within each batch, maintain a manifest of file locations, selection bounds, schema version, counts and integrity evidence. Do not expose partially written exports as completed archives. The current ColdCTL agent journals local batch progress; object upload and publication are future integration work.

Read the columns and periods you need

DuckDB can query Parquet files with SQL and push projections and filters into the reader. Statistics and layout determine how much work can be skipped. A narrow result does not necessarily imply a narrow scan: an unselective filter or unhelpful row-group ranges may still require substantial I/O.

This example reads local files that already exist. It is independent DuckDB SQL, not a ColdCTL retrieval command. Adapt the schema and path to your export. Remote object access requires the appropriate extension, credentials and endpoint configuration.

SELECT event_type, count(*) AS events
FROM read_parquet('archive/audit_events/2024-06/*.parquet')
WHERE created_at >= TIMESTAMP '2024-06-01 00:00:00'
  AND created_at < TIMESTAMP '2024-07-01 00:00:00'
GROUP BY event_type;

• Plan an on-demand archive query layer

An archive file is not a transactional database

Parquet suits retained, mostly immutable history and analytical scans. Frequent point updates, transactional writes and serving live application requests call for different storage behavior. Updates usually require rewriting files or introducing another layer; the format alone does not supply transaction coordination.

Keep backup and recovery obligations separate from archival. Verify exported values and test retrieval before considering removal of source records. ColdCTL source deletion requires explicit confirmation and matching verified records; verification alone never authorizes it.

• Try the current local export workflow