Why Parquet Works Well for Database Archives

The format must outlive the export job

A historical export is easy to produce and harder to keep useful. Several years later, the database schema may have changed, the writer may have been upgraded, and a reader may only need three fields from a wide event record. A good archive stores values in a representation that can be interpreted independently of the original process.

Parquet is a columnar file format used by analytical readers, including DuckDB. It is a useful fit for mostly immutable structured history and selective scans. It is not a transactional database, a backup policy or a guarantee of compression. The export contract still determines whether the retained records are complete and meaningful.

Columns and row groups support selective reads

Within a row group, each column has its own column chunk. Footer metadata identifies where those chunks are stored. A reader can select relevant columns instead of decoding every field, and available statistics can help reject row groups that cannot match a predicate.

Suppose a query needs event_type and created_at but not a large payload. Column selection can avoid decoding that payload. If records are organized into useful time ranges, a time filter may also avoid some row groups. Neither behavior is a promise that every query reads only its result rows: statistics, data distribution and reader support determine what work can be skipped.

• Use projection and filters in a DuckDB query →

Compression depends on the data and the writer

Repeated categories, numerical fields and long text have different encoding and compression behavior. Already-compressed payloads may offer little additional reduction. Codec choice trades storage against CPU and compatibility with the intended reader; measure an actual sample rather than applying a universal database-to-Parquet ratio.

For an archival trial, record raw exported bytes, final file bytes, codec, writer version and representative query behavior. Those measurements describe that trial, not a guaranteed saving for another table. No benchmark result or compression ratio is assumed in this guide.

Logical types need a mapping contract

Physical storage alone does not express all database meaning. Parquet logical types annotate values such as decimals and timestamps. Document precision, scale, time units and timezone interpretation. Decide how unsigned source integers, binary values and fields outside the reader’s supported range will be represented.

Test values at the edges, not only ordinary rows: a large identifier, a high-precision decimal, a null timestamp and a non-ASCII string. For MongoDB documents, include mixed field types, missing fields, explicit nulls and nested arrays. A normalized analytical schema and an exact BSON round trip are different contracts.

Treat schema evolution as a read policy

Files written at different times can have different fields or types. Define whether an added field becomes nullable, an incompatible type starts a new schema version, or historical files are rewritten. Retain the mapping version and source schema context with each completed batch.

Combining files by column name can help compatible additions, but it cannot decide that a text value and a numerical value mean the same thing. Test the reader across the oldest and newest schema versions. Parquet also does not reconstruct application indexes, foreign keys, triggers or full DDL; preserve additional metadata when restoration requires it.

Choose partition and file boundaries deliberately

Partition directories should reflect common retrieval boundaries, such as event month, without creating a directory and file for every identifier. Many small files add listing and metadata work. Very large files can make failed exports expensive to retry or create less convenient units for handling archive damage.

File size and row-group size are separate choices. Row groups influence parallelism and the granularity of statistics-based skipping; files influence publication, cataloging and retry units. Sorting by a commonly filtered field may improve statistical selectivity, but sorting itself takes resources. Evaluate the whole write-and-read workflow on representative data.

Object storage is a destination, not a catalog

Parquet files can live in customer-controlled object storage, but an object key alone does not tell a user which historical records are available. Maintain a manifest that records dataset, time range, source identifiers or selection evidence, schema version, file locations and batch completion.

Keep partial uploads separate from published archives and make retries idempotent. Readers need the appropriate storage access and a policy for listing eligible files. Retrieval also depends on encryption-key availability, archive expiration and the ability to distinguish duplicated exports.

• Searchable archives need a catalog and controlled retrieval →

Know when another representation is a better fit

Frequent point updates, transactional writes and low-latency application lookups do not follow the same access pattern as scans over immutable files. Updates to a plain file archive usually require rewriting files or introducing an additional coordination layer. Irregular documents may need a preserved original representation alongside queryable fields.

Choose Parquet when its schema and read behavior suit the retention contract. Verify value fidelity and retrieval before any source cleanup, and keep backups for operational recovery. ColdCTL implements PostgreSQL, MySQL 8.4 and MongoDB 8.0 archival through separate connectors, with local Parquet, S3 and Azure Blob destinations, verification and explicitly authorized source deletion. Its customer-hosted Archive Gateway supports browser retrieval and bounded downloads. Engine-specific limits and production qualification still apply.

• ColdCTL’s database-to-Parquet workflow →
• Separate archives from recovery backups →