Retain old MongoDB documents outside the working set.

TTL expires documents; archiving retains them

MongoDB TTL indexes remove expired documents through a background process. They do not export a retained copy first, and expiration does not guarantee immediate deletion. TTL is useful when a record can cease to exist after its lifetime. Archiving fits records that must remain retrievable after leaving the collection.

An existing TTL policy can race an exporter. If retention requires an archive, coordinate expiry with verified completion rather than assuming the exporter will always catch up. Account for failed jobs, late documents and holds when choosing the policy.

Scan a stable timestamp and a unique tie-breaker

Use an application timestamp with an explicit meaning: event occurrence, completion or ingestion. ObjectId order can help pagination, but its timestamp is not a substitute for the business retention field. Custom identifiers and late ingestion make that distinction especially important.

For immutable createdAt values and ObjectId identifiers, evaluate a compound (createdAt, _id) index with explain. The mongosh example is a read-only subsequent-batch query. Persist both cursor values; a timestamp alone can skip documents sharing that timestamp. Growing skip offsets are a poor default for a long export.

const cutoff = ISODate("2025-01-01T00:00:00Z");
const lastTime = ISODate("2024-06-01T12:00:00Z");
const lastId = ObjectId("665b0d000000000000000001");
db.audit_events.find({
  createdAt: { $lt: cutoff },
  $or: [
    { createdAt: { $gt: lastTime } },
    { createdAt: lastTime, _id: { $gt: lastId } }
  ]
}).sort({ createdAt: 1, _id: 1 }).limit(1000);

Preserve document meaning in Parquet

Collections can contain nested arrays, missing fields, explicit nulls and values with different types under the same field name. Decide whether to normalize into a documented schema, preserve an original representation, or reject incompatible documents for review. Flattening every document without a mapping contract can lose information.

Test ObjectId, Decimal128, dates, binary data and nested structures through the intended reader. Store the mapping version and source namespace with each export. Decide whether restoration requires the original BSON document or whether a defined analytical representation is sufficient.

• Understand Parquet schemas and trade-offs

Verify before any batch deletion

Capture fixed selection bounds and make the source consistency assumptions explicit. A cursor is not a snapshot: updates, missing timestamps and backdated inserts can change eligibility during a run. Use stable completed documents, an appropriate consistency mechanism, or reconciliation to establish a complete record set.

Verify readable files, counts, identifiers and mapped values, then test a retrieval path. ColdCTL deletion targets verified ObjectIds and compares exact BSON in snapshot transactions with majority commit. Bound the work, observe replication and application load, and stop when verification or eligibility checks fail. The connector requires quiescent-copy and rejects unsupported topologies and oversized documents; see the current database limits.

Plan how historical documents will be found

Time partitions and archive metadata should narrow an investigation to a known dataset and period. Document the fields users can query and which original values can be reconstructed. A cold archive will have a different latency and concurrency profile from a live MongoDB collection.

ColdCTL’s gateway uses Parquet and DuckDB inside customer-controlled infrastructure. MongoDB archives retain a raw-BSON envelope: supported envelope fields can be filtered, but arbitrary nested document fields cannot. Test that this retrieval contract answers your investigation needs before expanding the workflow.

• Read the searchable-archive design