How to Archive Old MongoDB Documents

Decide which documents are actually historical

Your audit_events collection keeps completed activity for years, although normal application queries use only recent documents. The business still needs historical investigations. A timestamp filter can locate candidates, but it does not establish that a document can leave the collection. Define the retention field, exclude holds and active references, and decide whether old documents can still change.

The examples assume an ordinary collection of immutable event documents with a BSON Date createdAt and a unique ObjectId _id. Time-series collections, sharded workloads and custom identifier types need their own plan. Start with a disposable dataset, and keep the cutoff and the schema assumptions in a run manifest.

TTL is expiration, not an export workflow

A TTL index removes expired documents in a background process. It does not first write an archive and does not promise immediate expiration. If history must survive, coordinate existing TTL behavior with the export rather than relying on an exporter to always finish before expiration.

Consider documents missing the retention field or containing the wrong type. A range query may omit them, even though the business expects them to be retained. Quarantine or explicitly classify such documents. Do not silently interpret missing timestamps as permission to discard data.

Use a timestamp with a unique tie-breaker

ObjectId values contain a creation-time component, but they are not a reliable substitute for an application event timestamp or a strictly monotonic ingestion sequence. Late ingestion, custom IDs and client behavior can change the meaning of identifier order. Choose the business timestamp first, then use _id to break ties within that timestamp.

Evaluate a compound { createdAt: 1, _id: 1 } index and inspect the plan for your query. The first batch below has a fixed UTC cutoff and a deterministic sort. The projection names the archived fields; preserve other fields if the retention contract requires the full document.

const cutoff = ISODate("2025-01-01T00:00:00Z");
const firstBatch = db.audit_events.find(
  { createdAt: { $lt: cutoff } },
  { _id: 1, createdAt: 1, eventType: 1, payload: 1 }
).sort({ createdAt: 1, _id: 1 }).limit(1000).toArray();

Continue strictly after the completed cursor

Save createdAt and _id together only when the batch is durably exported and accepted. Increasing skip offsets requires walking past more unwanted results. A range cursor avoids that growing offset and makes the continuation boundary explicit.

The example is a subsequent read after a completed batch. With immutable selection fields it includes later timestamps and remaining documents at the same timestamp. It does not solve concurrent mutations or backdated inserts behind the cursor. Use a stable historical cohort, an appropriate read-consistency strategy, or reconciliation, and test a restart during those cases.

const lastTime = ISODate("2024-06-01T12:00:00Z");
const lastId = ObjectId("665b0d000000000000000001");
const nextBatch = db.audit_events.find({
  createdAt: { $lt: cutoff },
  $or: [
    { createdAt: { $gt: lastTime } },
    { createdAt: lastTime, _id: { $gt: lastId } }
  ]
}).sort({ createdAt: 1, _id: 1 }).limit(1000).toArray();

Preserve BSON meaning rather than just field names

An export format needs a deliberate mapping for ObjectId, Decimal128, dates, binary values, nested arrays and mixed field types. Missing fields and explicit nulls may carry different meanings. A convenient JSON conversion or automatic flattening can lose that distinction or change numerical precision.

If Parquet is the destination, define a schema mapping version and test it with representative documents and boundary values. Decide whether you need a queryable analytical representation, an exact reconstructable document, or both. Keeping an original serialized representation has different storage and query trade-offs from a normalized columnar schema.

• Parquet archive schemas and their limits →

Write, publish and verify each batch

Keep batch identities, fixed bounds, source namespace, selected identifiers and mapping version in a manifest. Finish files or uploads before marking the batch complete. A worker that loses its connection after writing must recognize the previously completed artifact rather than duplicate it or skip ahead.

Verify file readability, document counts, identifier sets and the fidelity required by the retention contract. Test retrieval through the intended query or reconstruction path. Equal counts can hide replacement or duplication, and checking a file hash cannot detect a wrong source predicate. Review rejected or malformed documents before accepting run completeness.

Treat deletion as a separate, bounded operation

An approved cleanup worker should use exact identifiers from verified manifests, not run deleteMany on the original cutoff across the whole collection. For immutable documents, a bounded verified ID set is a useful unit of work. For mutable documents, include version conditions and skip any document whose contents changed after export.

MongoDB deleteMany has no batch-size limit option comparable to a query limit. Bound the identifier list or individually guarded operations before calling it. Observe write results, replication and application load, and record deletion progress separately. Review shard-key targeting and concurrent migration behavior for a sharded collection rather than assuming the unsharded example transfers unchanged.

manifest = load_completed_verified_batch()
for bounded_group in manifest.documents:
    eligible = recheck_versions_and_retention(bounded_group)
    delete_only_verified_unchanged_documents(eligible)
    record_acknowledged_cleanup_result()
# Never substitute a broad cutoff-only deleteMany here.

Keep retrieval and backups in the design

Choose archive partitions and catalog fields around the investigations people actually perform. Record which document fields remain queryable and which can be reconstructed. A cold archive will not have the same latency, write behavior or indexing as the original collection.

The usable outcome is a verified, restartable export with a defined way to retrieve history and a separately reviewed cleanup path. Backups still handle operational recovery. ColdCTL implements PostgreSQL, MySQL 8.4 and MongoDB 8.0 archival through separate connectors, with local Parquet, S3 and Azure Blob destinations, verification and explicitly authorized source deletion. Its customer-hosted Archive Gateway supports browser retrieval and bounded downloads. Engine-specific limits and production qualification still apply.

• MongoDB archival design and capability status →
• Searchable archive architecture →