Incremental backup and point-in-time recovery

Incremental Backup and Point-in-Time Recovery Concepts

Incremental backup is a strategy that only backs up data that has changed since the last backup. Compared to full backups, it saves storage space and backup time. Point-in-Time Recovery (PITR) allows restoring a database to its state at a specific point in time, which is crucial for recovery after data misoperations or system failures. MongoDB supports these features through the oplog and storage engine mechanisms.

Oplog Mechanism in MongoDB

The oplog (operation log) is a core component of MongoDB's replica set, recording all operations that modify data. It is a capped collection located in the local database. Oplog entries contain metadata such as operation type (insert/update/delete), namespace, document ID, and operation timestamp.

// View oplog status
rs.printReplicationInfo()
// Example output:
configured oplog size:   1024MB
log length start to end: 7423secs (2.06hrs)
oplog first event time:  Thu Oct 05 2023 10:00:00 GMT+0800
oplog last event time:   Thu Oct 05 2023 12:06:43 GMT+0800

Configuring Incremental Backup

MongoDB's official tool mongodump supports incremental backup. The key parameter --oplog records the oplog position during backup, providing a point-in-time reference for recovery.

# Full backup
mongodump --host rs0/primary.example.com:27017 --out /backup/full

# Incremental backup (12 hours after the last backup)
mongodump --host rs0/primary.example.com:27017 --oplog --out /backup/incr_$(date +%Y%m%d)

Steps for Point-in-Time Recovery

1. Restore the most recent full backup
2. Replay the oplog to the specified point in time
3. Verify data consistency

# Restore the base backup
mongorestore --host rs0/newprimary.example.com:27017 /backup/full

# Apply oplog to a specific point in time
mongorestore --host rs0/newprimary.example.com:27017 \
  --oplogReplay \
  --oplogLimit "1633000000:1" \
  /backup/incr_20231005/oplog.bson

Using Storage Engine Snapshots

The WiredTiger storage engine supports snapshot functionality, which can be used with filesystem-level tools:

# Create a snapshot (Linux LVM example)
lvcreate --size 1G --snapshot --name mongo_snap /dev/vg0/mongo_lv

# Restore from snapshot
mongod --dbpath /var/lib/mongo --wiredTigerCacheSizeGB 2 \
  --wiredTigerCollectionBlockCompressor snappy \
  --wiredTigerJournalCompressor snappy

Implementation in Cloud Environments

AWS DocumentDB and MongoDB Atlas offer managed PITR services. Atlas enables recovery to the exact second through continuous backups:

// Atlas API recovery example
const restoreOptions = {
  deliveryType: "download",
  snapshotId: "5f4d3c2b1a0f9e8d7c6b5a4",
  pointInTimeUTC: new Date("2023-10-05T12:00:00Z")
};

atlasClient.database.createRestoreJobs(restoreOptions);

Monitoring and Optimization Strategies

An effective backup system requires monitoring key metrics:

1. Oplog window time (must cover the backup interval)
2. Backup storage growth trends
3. Recovery test success rate

// Script to monitor oplog window
const oplog = db.getSiblingDB("local").oplog.rs;
const first = oplog.find().sort({$natural: 1}).limit(1).next();
const last = oplog.find().sort({$natural: -1}).limit(1).next();
const hours = (last.ts.getTime() - first.ts.getTime())/3600000;
print(`Oplog window: ${hours.toFixed(2)} hours`);

Handling Common Failure Scenarios

Scenario 1: Accidental collection deletion

# Locate the drop operation in the oplog
use local
db.oplog.rs.find({
  "op": "c",
  "ns": "test.$cmd",
  "o.drop": "users"
}).sort({$natural: -1})

Scenario 2: Data corruption

# Attempt repair using --repair option
mongod --dbpath /data/db --repair
# Then restore from backup

Example of an Automated Backup Solution

A complete automated solution might include:

// Node.js backup script example
const { execSync } = require('child_process');
const moment = require('moment');

function runBackup() {
  try {
    const date = moment().format('YYYYMMDD_HHmmss');
    const dir = `/backups/mongo/${date}`;
    
    // Perform incremental backup
    execSync(`mongodump --host replicaSet/primary:27017,secondary1:27017 \
      --oplog --out ${dir} --gzip`);
    
    // Upload to cloud storage
    execSync(`aws s3 sync ${dir} s3://backup-bucket/mongo/${date}`);
    
    // Clean up backups older than 7 days
    execSync(`find /backups/mongo -type d -mtime +7 -exec rm -rf {} +`);
  } catch (err) {
    console.error('Backup failed:', err);
    // Send alert notification
  }
}

// Execute daily at midnight
setInterval(runBackup, 24 * 60 * 60 * 1000);

Performance Impact and Trade-offs

Factors affecting production environments during incremental backups include:

1. Oplog read pressure
2. Network bandwidth usage
3. Storage I/O performance

It is recommended to perform backups on secondary nodes with the following configurations to mitigate impact:

# Excerpt from mongod.conf
storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 2
operationProfiling:
  mode: slowOp
  slowOpThresholdMs: 100
replication:
  oplogSizeMB: 2048  # Adjust based on write volume