Disaster recovery and data migration

Basic Concepts of Disaster Recovery

MongoDB disaster recovery refers to the process of quickly restoring data and ensuring business continuity when the database encounters unexpected situations such as hardware failures, human errors, or natural disasters. The core objectives are to ensure data integrity and availability while minimizing downtime. Common disaster scenarios include:

• Server hardware failures (e.g., disk damage)
• Data center power outages or network interruptions
• Accidental deletion of important collections or documents
• Data corruption caused by malicious attacks

Backup Strategy Design

Effective backups are the foundation of disaster recovery. MongoDB offers multiple backup methods:

1. Logical Backup (mongodump/mongorestore)

// Backup the entire database
mongodump --uri="mongodb://localhost:27017" --out=/backup/2023-08

// Restore a specific collection
mongorestore --uri="mongodb://localhost:27017" --collection=users --db=production /backup/2023-08/production/users.bson

2. Physical Backup (File System Snapshots)

# Create an LVM snapshot
lvcreate --size 10G --snapshot --name mongo-snap /dev/vg0/mongo-data

3. Ops Manager/Cloud Manager (Enterprise-grade automated backup solutions)

Backup frequency should be determined based on data change frequency:

• Critical business data: Hourly incremental backups + daily full backups
• Regular data: Daily full backups
• Archival data: Weekly full backups

High Availability Mechanism of Replica Sets

MongoDB replica sets are a built-in disaster recovery solution. It is recommended to configure at least 3 nodes:

// Initialize replica set configuration
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017", priority: 2 },
    { _id: 1, host: "mongo2:27017", priority: 1 },
    { _id: 2, host: "mongo3:27017", arbiterOnly: true }
  ]
})

Failover process:

1. Primary node becomes unreachable (heartbeat timeout)
2. Secondary nodes initiate an election
3. The node with the majority of votes becomes the new primary
4. Applications automatically reconnect to the new primary

Disaster Protection for Sharded Clusters

For sharded clusters, special consideration must be given to the recovery of config servers and mongos:

// Check shard status
sh.status()

// Add a new shard as a disaster recovery node
sh.addShard("rs1/mongo4:27017,mongo5:27017,mongo6:27017")

Key protection measures:

• Config servers must be deployed as a 3-node replica set
• Each shard should be a replica set
• Maintain at least one hidden node for delayed synchronization

Data Migration Technical Solutions

1. Full Migration (Suitable for small databases)

// Export using mongodump and then import
mongodump --uri="mongodb://source:27017" --archive | mongorestore --uri="mongodb://target:27017" --archive

2. Incremental Migration (Essential for large databases)

// Use change streams to capture real-time operations
const pipeline = [{ $match: { operationType: { $in: ["insert", "update"] } } }];
const changeStream = db.collection.watch(pipeline);
changeStream.on("change", (change) => {
  // Apply changes to the target cluster
});

3. Hybrid Cloud Migration Example (On-premises to AWS)

# Use AWS Database Migration Service
aws dms create-replication-task \
  --source-endpoint-arn arn:aws:dms:us-east-1:123456789012:endpoint:SOURCE \
  --target-endpoint-arn arn:aws:dms:us-east-1:123456789012:endpoint:TARGET \
  --replication-instance-arn arn:aws:dms:us-east-1:123456789012:rep:6UTDJGBOUS3IB4HZLLEXAMPLE \
  --migration-type full-load-and-cdc

Monitoring and Automated Recovery

Establishing a comprehensive monitoring system can help detect potential issues early:

// Example using MongoDB Atlas monitoring API
const { MongoClient } = require("mongodb");
const client = new MongoClient(process.env.ATLAS_URI);
await client.connect();
const alerts = client.db("admin").collection("alerts");
alerts.watch().on("change", (change) => {
  if (change.operationType === "insert") {
    triggerRecoveryProcedure(change.fullDocument);
  }
});

Key monitoring metrics:

• Replication lag (oplog application time)
• Disk space usage
• Abnormal fluctuations in connection counts
• Query performance degradation

Practical Recovery Scenarios

Scenario 1: Accidental Collection Deletion

// Restore a single document from the oplog
use local
db.oplog.rs.find({
  ns: "shop.orders",
  op: "d",
  "o._id": ObjectId("5f4d7a9b6c3b2a1d0e8f7c6d")
}).sort({ $natural: -1 }).limit(1)

Scenario 2: Primary Node Data Corruption

# Force recovery from a healthy secondary node
mongod --dbpath /data/db --replSet rs0 --recoverFromOplog

Scenario 3: Cross-Version Migration Compatibility Issues

// Use BSON-to-JSON intermediate format to handle schema changes
mongoexport --collection=products --db=oldDB --out=products.json
mongoimport --collection=products --db=newDB --file=products.json

Performance Optimization and Resource Planning

Disaster recovery systems require sufficient resources:

1. Network bandwidth calculation:

   Required bandwidth (Mbps) = (Data volume (GB) × 8) / Allowed time window (hours) / 3600
   

2. Storage planning formula:

   Backup storage requirement = Original data × (Number of retained versions + 1) × Compression rate (typically 0.7)
   

3. Typical Recovery Time Objective (RTO):

   • Critical systems: <15 minutes
   • Important systems: <4 hours
   • Regular systems: <24 hours

Security Protection Measures

Security considerations during data migration:

// Use TLS to encrypt the migration channel
mongodump --uri="mongodb://admin:password@source:27017" --ssl --authenticationDatabase=admin

Key security practices:

• Use temporary migration accounts with the principle of least privilege
• Rotate credentials immediately after completion
• Enable audit logs throughout the process
• Implement field-level encryption for sensitive fields