Replica set monitoring and troubleshooting

Replica Set Monitoring Basics

MongoDB replica sets maintain multiple data copies to provide high availability. Monitoring replica set status is a critical task for ensuring system stability. The rs.status() command returns a document containing detailed data such as member status, optime, and heartbeat information. A typical output includes the following core fields:

{
  "set": "replSet",
  "date": ISODate("2023-08-20T08:00:00Z"),
  "members": [
    {
      "_id": 0,
      "name": "mongo1:27017",
      "health": 1,
      "state": 1,
      "stateStr": "PRIMARY",
      "uptime": 86400,
      "optime": { "ts": Timestamp(1692528000, 1) },
      "optimeDate": ISODate("2023-08-20T07:20:00Z"),
      "pingMs": 2
    }
  ]
}

Health checks should focus on health (1 indicates healthy), stateStr (PRIMARY/SECONDARY), and optime differences. Replication lag can be detected by periodically comparing the optime of nodes:

const primaryOptime = rs.status().members[0].optime.ts;
const secondaryOptime = rs.status().members[1].optime.ts;
const lag = primaryOptime - secondaryOptime;
if (lag > 60) console.warn(`Replication lag exceeds 60 seconds: ${lag}`);

Key Metrics Monitoring System

Operation Counters

The opcounters module returned by db.serverStatus() records CRUD operation volumes. Sudden write surges may cause replication backlogs:

const preStats = db.serverStatus().opcounters;
// After 1 hour
const postStats = db.serverStatus().opcounters;
const writeIncrease = postStats.insert - preStats.insert;
if (writeIncrease > 10000) alert("Write surge warning");

Replication Buffer Monitoring

The replicationBuffer field in the replSetGetStatus command shows memory buffer usage. Buffer overflow triggers slower disk synchronization:

"replicationBuffer": {
  "sizeBytes": 104857600,
  "count": 423,
  "maxSizeBytes": 1073741824
}

Network Latency Detection

Inter-node latency is reflected in pingMs. Persistent high latency may lead to election timeouts:

rs.status().members.forEach(member => {
  if (member.pingMs > 500) {
    console.error(`High network latency: ${member.name} ${member.pingMs}ms`);
  }
});

Handling Typical Failure Scenarios

Primary Node Unavailable

When rs.status() shows health:0 for the primary node, the replica set triggers an election within 30 seconds. Confirm the duration of the failure before manual intervention:

const primaryState = rs.status().members.find(m => m.stateStr === "PRIMARY");
if (!primaryState || primaryState.health === 0) {
  setTimeout(() => {
    if (rs.status().members.every(m => m.stateStr !== "PRIMARY")) {
      rs.stepDown(60); // Force re-election
    }
  }, 30000);
}

Broken Replication Chain

In a cascading replication structure, a middle node failure can cause downstream nodes to lose connection. Check the replication link via the syncingTo field:

const topology = rs.status().members.map(m => ({
  node: m.name,
  source: m.syncingTo || "none"
}));
// Visually inspect for breakpoints

Rollback Data Recovery

Primary node demotion may cause data rollback. BSON files in the rollback directory contain unreplicated operations:

# View rollback files
ls /data/db/rollback/replSetName/
# Apply rollback
mongorestore --db=test --collection=users /data/db/rollback/replSetName/users.0.bson

Advanced Diagnostic Tools

Replication Stream Analysis

Enable the replSetMonitor log component to obtain detailed replication events:

db.adminCommand({
  setParameter: 1,
  logComponentVerbosity: {
    replication: { verbosity: 2 },
    "replSet": { verbosity: 3 }
  }
});

Oplog Window Calculation

An oplog time window that is too small may prevent full synchronization after prolonged downtime:

const oplog = db.getSiblingDB("local").oplog.rs;
const first = oplog.find().sort({$natural: 1}).limit(1).next();
const last = oplog.find().sort({$natural: -1}).limit(1).next();
const windowHours = (last.ts - first.ts) / 3600;
if (windowHours < 24) console.error("Oplog window less than 24 hours");

Heartbeat Packet Analysis

Adjust heartbeat intervals and timeout thresholds to optimize for unstable network environments:

cfg = rs.conf();
cfg.settings.heartbeatIntervalMillis = 2000;
cfg.settings.heartbeatTimeoutSecs = 10;
rs.reconfig(cfg);

Automated Monitoring Implementation

Prometheus Metrics Export

Use mongodb_exporter to collect key metrics:

# prometheus.yml configuration example
scrape_configs:
  - job_name: 'mongodb'
    static_configs:
      - targets: ['mongodb_exporter:9216']

Custom Alert Rules

Set up replication lag alerts in Grafana:

{
  "alert": "HighReplicationLag",
  "expr": "mongodb_replset_oplog_replication_lag > 30",
  "for": "5m",
  "annotations": {
    "description": "Replication lag exceeds 30 seconds"
  }
}

Node Status Dashboard

The MongoDB Atlas replica set monitoring interface includes:

• Real-time election counts
• Replication lag heatmap for each node
• Operation type distribution
• Network throughput trends

Performance Optimization Practices

Read/Write Concern Configuration

Appropriate writeConcern settings balance safety and performance:

// Ensure operations are replicated to majority nodes
db.products.insert(
  { sku: "xyz123", qty: 250 },
  { writeConcern: { w: "majority", wtimeout: 5000 } }
);

Index Synchronization Verification

Ensure all nodes have identical indexes:

const primaryIndexes = db.getSiblingDB("admin").runCommand({
  listIndexes: "products",
  $readPreference: { mode: "primary" }
});
const secondaryIndexes = db.getSiblingDB("admin").runCommand({
  listIndexes: "products",
  $readPreference: { mode: "secondary" }
});
assert.eq(primaryIndexes.cursor.firstBatch.length, secondaryIndexes.cursor.firstBatch.length);

Bulk Operation Optimization

Use ordered bulk operations for large writes to reduce replication pressure:

const bulk = db.items.initializeOrderedBulkOp();
for (let i = 0; i < 10000; i++) {
  bulk.insert({ item: `product-${i}` });
}
bulk.execute({ writeConcern: { w: 2 } });