Replica Set architecture

Basic Concepts of Replica Set Architecture

A MongoDB replica set is a group of mongod processes that maintain the same dataset, providing data redundancy and high availability. A replica set consists of multiple data-bearing nodes and an optional arbiter node. The data-bearing nodes are divided into a primary node (Primary) and several secondary nodes (Secondary). The primary node receives all write operations, while the secondary nodes keep their data synchronized by replicating the primary node's operation log (oplog).

The minimum recommended configuration for a replica set includes three members: one primary node and two secondary nodes. This configuration ensures that if the primary node becomes unavailable, the system can automatically elect a new primary node, achieving automatic failover. A replica set can support up to 50 members, but only 7 members can participate in voting.

// Example code for connecting to a replica set
const { MongoClient } = require('mongodb');
const uri = "mongodb://host1:port1,host2:port2,host3:port3/?replicaSet=myReplicaSet";
const client = new MongoClient(uri);

async function run() {
  try {
    await client.connect();
    const database = client.db("sampleDB");
    const collection = database.collection("sampleCollection");
    // Perform operations...
  } finally {
    await client.close();
  }
}
run().catch(console.dir);

Replica Set Member Roles and Types

Nodes in a replica set can assume different roles, each with specific functions. The primary node is the only member that receives write operations, and all secondary nodes replicate data changes from the primary node. Secondary nodes can be further categorized into several types: regular secondary nodes, hidden nodes, delayed nodes, and voting nodes (arbiters).

Hidden nodes do not participate in elections and are typically used for dedicated reporting or backup scenarios. Delayed nodes intentionally maintain a lag in data synchronization with the primary node to prevent data loss caused by human errors. Arbiter nodes do not store data and only participate in election voting, helping to break ties in cases of an even number of members.

// Example of configuring replica set members
rs.initiate({
  _id: "myReplicaSet",
  members: [
    { _id: 0, host: "mongodb1:27017", priority: 2 },
    { _id: 1, host: "mongodb2:27017", priority: 1 },
    { _id: 2, host: "mongodb3:27017", arbiterOnly: true }
  ]
})

Election Mechanism of Replica Sets

MongoDB uses a variant of the Raft consensus algorithm to implement replica set elections. When the primary node becomes unreachable, eligible secondary nodes initiate an election to become the new primary node. A successful election requires support from a majority of voting members, where "majority" refers to more than half.

Elections are triggered under conditions such as: the primary node becoming unreachable, replica set initialization, adding a new node, or configuration changes. Node priority influences election outcomes, with higher-priority nodes more likely to become the primary node. Nodes must meet the following conditions to participate in elections: maintain connectivity with a majority of members, have the latest data, and have a priority greater than 0.

// Command to view replica set status
rs.status()
// Example output
{
  "set": "myReplicaSet",
  "date": ISODate("2023-05-01T10:00:00Z"),
  "myState": 1,
  "members": [
    {
      "_id": 0,
      "name": "mongodb1:27017",
      "health": 1,
      "state": 1,
      "stateStr": "PRIMARY",
      "uptime": 1000,
      "optime": { "ts": Timestamp(1682928000, 1), "t": 1 },
      "optimeDate": ISODate("2023-05-01T00:00:00Z"),
      "lastHeartbeat": ISODate("2023-05-01T10:00:00Z"),
      "pingMs": 10
    },
    // Other member information...
  ]
}

Data Synchronization and Oplog Mechanism

Replica sets achieve data synchronization through the oplog (operation log). The oplog is a capped collection that records all operations that modify data. The primary node records write operations in its oplog, and secondary nodes asynchronously replicate and apply these operations.

Oplog size is crucial. An oplog that is too small may cause secondary nodes to fail to synchronize in time, requiring a full resynchronization. By default, the oplog size depends on the storage engine and disk space: the WiredTiger engine typically allocates 5% of available space, with a minimum of 1GB and a maximum of 50GB.

// Command to view oplog status
use local
db.oplog.rs.find().limit(1).sort({$natural:-1}).pretty()
// Example output
{
  "ts": Timestamp(1682928000, 1),
  "t": 1,
  "h": NumberLong("1234567890123456789"),
  "v": 2,
  "op": "i",
  "ns": "test.users",
  "o": {
    "_id": ObjectId("645a1b2e3c4d5e6f78901234"),
    "name": "John Doe",
    "email": "john@example.com"
  }
}

Read and Write Concerns and Consistency Guarantees

MongoDB provides different levels of read and write concerns (Read Concern/Write Concern) to control data consistency and durability. Write concern determines when a write operation is considered successful, while read concern determines the version of data returned by a read operation.

Common write concern levels include: w:1 (default, primary node acknowledgment), w:majority (majority of nodes acknowledgment), and w:"all" (all nodes acknowledgment). Read concern levels include: local (default, reads the node's latest data), available (specific to sharded clusters), majority (reads data written to a majority of nodes), and linearizable (linearizable reads).

// Example of using write concern
db.products.insertOne(
  { item: "envelope", qty: 100, type: "Clasp" },
  { writeConcern: { w: "majority", wtimeout: 5000 } }
)

// Example of using read concern
db.products.find({ item: "envelope" }).readConcern("majority")

Deployment and Maintenance of Replica Sets

Deploying a replica set requires consideration of hardware configuration, network topology, and geographic distribution. In production environments, it is recommended to deploy members in different racks or availability zones to prevent single points of failure. Maintenance operations include adding/removing members, reconfiguring priorities, and forcing reelections.

Monitoring the health status of a replica set is crucial. Key metrics include replication lag, oplog window, member status, and heartbeat latency. MongoDB provides various tools for monitoring replica sets, such as mongostat, mongotop, and Cloud Manager.

// Example of adding a new member
rs.add("mongodb4:27017")

// Example of removing a member
rs.remove("mongodb4:27017")

// Example of configuring member priority
rs.reconfig({
  _id: "myReplicaSet",
  version: 2,
  members: [
    { _id: 0, host: "mongodb1:27017", priority: 3 },
    { _id: 1, host: "mongodb2:27017", priority: 2 },
    { _id: 2, host: "mongodb3:27017", priority: 1 }
  ]
})

Fault Handling and Recovery Strategies

Replica sets may encounter various failure scenarios, such as network partitions, split-brain situations, or primary node crashes. Understanding the failure detection mechanism is important: members send heartbeats every 2 seconds, and if no response is received within 10 seconds, the member is considered unreachable.

Common troubleshooting methods include forcibly reconfiguring the replica set, manually intervening in elections, and handling rollback scenarios. Rollbacks occur when the original primary node recovers and contains write operations that were not replicated to a majority of nodes. These operations are rolled back and saved to the rollback directory.

// Forcibly reconfigure the replica set (use with caution)
rs.reconfig(config, {force: true})

// Manually trigger an election (run on the primary node)
rs.stepDown(300) // Do not participate in elections for 300 seconds

// Command to view rollback data
db.getSiblingDB("admin").aggregate([
  { $currentOp: {} },
  { $match: { "operationType": "command", "command.rollback": 1 } }
])

Relationship Between Replica Sets and Sharded Clusters

Replica sets are the foundational components of MongoDB sharded clusters. In a sharded cluster, each shard is typically a replica set, and the config servers are also replica sets. This architecture combines the advantages of horizontal scaling (sharding) and high availability (replica sets).

In a sharded cluster, the query router (mongos) automatically handles changes in replica set members, so application developers usually do not need to concern themselves with underlying topology changes. However, understanding this relationship helps in better application design and troubleshooting.

// Example of connecting to a sharded cluster
const uri = "mongodb://mongos1:27017,mongos2:27017/?replicaSet=shardRS";
const client = new MongoClient(uri);

// Command to view shard status
sh.status()
// Example output
{
  "shardingVersion": {
    "_id": 1,
    "minCompatibleVersion": 5,
    "currentVersion": 6,
    "clusterId": ObjectId("645a1b2e3c4d5e6f78901235")
  },
  "shards": [
    {
      "_id": "shard0000",
      "host": "shard1/myShard1-1:27018,myShard1-2:27018",
      "state": 1
    },
    // Other shard information...
  ]
}

Performance Optimization for Replica Sets

Optimizing replica set performance involves multiple aspects: configuring oplog size appropriately, optimizing network latency, and adjusting write concern levels. Read operations can be distributed to secondary nodes using read preferences (Read Preference) to reduce the load on the primary node.

Monitoring replication lag is critical, as excessive lag can lead to read inconsistencies and failover issues. Optimization recommendations include using dedicated network connections, ensuring similar hardware configurations for members, and avoiding long-running operations that affect replication.

// Example of setting read preference
const client = new MongoClient(uri, {
  readPreference: 'secondary',
  readPreferenceTags: [{ dc: 'east', usage: 'reporting' }]
});

// Commands to monitor replication lag
db.printReplicationInfo()
db.printSlaveReplicationInfo()

// Example output
configured oplog size:   2048MB
log length start to end: 1234567secs (342.94hrs)
oplog first event time:  Thu May 01 2023 10:00:00 GMT+0800
oplog last event time:   Thu May 15 2023 12:00:00 GMT+0800
now:                     Thu May 15 2023 12:00:01 GMT+0800

Security and Access Control

Security configurations for replica sets include enabling authentication, configuring network encryption, and setting appropriate role permissions. X.509 certificate authentication is commonly used for inter-member communication, while Kerberos authentication is suitable for enterprise environments.

Key security measures include restricting inter-member communication ports, configuring firewall rules, and regularly rotating keys. MongoDB provides built-in roles such as clusterAdmin, clusterManager, and clusterMonitor for managing replica sets.

// Example configuration for enabling replica set authentication
security:
  keyFile: /path/to/keyfile
  authorization: enabled

// Example of creating a cluster admin user
use admin
db.createUser({
  user: "clusterAdmin",
  pwd: "securePassword",
  roles: [ { role: "clusterAdmin", db: "admin" } ]
})

// Example of connecting to a replica set using X.509 authentication
const uri = "mongodb://host1:port1,host2:port2/?replicaSet=myReplicaSet&ssl=true";
const client = new MongoClient(uri, {
  tlsCertificateKeyFile: '/path/to/client.pem',
  authMechanism: 'MONGODB-X509'
});