Shard balancer and data migration

In MongoDB, the shard balancer (Balancer) is a critical component that ensures even data distribution by automatically migrating chunks to optimize cluster performance. Data migration is the core operation to achieve this goal, involving cross-shard chunk movement, metadata updates, and load balancing strategies.

How the Shard Balancer Works

The shard balancer runs as part of mongos and is enabled by default. It achieves balancing through the following steps:

1. Monitoring Shard Status: Periodically checks the chunk count differences between shards
2. Triggering Migration Conditions: When the difference between the maximum and minimum chunk counts exceeds the threshold (default: 2)
3. Selecting Migration Chunks: Prioritizes chunks with clear boundaries for migration

// Example of checking balancer status  
sh.getBalancerState()  
// Example return: { "mode" : "full", "inBalancerRound" : false }  

Detailed Process of Data Migration

Data migration occurs in three phases:

1. Initialization Phase:

   • The source shard initiates the migration process
   • Creates a temporary collection for data transfer
   • Records the oplog position at the start of migration

2. Data Transfer Phase:

   • Bulk copies documents to the target shard
   • Continuously synchronizes new oplog records
   • Manually triggers migration using commands like:

   db.adminCommand({ moveChunk: "test.users",   
                    find: { _id: 1000 },   
                    to: "shard2" })  
   

3. Commit Phase:

   • Updates metadata on the config server
   • The source shard deletes migrated data
   • The target shard builds necessary indexes

Migration Strategies and Performance Impact

MongoDB employs multiple strategies to optimize migration efficiency:

Parallel Migration:

• Allows 2 concurrent migrations by default
• Can be adjusted with:

use config  
db.settings.update(  
  { _id: "balancer" },  
  { $set: { "_secondaryThrottle" : true } },  
  { upsert: true }  
)  

Migration Window Configuration:

db.settings.update(  
  { _id: "balancer" },  
  { $set: { activeWindow :   
           { start : "23:00", stop : "6:00" } } },  
  { upsert: true }  
)  

Common Issues and Solutions

Migration Stalling:

• Check network latency: Observe migration operations with db.currentOp()
• Adjust chunk size (default: 64MB):

use config  
db.settings.save({ _id:"chunksize", value: 32 })  

Imbalanced Hotspots:

• Monotonically increasing shard keys may create "hot shards"
• Solution example using hashed sharding:

sh.shardCollection("test.logs", { _id: "hashed" })  

Advanced Monitoring Techniques

Use aggregation pipelines to analyze migration history:

use config  
db.changelog.aggregate([  
  { $match: { what: "moveChunk.start" } },  
  { $group: {   
      _id: "$details.from",   
      count: { $sum: 1 }   
  } }  
])  

Key Metrics to Monitor:

• Migration queue length: sh.status(true).balancer
• Migration duration: db.changelog.find({ what: /moveChunk/ })
• Disk space fluctuations: db.serverStatus().storageEngine