MongoDB storage engines (WiredTiger, In-Memory)

Overview of MongoDB Storage Engines

MongoDB supports multiple storage engines, each optimized for different scenarios. WiredTiger is the default storage engine, offering high performance and rich features, while the In-Memory engine stores data entirely in memory, making it suitable for applications extremely sensitive to latency. The choice of storage engine directly impacts database performance, resource usage, and data persistence.

WiredTiger Storage Engine

WiredTiger is the default storage engine since MongoDB 3.2, adopting modern database design principles with the following core features:

1. Document-Level Concurrency Control: Uses MVCC (Multi-Version Concurrency Control) to prevent read and write operations from blocking each other.
2. Compression Techniques: Supports multiple compression algorithms like Snappy, zlib, and zstd.
3. Checkpoint Mechanism: Periodically flushes data from memory to disk.
4. B+ Tree Index Structure: Optimizes read and write performance.

// Specify WiredTiger engine configuration when connecting to MongoDB
const { MongoClient } = require('mongodb');
const client = new MongoClient('mongodb://localhost:27017', {
  engine: 'wiredTiger',
  wiredTiger: {
    collectionConfig: {
      blockCompressor: 'snappy' // Use Snappy compression
    },
    indexConfig: {
      prefixCompression: true // Enable index prefix compression
    }
  }
});

WiredTiger Cache Management

WiredTiger uses an internal cache to manage active datasets, with a default size of:

• 50% of (RAM - 1GB)
• Or 256MB (whichever is larger)

It can be adjusted as follows:

# Specify cache size when starting mongod
mongod --wiredTigerCacheSizeGB 2

Compression Strategy Examples

WiredTiger supports separate compression configurations for collections and indexes:

In-Memory Storage Engine

The MongoDB Enterprise Edition provides the In-Memory storage engine, with features including:

1. Zero Disk I/O: All data resides in memory.
2. Predictable Performance: Consistently low latency.
3. No Persistence: Data is lost after server restart.
4. Ideal Use Cases:
   • Real-time analytics
   • Session storage
   • High-speed caching

Configuring the In-Memory Engine

// Example configuration for the In-Memory engine
const inMemoryConfig = {
  storage: {
    engine: 'inMemory',
    inMemory: {
      engineConfig: {
        inMemorySizeGB: 4 // Allocate 4GB of memory
      }
    }
  }
};

Performance Comparison Test

Under the same hardware conditions (16-core CPU, 32GB RAM):

Storage Engine Selection Guide

Scenarios for Choosing WiredTiger

1. Data persistence is required.
2. Working set exceeds available memory.
3. Balanced read and write performance is needed.
4. Migrating from MMAPv1.

Scenarios for Choosing In-Memory

1. Extremely low latency (<1ms) is required.
2. Data can be rebuilt or is temporary.
3. Sufficient memory resources are available.
4. Used as a caching layer.

Advanced Configuration Tips

WiredTiger Tuning Parameters

# Example mongod.yml configuration
storage:
  wiredTiger:
    engineConfig:
      cacheSizeGB: 2
      journalCompressor: snappy
    collectionConfig:
      blockCompressor: zstd
    indexConfig:
      prefixCompression: true

In-Memory Engine Monitoring

Key monitoring metrics:

• mem.resident: Resident memory size.
• mem.virtual: Virtual memory usage.
• metrics.cursor.open: Number of open cursors.
• metrics.operation.latency: Operation latency.

// Monitor memory usage with the MongoDB Node.js driver
db.serverStatus().then(stats => {
  console.log('Memory usage:', stats.mem);
  console.log('Operations latency:', stats.metrics.operation.latency);
});

Practical Use Cases

E-Commerce Inventory System

// Using the In-Memory engine for flash sale inventory
const inventoryDB = client.db('inventory', {
  engine: 'inMemory',
  inMemory: { sizeGB: 2 }
});

// Atomic stock reduction operation
async function reduceStock(productId, quantity) {
  const session = client.startSession();
  try {
    session.startTransaction();
    const product = await inventoryDB.collection('products').findOne(
      { _id: productId, stock: { $gte: quantity } },
      { session }
    );
    if (!product) throw new Error('Insufficient stock');
    
    await inventoryDB.collection('products').updateOne(
      { _id: productId },
      { $inc: { stock: -quantity } },
      { session }
    );
    
    await session.commitTransaction();
    return true;
  } catch (error) {
    await session.abortTransaction();
    return false;
  } finally {
    session.endSession();
  }
}

IoT Time-Series Data Processing

// Using WiredTiger for device sensor data
const iotDB = client.db('iot', {
  engine: 'wiredTiger',
  wiredTiger: {
    collectionConfig: {
      blockCompressor: 'zstd'
    }
  }
});

// Create a time-series collection
await iotDB.createCollection('sensor_data', {
  timeseries: {
    timeField: 'timestamp',
    metaField: 'sensor_id',
    granularity: 'hours'
  },
  storageEngine: {
    wiredTiger: {
      configString: 'block_compressor=zstd'
    }
  }
});

Hybrid Usage Patterns

In production environments, both engines can be used together:

// Primary database uses WiredTiger for persistence
const primaryDB = client.db('primary', {
  engine: 'wiredTiger'
});

// Reporting database uses In-Memory for faster queries
const reportingDB = client.db('reporting', {
  engine: 'inMemory'
});

// Set up a change stream to sync data to the in-memory database
primaryDB.collection('orders').watch().on('change', (change) => {
  reportingDB.collection('orders_cache').bulkWrite([
    {
      replaceOne: {
        filter: { _id: change.documentKey._id },
        replacement: change.fullDocument,
        upsert: true
      }
    }
  ]);
});

Performance Optimization Practices

WiredTiger Index Optimization

1. Use covering indexes to reduce I/O:

// Create compound indexes
await db.collection('users').createIndexes([
  { name: 1, email: 1 },
  { location: "2dsphere" }
]);

// Query using a covering index
const users = await db.collection('users').find(
  { name: /^张/ },
  { projection: { _id: 0, name: 1, email: 1 } }
).toArray();

In-Memory Engine Memory Management

1. Set reasonable memory limits:

mongod --storageEngine inMemory --inMemorySizeGB 8

2. Monitor memory usage:

setInterval(async () => {
  const status = await db.command({ serverStatus: 1 });
  const memoryUsage = (status.mem.resident / status.mem.available * 100).toFixed(2);
  console.log(`Memory usage: ${memoryUsage}%`);
}, 5000);