Optimization strategies for high-concurrency scenarios

Author：Chuan Chen 阅读数：23916人阅读分类： MongoDB

Challenges in High-Concurrency Scenarios

In high-concurrency scenarios, the main challenges faced by systems include resource contention, database connection pool exhaustion, and increased response times. Taking Mongoose as an example, when a large number of requests simultaneously access MongoDB, improper connection management can lead to a sharp decline in performance. Typical symptoms include request queuing, increased timeout errors, and even service unavailability.

Connection Pool Optimization

The default connection pool size for Mongoose is 5, which is sufficient for low-concurrency scenarios but needs adjustment for high-concurrency situations. The poolSize parameter can be used to increase the number of connections:

mongoose.connect('mongodb://localhost/test', {
  poolSize: 50, // Increase to 50 connections
  socketTimeoutMS: 30000,
  connectTimeoutMS: 30000
});

However, note that more connections are not always better. The optimal value should be determined based on server memory and load testing results. Key monitoring metrics include:

Connection wait queue length
Average connection acquisition time
Peak active connection count

Query Optimization Strategies

Index Optimization

Ensure high-frequency query fields are indexed. Analyze query execution plans using explain():

const explain = await Model.find({ status: 'active' })
  .sort({ createdAt: -1 })
  .limit(100)
  .explain('executionStats');

Focus on:

totalDocsExamined: Number of documents scanned
executionTimeMillis: Execution time
stage: Query stage type

Batch Operations Instead of Loops

Avoid executing single database operations in loops:

// Incorrect approach
for (const item of items) {
  await Model.create(item);
}

// Correct approach
await Model.insertMany(items, { ordered: false });

ordered: false allows remaining operations to continue even if some fail.

Cache Layer Implementation

Query Result Caching

Use Redis to cache high-frequency query results:

async function getProducts(category) {
  const cacheKey = `products:${category}`;
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const data = await Product.find({ category });
  await redis.setex(cacheKey, 3600, JSON.stringify(data));
  return data;
}

Cache invalidation strategies should consider:

Time-based expiration (TTL)
Active clearing upon data changes
Protection against cache breakdown

Read-Write Separation

Configure Mongoose to use different connections for read and write operations:

const readDB = mongoose.createConnection('mongodb://read1,read2/test', {
  readPreference: 'secondaryPreferred'
});

const writeDB = mongoose.createConnection('mongodb://primary/test');

Select connections based on the scenario when querying:

// Write operation
await writeDB.model('User').create(data);

// Read operation
await readDB.model('User').find({ active: true });

Bulk Write Optimization

Use bulkWrite for large-scale writes:

const ops = changes.map(change => ({
  updateOne: {
    filter: { _id: change.id },
    update: { $set: change.fields }
  }
}));

await Model.bulkWrite(ops, { ordered: false });

Parameter explanations:

ordered: false: Unordered execution; failed operations do not affect subsequent ones
bypassDocumentValidation: true: Skip document validation to improve speed

Transaction Control

For operations requiring ACID guarantees, use transactions:

const session = await mongoose.startSession();
session.startTransaction();

try {
  await Order.create([orderData], { session });
  await Inventory.updateOne(
    { productId: orderData.productId },
    { $inc: { quantity: -orderData.quantity } },
    { session }
  );
  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

Notes:

Transactions significantly reduce throughput
Use only when necessary
Control transaction duration

Connection Monitoring and Tuning

Use Mongoose's built-in events to monitor connection status:

mongoose.connection.on('connected', () => {
  console.log('MongoDB connected');
});

mongoose.connection.on('disconnected', () => {
  console.log('MongoDB disconnected');
});

mongoose.connection.on('fullsetup', () => {
  console.log('All replica set members connected');
});

Key metrics to monitor:

Connection pool wait queue length
Average query execution time
Error rate trends

Pagination Query Optimization

Avoid using skip/limit for deep pagination:

// Inefficient approach
const page = await Model.find({})
  .skip(10000)
  .limit(10);

// Efficient approach (range-based query)
const lastDoc = await Model.findById(lastId);
const page = await Model.find({ _id: { $gt: lastDoc._id } })
  .sort({ _id: 1 })
  .limit(10);

For pagination requiring random access, consider:

Precomputing pagination keys
Using materialized views
Caching popular page data

Schema Design Optimization

Appropriate Denormalization

Embed frequently accessed related data:

const userSchema = new Schema({
  name: String,
  recentOrders: [{
    orderId: Schema.Types.ObjectId,
    amount: Number,
    date: Date
  }]
});

Trade-offs to consider:

Read performance vs. write complexity
Data consistency requirements
Update frequency

Bucket Pattern

For time-series data, use bucket storage:

const metricsSchema = new Schema({
  date: Date,
  hour: Number,
  readings: [{
    timestamp: Date,
    value: Number
  }],
  stats: {
    avg: Number,
    max: Number,
    min: Number
  }
});

Advantages:

Reduces document count
Pre-aggregation improves query speed
Better data locality

Performance Testing and Monitoring

Establish a benchmark test suite:

const { performance } = require('perf_hooks');

async function runBenchmark() {
  const start = performance.now();
  await testQuery();
  const duration = performance.now() - start;
  recordMetric('query_time', duration);
}

Monitor key metrics: