Performance tuning and query optimization

Performance Tuning and Query Optimization

Mongoose, as the most popular MongoDB ODM in Node.js, provides powerful data modeling and query capabilities. However, in practical applications, performance issues can emerge as data volume grows and query complexity increases. Proper performance tuning and query optimization can significantly improve application responsiveness.

Index Optimization

Indexes are the core of MongoDB query performance. In Mongoose, indexes can be defined through schemas:

const userSchema = new mongoose.Schema({
  username: { type: String, index: true },
  email: { type: String, unique: true },
  createdAt: { type: Date }
});

// Compound index
userSchema.index({ username: 1, createdAt: -1 });

Common index strategies include:

1. Creating indexes for high-frequency query fields
2. Indexing sorting fields
3. Using compound indexes to cover queries
4. Avoiding indexes on low-selectivity fields

Use the explain() method to analyze query execution plans:

const explanation = await User.find({ username: 'john' })
  .explain('executionStats');
console.log(explanation.executionStats);

Query Optimization Techniques

Selective Field Projection

Querying only the required fields reduces data transfer:

// Only fetch username and email fields
const users = await User.find({}, 'username email');

Batch Operation Optimization

Use batch operations instead of looping through single operations:

// Batch insert
await User.insertMany([
  { username: 'user1' },
  { username: 'user2' }
]);

// Batch update
await User.updateMany(
  { status: 'inactive' },
  { $set: { status: 'active' } }
);

Cursor Pagination

Avoid using skip/limit for deep pagination:

// Pagination based on the last record's ID
const lastId = '...'; // ID of the last record from the previous page
const users = await User.find({ _id: { $gt: lastId } })
  .limit(10)
  .sort({ _id: 1 });

Aggregation Pipeline Optimization

Mongoose's aggregate() method supports MongoDB aggregation pipelines:

const result = await Order.aggregate([
  { $match: { status: 'completed' } },
  { $group: { 
    _id: '$customerId',
    total: { $sum: '$amount' }
  }},
  { $sort: { total: -1 } },
  { $limit: 10 }
]);

Optimization recommendations:

1. Use $match early to reduce document count
2. Use $project judiciously to limit fields
3. Avoid unnecessary $unwind stages
4. Use $facet for multi-branch aggregations

Connection Pool Configuration

Mongoose uses connection pools to manage database connections. Proper configuration improves concurrency performance:

mongoose.connect(uri, {
  poolSize: 10, // Connection pool size
  socketTimeoutMS: 30000,
  connectTimeoutMS: 30000
});

Monitor connection pool status:

const pool = mongoose.connection.getClient().s.options.pool;
console.log(pool.currentCheckedOutCount); // Currently checked-out connections

Middleware Optimization

Mongoose middleware (pre/post hooks) can become performance bottlenecks:

userSchema.pre('save', function(next) {
  // Avoid time-consuming operations in hooks
  if (this.isModified('password')) {
    this.password = hashPassword(this.password);
  }
  next();
});

Optimization recommendations:

1. Avoid network requests in middleware
2. Move complex logic to the business layer
3. Ensure proper Promise handling in async middleware

Document Design Optimization

Proper document structure design significantly improves performance:

// Embedded documents are suitable for data frequently queried together
const blogSchema = new mongoose.Schema({
  title: String,
  comments: [{
    text: String,
    author: String
  }]
});

// Referential design is suitable for independent entities
const authorSchema = new mongoose.Schema({
  name: String,
  posts: [{ type: mongoose.Schema.Types.ObjectId, ref: 'Post' }]
});

Selection strategies:

1. Use embedding for one-to-few relationships
2. Consider references for one-to-many or many-to-many relationships
3. Embedded data is suitable for frequently read but rarely updated data

Caching Strategies

Use caching judiciously to reduce database queries:

const getUser = async (userId) => {
  const cacheKey = `user:${userId}`;
  let user = await redis.get(cacheKey);
  if (!user) {
    user = await User.findById(userId).lean();
    await redis.set(cacheKey, JSON.stringify(user), 'EX', 3600);
  }
  return user;
};

Cache invalidation strategies:

1. Clear related caches during write operations
2. Set appropriate TTLs
3. Use in-memory caching for hot data

Batch Query Optimization

Use $in instead of multiple queries:

// Not recommended
const users = await Promise.all(
  userIds.map(id => User.findById(id))
);

// Recommended
const users = await User.find({
  _id: { $in: userIds }
});

Read/Write Separation

For read-heavy scenarios, use read preference settings:

mongoose.connect(uri, {
  readPreference: 'secondaryPreferred'
});

// Or for specific queries
const data = await Model.find().read('secondary');

Monitoring and Analysis

Use Mongoose debugging tools to monitor performance:

mongoose.set('debug', function(collectionName, method, query, doc) {
  console.log(`Mongoose: ${collectionName}.${method}`, query);
});

Integrate APM tools to monitor slow queries:

const mongoose = require('mongoose');
const apm = require('elastic-apm-node');

mongoose.plugin((schema) => {
  schema.post('find', function(docs) {
    apm.setTransactionName(`Mongoose: ${this.model.modelName}.find`);
  });
});

Transaction Performance Optimization

MongoDB 4.0+ supports transactions, but be mindful of performance impact:

const session = await mongoose.startSession();
session.startTransaction();

try {
  await Order.create([{ item: 'book' }], { session });
  await Inventory.updateOne(
    { item: 'book' },
    { $inc: { qty: -1 } },
    { session }
  );
  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

Optimization recommendations:

1. Keep transaction durations as short as possible
2. Avoid time-consuming operations within transactions
3. Consider optimistic concurrency control as an alternative to transactions

Sharded Cluster Optimization

For large-scale applications, sharded clusters are an option for scaling:

// Shard key selection strategy
const productSchema = new mongoose.Schema({
  sku: { type: String, required: true },
  category: { type: String, index: true }
});

// Enable sharding
sh.shardCollection("db.products", { sku: 1 });

Shard key selection principles:

1. Choose high-cardinality fields
2. Avoid monotonically increasing shard keys
3. Ensure queries can target specific shards

Performance Testing and Benchmarking

Establish performance benchmarks and continuous monitoring:

const { performance } = require('perf_hooks');

async function benchmark() {
  const start = performance.now();
  await User.find({ status: 'active' });
  const duration = performance.now() - start;
  console.log(`Query took ${duration}ms`);
}

Automated performance testing:

1. Use JMeter or k6 for load testing
2. Establish performance baselines
3. Integrate performance tests into CI/CD