Chained queries and query optimization

Basic Concepts of Chained Queries

Chained queries are a common query-building approach in Mongoose, allowing the construction of complex query conditions through consecutive method calls. This pattern is similar to jQuery's chaining, where each method call returns the query object itself, making the code more concise and readable. The core of chained queries lies in the fact that each method modifies the state of the query object, which is eventually executed via the exec() or then() method.

const User = mongoose.model('User', userSchema);

// Example of a chained query
User.find({ age: { $gt: 18 } })
  .sort({ name: 1 })
  .limit(10)
  .select('name email')
  .exec()
  .then(users => {
    console.log(users);
  })
  .catch(err => {
    console.error(err);
  });

Common Chained Query Methods

Mongoose provides a rich set of chained query methods. Here are some of the most commonly used ones:

1. Conditional Methods:
   • where(): Specifies query conditions
   • equals(): Equality condition
   • gt(), gte(), lt(), lte(): Numeric comparisons

User.where('age').gt(18).lt(30)

2. Result Processing Methods:

   • select(): Specifies fields to return
   • sort(): Sorts results
   • limit(): Limits the number of results
   • skip(): Skips a specified number of documents

3. Aggregation Methods:

   • count(): Counts matching documents
   • distinct(): Retrieves unique values of a field

Basic Principles of Query Optimization

When optimizing queries in Mongoose, several fundamental principles should be followed:

1. Filter Early: Reduce the result set size as early as possible in the query
2. Use Indexes Wisely: Ensure query conditions leverage database indexes
3. Limit Returned Fields: Select only necessary fields to reduce data transfer
4. Prefer Batch Operations: Use batch operations instead of individual operations in loops

Indexes and Query Performance

Indexes are key to query optimization. In Mongoose, indexes can be specified during schema definition:

const userSchema = new mongoose.Schema({
  username: { type: String, index: true },
  email: { type: String, unique: true },
  age: Number
});

// Compound index
userSchema.index({ username: 1, age: -1 });

For complex queries, the explain() method can analyze the query execution plan:

User.find({ age: { $gt: 25 } })
  .explain()
  .then(plan => {
    console.log(plan.executionStats);
  });

Using Query Middleware

Mongoose middleware can insert logic before or after query execution, which is helpful for optimization:

userSchema.pre('find', function(next) {
  this.start = Date.now();
  next();
});

userSchema.post('find', function(docs, next) {
  console.log(`Query duration: ${Date.now() - this.start}ms`);
  next();
});

Batch Operation Optimization

When dealing with large amounts of data, batch operations are far more efficient than individual operations:

// Not recommended
for (const user of users) {
  await new User(user).save();
}

// Recommended approach
await User.insertMany(users);

For update operations, use batch updates:

// Update all matching documents
await User.updateMany(
  { status: 'inactive' },
  { $set: { lastLogin: new Date() } }
);

Considerations for Query Caching

In certain scenarios, query caching can significantly improve performance:

const cachedUsers = await User.find({ role: 'admin' })
  .cache({ key: 'adminUsers' })
  .exec();

Note that caching requires appropriate caching strategies and invalidation mechanisms.

Building Complex Queries

For complex queries, build query conditions step by step:

const query = User.find();

if (req.query.minAge) {
  query.where('age').gte(parseInt(req.query.minAge));
}

if (req.query.maxAge) {
  query.where('age').lte(parseInt(req.query.maxAge));
}

if (req.query.sortBy) {
  query.sort(req.query.sortBy);
}

const results = await query.exec();

Virtual Fields and Queries

Virtual fields, though not stored in the database, can be used to format query results:

userSchema.virtual('fullName').get(function() {
  return `${this.firstName} ${this.lastName}`;
});

// Include virtual fields in queries
const user = await User.findOne().lean({ virtuals: true });

Optimizing Related Queries

When handling related data, use populate judiciously:

// Basic populate
await Order.find().populate('user');

// Selective populate
await Order.find().populate({
  path: 'user',
  select: 'name email',
  match: { active: true }
});

// Multi-level populate
await BlogPost.find().populate({
  path: 'comments',
  populate: {
    path: 'author',
    model: 'User'
  }
});

For performance-sensitive related queries, consider using aggregation pipelines instead of populate.

Using Aggregation Pipelines

Mongoose's aggregation pipelines offer powerful data processing capabilities:

const results = await User.aggregate([
  { $match: { age: { $gt: 21 } } },
  { $group: { 
    _id: '$city',
    total: { $sum: 1 },
    averageAge: { $avg: '$age' }
  }},
  { $sort: { total: -1 } },
  { $limit: 10 }
]);

Implementing Pagination Queries

Efficient pagination queries require attention to:

// Basic pagination
const page = parseInt(req.query.page) || 1;
const limit = parseInt(req.query.limit) || 10;
const skip = (page - 1) * limit;

const users = await User.find()
  .skip(skip)
  .limit(limit)
  .exec();

// Cursor-based pagination (for large datasets)
const cursor = User.find().cursor();
for (let i = 0; i < skip; i++) {
  await cursor.next();
}
const pageResults = [];
for (let i = 0; i < limit; i++) {
  const doc = await cursor.next();
  if (!doc) break;
  pageResults.push(doc);
}

Query Timeouts and Cancellation

Handling long-running queries:

// Set query timeout
await User.find().maxTimeMS(5000).exec();

// Cancel query (requires MongoDB 4.2+)
const controller = new AbortController();
const signal = controller.signal;

setTimeout(() => controller.abort(), 5000);

try {
  await User.find().signal(signal).exec();
} catch (err) {
  if (err.name === 'AbortError') {
    console.log('Query canceled');
  }
}

Query Logging and Monitoring

Recording and analyzing query performance:

mongoose.set('debug', function(collectionName, method, query, doc) {
  console.log(`Mongoose: ${collectionName}.${method}`, JSON.stringify(query));
});

// Or use more detailed logging
mongoose.set('debug', true);

Query Builder Pattern

For complex query conditions, use the builder pattern:

class UserQueryBuilder {
  constructor() {
    this.query = User.find();
  }

  withAgeBetween(min, max) {
    this.query.where('age').gte(min).lte(max);
    return this;
  }

  activeOnly() {
    this.query.where('active').equals(true);
    return this;
  }

  sortBy(field, direction = 1) {
    this.query.sort({ [field]: direction });
    return this;
  }

  build() {
    return this.query;
  }
}

// Usage example
const query = new UserQueryBuilder()
  .withAgeBetween(18, 30)
  .activeOnly()
  .sortBy('name')
  .build();

const results = await query.exec();

Geospatial Queries

Mongoose supports rich geospatial queries:

const placeSchema = new mongoose.Schema({
  name: String,
  location: {
    type: { type: String, default: 'Point' },
    coordinates: { type: [Number] }
  }
});

placeSchema.index({ location: '2dsphere' });

// Nearby query
const results = await Place.find({
  location: {
    $near: {
      $geometry: {
        type: 'Point',
        coordinates: [longitude, latitude]
      },
      $maxDistance: 1000 // Within 1 km
    }
  }
});

Implementing Full-Text Search

Leverage MongoDB's full-text search capabilities:

const articleSchema = new mongoose.Schema({
  title: String,
  content: String,
  tags: [String]
});

articleSchema.index({ title: 'text', content: 'text' });

// Full-text search query
const results = await Article.find(
  { $text: { $search: 'mongodb tutorial' } },
  { score: { $meta: 'textScore' } }
).sort({ score: { $meta: 'textScore' } });

Query Performance Testing

Use benchmarking tools to evaluate query performance:

const { performance } = require('perf_hooks');

async function testQueryPerformance() {
  const start = performance.now();
  
  await User.find({ age: { $gt: 30 } })
    .sort({ name: 1 })
    .limit(100)
    .exec();
    
  const duration = performance.now() - start;
  console.log(`Query duration: ${duration.toFixed(2)}ms`);
}

// Run multiple tests for average
for (let i = 0; i < 5; i++) {
  await testQueryPerformance();
}

Query Rewriting Techniques

Sometimes rewriting queries can significantly improve performance:

// Inefficient query
await User.find({
  $or: [
    { name: /^john/i },
    { email: /@example\.com$/i }
  ]
});

// Optimized query
const conditions = [];
if (searchName) conditions.push({ name: new RegExp(`^${searchName}`, 'i') });
if (searchDomain) conditions.push({ email: new RegExp(`@${searchDomain}$`, 'i') });

await User.find(conditions.length ? { $or: conditions } : {});

Query Security Considerations

Prevent query injection attacks:

// Unsafe approach
const userInput = req.query.search;
await User.find({ name: userInput });

// Safe approach
const userInput = req.query.search;
await User.find({ name: { $eq: userInput } });

// Or use escaping
const escapedInput = escapeRegex(userInput);
await User.find({ name: new RegExp(escapedInput) });

function escapeRegex(text) {
  return text.replace(/[-[\]{}()*+?.,\\^$|#\s]/g, '\\$&');
}

Processing Query Results

Optimize how query results are processed:

// Stream processing for large datasets
const stream = User.find().cursor();

stream.on('data', (doc) => {
  // Process individual document
}).on('error', (err) => {
  // Handle error
}).on('end', () => {
  // Processing complete
});

// Use lean() for performance (when Mongoose document features aren't needed)
const plainObjects = await User.find().lean();

Query Optimization in Transactions

When executing queries in transactions, be mindful of:

const session = await mongoose.startSession();
session.startTransaction();

try {
  const user = await User.findOneAndUpdate(
    { _id: userId, balance: { $gte: amount } },
    { $inc: { balance: -amount } },
    { new: true, session }
  );

  if (!user) throw new Error('Insufficient balance');

  await Transaction.create([{
    userId,
    amount: -amount,
    type: 'payment'
  }], { session });

  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

Best Practices for Query Building

Summarizing best practices for query building:

1. Chaining Order: Place filtering conditions first, followed by sorting, and finally pagination
2. Avoid Over-Chaining: Excessively long chains may reduce readability; consider breaking into steps
3. Reuse Queries: For frequently used query conditions, create query builder functions
4. Use Native Driver When Appropriate: For extremely high-performance scenarios, consider using the MongoDB native driver directly

// Example query builder function
function buildUserQuery(filters = {}) {
  let query = User.find();
  
  if (filters.age) {
    query = query.where('age').gte(filters.age.min).lte(filters.age.max);
  }
  
  if (filters.name) {
    query = query.where('name', new RegExp(filters.name, 'i'));
  }
  
  return query;
}

// Usage example
const query = buildUserQuery({
  age: { min: 18, max: 30 },
  name: 'john'
});

const results = await query.sort('name').limit(10).exec();