Performance optimization for large-scale projects

Performance optimization in large-scale projects is a critical factor in enhancing user experience, especially in the Express framework. Proper optimization strategies can significantly reduce server load and shorten response times. From the code level to architectural design, performance optimization requires systematic thinking and the selection of appropriate solutions based on specific scenarios.

Code-Level Optimization

Minimize Synchronous Operations

Express defaults to asynchronous I/O processing, but synchronous blocking can still be accidentally introduced during development. For example, when reading files:

// Bad example: synchronous read
const data = fs.readFileSync('large-file.json')

// Correct approach: asynchronous read
fs.readFile('large-file.json', (err, data) => {
  if (err) throw err
  app.use('/data', (req, res) => res.send(data))
})

Lazy Loading for Routes

For large routing systems, use dynamic imports to reduce memory consumption during startup:

// Example of dynamic route loading
app.get('/admin', async (req, res) => {
  const adminRoutes = await import('./routes/admin.js')
  adminRoutes.handler(req, res)
})

Middleware Optimization

Review middleware execution order and remove unnecessary global middleware:

// Before optimization: all requests go through verification
app.use(verifyToken)

// After optimization: only API routes require verification
app.use('/api', verifyToken)

Database Interaction Optimization

Batch Queries Instead of Loop Queries

Solution for the classic N+1 query problem:

// Inefficient approach
const users = await User.find()
users.forEach(async user => {
  const posts = await Post.find({ author: user.id }) // Each loop queries the database
})

// Efficient solution
const users = await User.find().populate('posts') // Single query fetches associated data

Index Optimization

Add indexes for high-frequency query fields:

// Mongoose example
const productSchema = new Schema({
  sku: { type: String, index: true }, // Single index
  category: { type: String, index: true }
})

// Compound index
productSchema.index({ category: 1, price: -1 })

Cache Strategy Implementation

Multi-Level Cache Architecture

Implement a two-level cache with memory + Redis:

const memoryCache = new Map()

async function getProduct(id) {
  // First level: memory cache
  if (memoryCache.has(id)) return memoryCache.get(id)
  
  // Second level: Redis cache
  const redisData = await redis.get(`product:${id}`)
  if (redisData) {
    memoryCache.set(id, redisData)
    return redisData
  }

  // Fallback to source
  const dbData = await Product.findById(id)
  redis.set(`product:${id}`, dbData, 'EX', 3600)
  return dbData
}

Cache Breakdown Protection

Use mutex locks to prevent cache penetration during failures:

async function getWithMutex(key, fetchFn, ttl = 60) {
  const value = await redis.get(key)
  if (value) return JSON.parse(value)

  const lockKey = `lock:${key}`
  const locked = await redis.set(lockKey, '1', 'NX', 'EX', 5)
  if (!locked) {
    await new Promise(resolve => setTimeout(resolve, 100))
    return getWithMutex(key, fetchFn, ttl)
  }

  try {
    const data = await fetchFn()
    await redis.set(key, JSON.stringify(data), 'EX', ttl)
    return data
  } finally {
    await redis.del(lockKey)
  }
}

Concurrency Optimization

Connection Pool Configuration

Proper settings for database connection pools:

// MySQL connection pool example
const pool = mysql.createPool({
  connectionLimit: 50, // Adjust based on server CPU cores
  queueLimit: 1000,    // Waiting queue length
  acquireTimeout: 3000 // Connection acquisition timeout
})

Cluster Mode Deployment

Leverage Node.js cluster module to utilize multi-core CPUs:

const cluster = require('cluster')
const os = require('os')

if (cluster.isMaster) {
  const cpuCount = os.cpus().length
  for (let i = 0; i < cpuCount; i++) {
    cluster.fork()
  }
} else {
  const app = require('./app')
  app.listen(3000)
}

Frontend Resource Optimization

Static Resource Handling

Best practices for Express static file serving:

// Production environment configuration
app.use(express.static('public', {
  maxAge: '365d',      // Long-term caching
  immutable: true,     // Immutable resources
  setHeaders: (res, path) => {
    if (path.endsWith('.br')) {
      res.setHeader('Content-Encoding', 'br')
    }
  }
}))

Smart Compression Strategy

Choose the optimal compression algorithm based on client support:

const compression = require('compression')
const zlib = require('zlib')

app.use(compression({
  threshold: 1024,
  filter: (req) => !req.headers['x-no-compression'],
  level: zlib.constants.Z_BEST_COMPRESSION,
  strategy: zlib.constants.Z_RLE
}))

Monitoring and Tuning

Performance Instrumentation

Add performance monitoring for critical paths:

app.use((req, res, next) => {
  const start = process.hrtime()
  
  res.on('finish', () => {
    const diff = process.hrtime(start)
    const duration = diff[0] * 1e3 + diff[1] * 1e-6
    metrics.track('response_time', duration)
  })

  next()
})

Memory Leak Detection

Use heapdump for memory analysis:

const heapdump = require('heapdump')

// Schedule periodic heap snapshots
setInterval(() => {
  if (process.memoryUsage().heapUsed > 500 * 1024 * 1024) {
    heapdump.writeSnapshot()
  }
}, 60000)

Architectural-Level Optimization

Microservices Split

Break monolithic applications into independent services:

// Independent deployment of product service
const productGateway = express()

productGateway.get('/api/products/:id', async (req, res) => {
  const response = await fetch(`http://product-service/${req.params.id}`)
  res.json(await response.json())
})

Read/Write Separation

Database read/write separation configuration example:

// Sequelize configuration
const sequelize = new Sequelize('database', null, null, {
  replication: {
    read: [
      { host: 'read1.example.com' },
      { host: 'read2.example.com' }
    ],
    write: { host: 'write.example.com' }
  }
})

Exception Handling

Circuit Breaker Implementation

Interface-level circuit breaker protection:

const CircuitBreaker = require('opossum')

const breaker = new CircuitBreaker(async (url) => {
  const response = await fetch(url)
  return response.json()
}, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000
})

app.get('/proxy', async (req, res) => {
  try {
    const data = await breaker.fire('http://external-service')
    res.json(data)
  } catch (err) {
    res.status(503).send('Service unavailable')
  }
})

Graceful Degradation

Fallback plan for core interfaces:

app.get('/recommend', async (req, res) => {
  try {
    const data = await recommendationService.get()
    res.json(data)
  } catch (err) {
    // Fallback to cached data
    const cached = await redis.get('fallback:recommend')
    res.json(cached || defaultRecommendations)
  }
})