Observability tools for event loops

The Node.js event loop is the core mechanism of asynchronous programming, and understanding its internal workings is crucial for performance optimization and issue troubleshooting. Observability tools can help developers delve into the details of the event loop, monitor task queue states, latency, and resource consumption, thereby enabling more efficient debugging and optimization of applications.

Basic Structure of the Event Loop

The Node.js event loop consists of multiple phases, each handling specific types of tasks. Typical phases include:

1. Timers: Executes setTimeout and setInterval callbacks
2. Pending callbacks: Processes callbacks for system operations (e.g., TCP errors)
3. Idle/Prepare: Internal-use phase
4. Poll: Retrieves new I/O events and executes related callbacks
5. Check: Executes setImmediate callbacks
6. Close callbacks: Processes callbacks for close events (e.g., socket.on('close'))

// Example: Observing execution order across phases
setImmediate(() => console.log('Check phase - setImmediate'));
setTimeout(() => console.log('Timers phase - setTimeout'), 0);

process.nextTick(() => {
  console.log('nextTick queue - executes before the event loop');
});

Built-in Observability Interfaces

Node.js provides several built-in APIs for monitoring the event loop:

process._getActiveRequests() and process._getActiveHandles()

These methods return currently active underlying resources and handles, useful for detecting resource leaks:

setInterval(() => {
  const requests = process._getActiveRequests();
  const handles = process._getActiveHandles();
  console.log(`Active requests: ${requests.length}, Active handles: ${handles.length}`);
}, 1000);

perf_hooks Module

The Performance Hooks module can measure latency across event loop phases:

const { monitorEventLoopDelay } = require('perf_hooks');

const histogram = monitorEventLoopDelay();
histogram.enable();

setInterval(() => {
  console.log(`Event loop latency(ms): 
    p50: ${histogram.percentile(50)},
    p99: ${histogram.percentile(99)}`);
}, 5000);

Third-Party Observability Tools

Clinic.js

A diagnostics toolkit developed by NearForm, consisting of three main components:

1. Clinic Doctor: Detects common performance issue patterns
2. Clinic Bubbleprof: Analyzes asynchronous flows and latency
3. Clinic Flame: Generates flame graphs to identify hotspots

Installation and usage example:

npm install -g clinic
clinic doctor -- node server.js

0x

Generates flame graphs to analyze event loop blocking:

npx 0x -o server.js

The generated flame graphs clearly show which synchronous operations are blocking the event loop.

Custom Monitoring Implementations

Developers can build their own event loop monitoring systems:

Event Loop Latency Detection

class EventLoopMonitor {
  constructor() {
    this.last = process.hrtime.bigint();
    this.delays = [];
    
    setInterval(() => {
      const now = process.hrtime.bigint();
      const delay = Number(now - this.last) / 1e6; // Convert to milliseconds
      this.delays.push(delay);
      this.last = now;
      
      if (this.delays.length > 10) {
        const avg = this.delays.reduce((a,b) => a+b) / this.delays.length;
        console.log(`Average event loop latency: ${avg.toFixed(2)}ms`);
        this.delays = [];
      }
    }, 100).unref();
  }
}

new EventLoopMonitor();

Promise Execution Tracking

const promises = new Map();

global.Promise = class TrackedPromise extends Promise {
  constructor(executor) {
    const stack = new Error().stack.split('\n').slice(2).join('\n');
    super(executor);
    promises.set(this, { createdAt: Date.now(), stack });
    
    this.finally(() => promises.delete(this));
  }
};

setInterval(() => {
  console.log(`Unresolved Promises count: ${promises.size}`);
  if (promises.size > 100) {
    console.log('Promise leak detected:');
    promises.forEach((meta, promise) => {
      console.log(`Duration: ${Date.now() - meta.createdAt}ms`);
      console.log(meta.stack);
    });
  }
}, 5000);

Production Environment Integration

OpenTelemetry Integration

Export event loop metrics to monitoring systems:

const opentelemetry = require('@opentelemetry/api');
const { MeterProvider } = require('@opentelemetry/metrics');

const meter = new MeterProvider().getMeter('event-loop-monitor');

const eventLoopLag = meter.createHistogram('event_loop_lag', {
  description: 'Event loop lag in milliseconds'
});

setInterval(() => {
  const start = process.hrtime.bigint();
  setImmediate(() => {
    const lag = Number(process.hrtime.bigint() - start) / 1e6;
    eventLoopLag.record(lag);
  });
}, 1000);

Kubernetes Health Checks

Implement intelligent health checks based on event loop state:

const http = require('http');
let eventLoopHealthy = true;

// Monitor event loop latency
setInterval(() => {
  const start = Date.now();
  setImmediate(() => {
    const delay = Date.now() - start;
    eventLoopHealthy = delay < 200; // Consider unhealthy if over 200ms
  });
}, 1000);

http.createServer((req, res) => {
  if (req.url === '/health') {
    res.statusCode = eventLoopHealthy ? 200 : 503;
    return res.end(eventLoopHealthy ? 'OK' : 'Event Loop Lagging');
  }
  // Normal request handling...
}).listen(3000);

Advanced Debugging Techniques

Blocking Operation Identification

Combine CPU profiling with event loop metrics:

const { performance, PerformanceObserver } = require('perf_hooks');

const obs = new PerformanceObserver((items) => {
  items.getEntries().forEach((entry) => {
    console.log(`Long-running operation: ${entry.name} took ${entry.duration}ms`);
  });
});
obs.observe({ entryTypes: ['function'] });

function suspectFunction() {
  performance.mark('start');
  // Simulate blocking operation
  for (let i = 0; i < 1e8; i++) Math.random();
  performance.mark('end');
  performance.measure('suspectFunction', 'start', 'end');
}

// Periodically execute suspicious function
setInterval(suspectFunction, 5000);

Microtask Queue Monitoring

let microtaskDepth = 0;

const originalThen = Promise.prototype.then;
Promise.prototype.then = function(onFulfilled, onRejected) {
  microtaskDepth++;
  console.log(`Microtask depth: ${microtaskDepth}`);
  
  return originalThen.call(this, 
    (...args) => {
      microtaskDepth--;
      return onFulfilled?.(...args);
    },
    (...args) => {
      microtaskDepth--;
      return onRejected?.(...args);
    }
  );
};

Promise.resolve().then(() => {
  return Promise.resolve().then(() => {
    console.log('Nested microtask example');
  });
});

Visualization Dashboards

Display event loop metrics using Grafana:

1. Collect metric data:

const { createServer } = require('http');
const { createClient } = require('prom-client');

const register = new createClient();
const eventLoopLag = new register.Gauge({
  name: 'node_event_loop_lag_ms',
  help: 'Current event loop lag in milliseconds'
});

setInterval(() => {
  const start = process.hrtime.bigint();
  setImmediate(() => {
    const lag = Number(process.hrtime.bigint() - start) / 1e6;
    eventLoopLag.set(lag);
  });
}, 1000);

createServer(async (req, res) => {
  if (req.url === '/metrics') {
    res.setHeader('Content-Type', register.contentType);
    res.end(await register.metrics());
  }
}).listen(3000);

2. Grafana query expression:

rate(node_event_loop_lag_ms[1m])