Integration and configuration of the logging system

A logging system is an indispensable part of web application development, helping developers record critical information during runtime for debugging and monitoring purposes. Koa2, as a lightweight Node.js framework, can flexibly integrate logging functionality through its middleware mechanism. This article will provide a detailed discussion on the configuration and integration of logging systems.

Basic Concepts of Logging Systems

Logging systems are typically divided into several levels, including debug, info, warn, error, etc., with each level used to record information of varying importance. In Koa2, middleware can intercept requests and responses to log relevant information.

A typical log entry includes the following information:

• Request timestamp
• Request method (GET, POST, etc.)
• Request path
• Response status code
• Response time
• Error message (if any)

Common Logging Library Options

In the Node.js ecosystem, several mature logging libraries are available:

1. winston: Powerful, supports multiple transport methods
2. pino: High-performance JSON logger
3. log4js: Flexible configuration, supports multiple output formats
4. morgan: Middleware specifically designed for HTTP request logging

Here’s a basic winston configuration example:

const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  transports: [
    new winston.transports.File({ filename: 'error.log', level: 'error' }),
    new winston.transports.File({ filename: 'combined.log' })
  ]
});

if (process.env.NODE_ENV !== 'production') {
  logger.add(new winston.transports.Console({
    format: winston.format.simple()
  }));
}

Integrating Logging Middleware in Koa2

Koa2’s middleware mechanism makes logging integration straightforward. Below is an example of a custom logging middleware:

async function loggerMiddleware(ctx, next) {
  const start = Date.now();
  try {
    await next();
    const ms = Date.now() - start;
    console.log(`${ctx.method} ${ctx.url} - ${ms}ms`);
  } catch (err) {
    const ms = Date.now() - start;
    console.error(`${ctx.method} ${ctx.url} - ${ms}ms - ERROR: ${err.message}`);
    throw err;
  }
}

app.use(loggerMiddleware);

Using Morgan for HTTP Request Logging

Morgan is middleware specifically designed for HTTP request logging and can be easily integrated into Koa2:

const Koa = require('koa');
const morgan = require('morgan');
const app = new Koa();

// Convert Morgan's Express middleware to Koa middleware
app.use(async (ctx, next) => {
  return new Promise((resolve) => {
    morgan('combined')(ctx.req, ctx.res, () => {
      resolve(next());
    });
  });
});

Morgan supports several predefined log formats:

• combined: Standard Apache combined log format
• common: Apache common log format
• dev: Colorful output for development
• short: Short format
• tiny: Minimal format

Advanced Logging Configuration

For production environments, more complex logging configurations are typically required:

1. Log rotation: Split logs by date or size
2. Log level filtering: Use different log levels for different environments
3. Log formatting: Customize log output formats
4. Multiple output targets: Output to both files and consoles

Here’s an advanced configuration example:

const { createLogger, format, transports } = require('winston');
const { combine, timestamp, printf } = format;

const myFormat = printf(({ level, message, timestamp }) => {
  return `${timestamp} [${level}]: ${message}`;
});

const logger = createLogger({
  level: process.env.NODE_ENV === 'production' ? 'info' : 'debug',
  format: combine(
    timestamp(),
    myFormat
  ),
  transports: [
    new transports.File({ 
      filename: 'logs/error.log', 
      level: 'error',
      maxsize: 5242880, // 5MB
      maxFiles: 5
    }),
    new transports.File({ 
      filename: 'logs/combined.log',
      maxsize: 5242880,
      maxFiles: 5
    }),
    new transports.Console()
  ]
});

Special Handling for Error Logs

In Koa2, error handling is typically implemented via middleware. Here’s an example of error logging:

app.use(async (ctx, next) => {
  try {
    await next();
  } catch (err) {
    logger.error({
      message: err.message,
      stack: err.stack,
      status: err.status || 500,
      method: ctx.method,
      url: ctx.url,
      headers: ctx.headers
    });
    
    ctx.status = err.status || 500;
    ctx.body = {
      error: process.env.NODE_ENV === 'development' ? 
        { message: err.message, stack: err.stack } : 
        { message: 'Internal Server Error' }
    };
  }
});

Logging with Contextual Information

In web applications, it’s often necessary to log request-related contextual information, such as user IDs or request IDs. This can be achieved using Koa2’s context object:

app.use(async (ctx, next) => {
  const start = Date.now();
  ctx.logger = logger.child({ 
    requestId: Math.random().toString(36).substring(2, 15),
    userId: ctx.state.user ? ctx.state.user.id : 'anonymous'
  });
  
  try {
    await next();
    const ms = Date.now() - start;
    ctx.logger.info(`${ctx.method} ${ctx.url} - ${ms}ms`);
  } catch (err) {
    const ms = Date.now() - start;
    ctx.logger.error(`${ctx.method} ${ctx.url} - ${ms}ms - ERROR: ${err.message}`);
    throw err;
  }
});

Performance Considerations

While logging is important, improper implementation can impact application performance:

1. Asynchronous logging: Avoid synchronous I/O operations blocking the event loop
2. Batch writing: Reduce disk I/O operations
3. Log level control: Disable unnecessary log levels in production
4. Sampling: Sample high-frequency logs

The pino library excels in performance and is particularly suitable for high-concurrency scenarios:

const pino = require('pino');
const logger = pino({
  level: process.env.NODE_ENV === 'production' ? 'info' : 'debug',
  formatters: {
    level(label) {
      return { level: label };
    }
  }
});

// Koa2 integration
app.use(async (ctx, next) => {
  ctx.log = logger.child({
    requestId: Math.random().toString(36).substring(2, 15)
  });
  await next();
});

Log Visualization and Analysis

For production environments, logs often need to be centralized and analyzed:

1. ELK Stack (Elasticsearch, Logstash, Kibana)
2. Splunk
3. Graylog
4. AWS CloudWatch Logs

Here’s an example of sending logs to Elasticsearch:

const { Client } = require('@elastic/elasticsearch');
const ecsFormat = require('@elastic/ecs-winston-format');
const { ElasticsearchTransport } = require('winston-elasticsearch');

const esClient = new Client({ node: 'http://localhost:9200' });

const esTransport = new ElasticsearchTransport({
  level: 'info',
  client: esClient,
  index: 'app-logs'
});

const logger = winston.createLogger({
  format: ecsFormat(),
  transports: [esTransport]
});

Security Considerations

Logging requires attention to security:

1. Sensitive information filtering: Avoid logging passwords, tokens, etc.
2. Access control: Ensure proper log file permissions
3. Data protection: Comply with regulations like GDPR
4. Log cleanup: Regularly purge outdated logs

Here’s an example of sensitive information filtering:

app.use(async (ctx, next) => {
  const sanitizedHeaders = { ...ctx.headers };
  if (sanitizedHeaders.authorization) {
    sanitizedHeaders.authorization = '***';
  }
  if (sanitizedHeaders.cookie) {
    sanitizedHeaders.cookie = sanitizedHeaders.cookie
      .split(';')
      .map(c => c.trim().startsWith('token=') ? 'token=***' : c)
      .join(';');
  }
  
  ctx.sanitizedHeaders = sanitizedHeaders;
  await next();
});

// Use sanitizedHeaders in logging middleware
logger.info({
  method: ctx.method,
  url: ctx.url,
  headers: ctx.sanitizedHeaders
});

Logging Configuration for Testing Environments

Testing environments often require special logging configurations:

1. More detailed log levels: For debugging
2. Colorized output: Improved readability
3. Structured logs: Facilitates automated test analysis
4. In-memory logging: For unit tests

const { createLogger, transports, format } = require('winston');
const { colorize, simple } = format;

const testLogger = createLogger({
  level: 'debug',
  format: format.combine(
    colorize(),
    simple()
  ),
  transports: [new transports.Console()]
});

// Use memory transport in tests
const memoryTransport = new transports.MemoryTransport();
testLogger.add(memoryTransport);

// Check logs in tests
assert(memoryTransport.errorOutput.some(log => log.includes('expected error')));

Multi-Environment Logging Strategies

Different environments should adopt different logging strategies:

1. Development environment:

   • Colorized console output
   • Detailed log levels (debug)
   • Structured logs

2. Testing environment:

   • File and console output
   • Info level
   • Includes test-related metadata

3. Production environment:

   • File/remote logging services only
   • Warn level and above
   • Log rotation and archiving
   • Performance-optimized configuration

function createLoggerForEnv(env) {
  const commonFormats = format.combine(
    format.timestamp(),
    format.errors({ stack: true })
  );
  
  switch(env) {
    case 'development':
      return createLogger({
        level: 'debug',
        format: format.combine(
          commonFormats,
          format.colorize(),
          format.simple()
        ),
        transports: [new transports.Console()]
      });
      
    case 'production':
      return createLogger({
        level: 'info',
        format: format.combine(
          commonFormats,
          format.json()
        ),
        transports: [
          new transports.File({ filename: 'logs/app.log' }),
          new transports.File({ filename: 'logs/error.log', level: 'error' })
        ]
      });
      
    default:
      return createLogger({
        level: 'info',
        format: format.combine(
          commonFormats,
          format.json()
        ),
        transports: [new transports.Console()]
      });
  }
}

Custom Log Formats

Custom log formats can be created based on business needs:

const { format } = require('winston');

const businessFormat = format.printf((info) => {
  const { timestamp, level, message, ...meta } = info;
  let log = `${timestamp} [${level}] ${message}`;
  
  if (meta.userId) {
    log += ` user:${meta.userId}`;
  }
  if (meta.transactionId) {
    log += ` tx:${meta.transactionId}`;
  }
  if (meta.durationMs) {
    log += ` duration:${meta.durationMs}ms`;
  }
  
  return log;
});

const logger = createLogger({
  format: format.combine(
    format.timestamp(),
    businessFormat
  ),
  transports: [new transports.Console()]
});

Integrating Logging with Monitoring Systems

Logging systems can be integrated with monitoring systems for alerts and metrics collection:

1. Prometheus: Expose metrics via logs
2. StatsD: Send performance metrics
3. Sentry: Error monitoring and alerts

const StatsD = require('hot-shots');
const dogstatsd = new StatsD();

app.use(async (ctx, next) => {
  const start = Date.now();
  try {
    await next();
    const ms = Date.now() - start;
    dogstatsd.increment('requests.total');
    dogstatsd.histogram('response_time', ms, {
      method: ctx.method,
      path: ctx.path,
      status: ctx.status
    });
  } catch (err) {
    dogstatsd.increment('requests.errors');
    throw err;
  }
});

Logging Performance Optimization Tips

1. Use child loggers: Avoid repeatedly creating logger instances
2. Batch writing: Use setImmediate or batch transports
3. Avoid synchronous operations: Ensure all transports are asynchronous
4. Set appropriate log levels: Reduce unnecessary logs in production
5. Use high-performance libraries: Such as pino

// Bad practice - Create new logger per request
app.use(async (ctx, next) => {
  const logger = createLogger({ /* config */ });
  // ...
});

// Good practice - Use child logger
const baseLogger = createLogger({ /* config */ });

app.use(async (ctx, next) => {
  ctx.log = baseLogger.child({ requestId: generateId() });
  // ...
});