Implementation of logging and auditing functions

Implementation of Logging and Auditing Functions

Logging and auditing functions are crucial components of backend systems, especially in ODM libraries like Mongoose. Proper logging design can effectively track data operations. By recording key actions and exceptions, developers can quickly identify issues while meeting compliance requirements.

Log Levels and Categories

Implementing logging in Mongoose first requires defining log levels:

const LogLevel = {
  DEBUG: 0,    // Development debugging information
  INFO: 1,     // Routine operation records
  WARN: 2,     // Potential issue warnings
  ERROR: 3,    // Error events
  AUDIT: 4     // Special level for auditing
};

Common log categories include:

• Operation logs: Record CRUD operations
• System logs: Record service start/stop events
• Security logs: Record sensitive operations like login authentication
• Audit logs: Special records for compliance requirements

Mongoose Plugin Implementation

Logging functionality can be uniformly added via Mongoose plugins:

function auditLogPlugin(schema) {
  schema.post('init', doc => {
    doc._originalData = doc.toObject();
  });

  schema.pre('save', function(next) {
    this._modifiedPaths = this.modifiedPaths();
    next();
  });

  schema.post('save', function(doc) {
    if (this._modifiedPaths.length > 0) {
      const changes = this._modifiedPaths.map(path => ({
        field: path,
        from: this._originalData[path],
        to: doc[path]
      }));
      AuditLog.create({
        model: doc.constructor.modelName,
        docId: doc._id,
        operation: 'update',
        changes,
        timestamp: new Date()
      });
    }
  });
}

Audit Trail Implementation

Deep auditing requires recording complete pre- and post-change states:

const auditSchema = new mongoose.Schema({
  operation: { type: String, enum: ['create', 'update', 'delete'] },
  model: String,
  docId: mongoose.Types.ObjectId,
  before: mongoose.Schema.Types.Mixed,
  after: mongoose.Schema.Types.Mixed,
  modifiedBy: { type: mongoose.Types.ObjectId, ref: 'User' },
  timestamp: { type: Date, default: Date.now }
}, { capped: { size: 1024*1024*10, max: 10000 } });

// Implement real-time auditing using change streams
const changeStream = Model.watch();
changeStream.on('change', change => {
  AuditLog.create({
    operation: change.operationType,
    model: change.ns.coll,
    docId: change.documentKey._id,
    before: change.updateDescription?.updatedFields,
    after: change.fullDocument
  });
});

Sensitive Data Masking

Sensitive fields in audit logs require special handling:

function maskSensitiveData(data) {
  const sensitiveFields = ['password', 'creditCard', 'ssn'];
  return JSON.parse(JSON.stringify(data, (key, value) => {
    return sensitiveFields.includes(key) 
      ? '***MASKED***' 
      : value;
  }));
}

userSchema.post('save', function(doc) {
  AuditLog.create({
    operation: 'update',
    model: 'User',
    docId: doc._id,
    before: maskSensitiveData(this._originalData),
    after: maskSensitiveData(doc.toObject())
  });
});

Performance Optimization Strategies

Consider performance impact for extensive logging:

1. Batch write optimization:

const logQueue = [];
setInterval(() => {
  if(logQueue.length > 0) {
    AuditLog.insertMany(logQueue.splice(0, 100));
  }
}, 5000);

2. Index optimization:

auditSchema.index({ docId: 1 });
auditSchema.index({ timestamp: -1 });
auditSchema.index({ model: 1, operation: 1 });

3. Log tiered storage:

const logSchema = new mongoose.Schema({
  // ...field definitions
}, { 
  timeseries: {
    timeField: 'timestamp',
    metaField: 'metadata',
    granularity: 'hours'
  }
});

Query and Analysis Interface

Provide powerful log query capabilities:

router.get('/audit', async (req, res) => {
  const { model, docId, startDate, endDate, operation } = req.query;
  const query = {};
  
  if(model) query.model = model;
  if(docId) query.docId = docId;
  if(operation) query.operation = operation;
  if(startDate || endDate) {
    query.timestamp = {};
    if(startDate) query.timestamp.$gte = new Date(startDate);
    if(endDate) query.timestamp.$lte = new Date(endDate);
  }

  const logs = await AuditLog.find(query)
    .sort('-timestamp')
    .limit(100)
    .lean();
    
  res.json(logs);
});

Security Protection Measures

Audit logs themselves require enhanced protection:

1. Encrypt sensitive fields

const encryptField = (value) => {
  const cipher = crypto.createCipheriv('aes-256-cbc', key, iv);
  return cipher.update(value, 'utf8', 'hex') + cipher.final('hex');
};

auditSchema.pre('save', function(next) {
  if(this.after?.password) {
    this.after.password = encryptField(this.after.password);
  }
  next();
});

2. Implement strict access control

router.get('/audit/:id', authMiddleware, async (req, res) => {
  if(!req.user.roles.includes('auditor')) {
    return res.status(403).send('Forbidden');
  }
  const log = await AuditLog.findById(req.params.id);
  res.json(log);
});

Visualization

Display log analysis results using chart libraries:

router.get('/stats', async (req, res) => {
  const stats = await AuditLog.aggregate([
    {
      $group: {
        _id: {
          model: "$model",
          operation: "$operation"
        },
        count: { $sum: 1 }
      }
    },
    { $sort: { count: -1 } }
  ]);
  
  res.json(stats);
});

Log Rotation and Archiving

Storage strategy for historical logs:

const archiveOldLogs = async () => {
  const cutoff = new Date();
  cutoff.setMonth(cutoff.getMonth() - 3);
  
  const logs = await AuditLog.find({ 
    timestamp: { $lt: cutoff } 
  }).lean();
  
  if(logs.length > 0) {
    await ArchiveLog.insertMany(logs);
    await AuditLog.deleteMany({ 
      _id: { $in: logs.map(l => l._id) } 
    });
  }
};

// Execute archiving monthly
cron.schedule('0 0 1 * *', archiveOldLogs);