Multi-process/multi-instance build solution

Multi-process/Multi-instance Build Solutions

During Webpack builds, the single-threaded nature of JavaScript can make build speed a bottleneck. Multi-process/multi-instance solutions significantly improve build efficiency by parallelizing tasks. Common implementations include thread-loader, HappyPack, and Webpack 5's alternatives to thread-loader.

Basic Usage of thread-loader

thread-loader runs time-consuming loaders in a separate worker pool. After installation, simply add it at the beginning of the loader chain:

npm install thread-loader --save-dev

Configuration example:

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        use: [
          {
            loader: 'thread-loader',
            options: {
              workers: 2,  // Launch 2 worker processes
              workerParallelJobs: 50,
              poolTimeout: 2000
            }
          },
          'babel-loader'
        ]
      }
    ]
  }
}

Key parameter explanations:

• workers: Number of worker processes (recommended: CPU cores - 1)
• workerParallelJobs: Number of parallel tasks per worker
• poolTimeout: Time (in ms) to keep idle worker pools alive

HappyPack Implementation

Although no longer maintained, HappyPack can still be used in legacy projects:

const HappyPack = require('happypack');
const os = require('os');
const happyThreadPool = HappyPack.ThreadPool({ size: os.cpus().length - 1 });

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        use: 'happypack/loader?id=js'
      }
    ]
  },
  plugins: [
    new HappyPack({
      id: 'js',
      threadPool: happyThreadPool,
      loaders: ['babel-loader']
    })
  ]
}

Webpack 5 Persistent Caching Solution

Webpack 5 reduces recompilation through persistent caching:

module.exports = {
  cache: {
    type: 'filesystem',
    buildDependencies: {
      config: [__filename]
    }
  },
  experiments: {
    backCompat: false  // Disable compatibility mode for speed
  }
}

Multi-instance Parallel Builds

For multi-page applications, use parallel-webpack for parallel builds:

// webpack.config.js
module.exports = [
  {
    entry: './pageA.js',
    output: { filename: 'pageA.bundle.js' }
  },
  {
    entry: './pageB.js',
    output: { filename: 'pageB.bundle.js' }
  }
]

Run command:

npx parallel-webpack --config webpack.config.js

Inter-process Communication Optimization

Reduce communication overhead with shared memory:

const { SharedArrayBuffer } = require('worker_threads');

// Main process
const sharedBuffer = new SharedArrayBuffer(1024);
new Worker('./worker.js', { workerData: { buffer: sharedBuffer } });

// worker.js
const { parentPort, workerData } = require('worker_threads');
const sharedArray = new Int32Array(workerData.buffer);

Performance Comparison

Build time comparison across solutions (test project: 2000 modules):

Common Issue Troubleshooting

1. Memory Leaks: Monitor worker memory usage

setInterval(() => {
  console.log(`Worker memory: ${process.memoryUsage().heapUsed / 1024 / 1024}MB`);
}, 1000);

2. Process Hangs: Add timeout mechanisms

const { timeout } = require('promise-timeout');
timeout(workerPromise, 30000).catch(() => process.kill(worker.pid));

3. Load Imbalance: Dynamic task allocation

class DynamicBalancer {
  constructor(workers) {
    this.queue = [];
    this.workers = workers.map(w => ({ busy: false, worker: w }));
  }
  
  dispatch(task) {
    const freeWorker = this.workers.find(w => !w.busy);
    if (freeWorker) {
      freeWorker.busy = true;
      freeWorker.worker.send(task, () => {
        freeWorker.busy = false;
        if (this.queue.length) this.dispatch(this.queue.shift());
      });
    } else {
      this.queue.push(task);
    }
  }
}

Advanced Configuration Tips

1. Conditional Multi-process Activation:

const useThreads = process.env.NODE_ENV === 'production';
const jsLoaders = useThreads 
  ? ['thread-loader', 'babel-loader'] 
  : ['babel-loader'];

2. Custom Worker Pools:

const { WorkerPool } = require('workerpool');
const pool = WorkerPool({
  minWorkers: 'max',
  maxWorkers: require('os').cpus().length
});

// Usage in loaders
pool.exec('transform', [asset.source]).then(result => {
  callback(null, result.code);
});

3. Process Priority Control:

# Linux systems
nice -n 10 webpack --progress

Integration with Other Optimizations

1. Combining with Caching:

module.exports = {
  module: {
    rules: [
      {
        test: /\.js$/,
        use: [
          {
            loader: 'thread-loader',
            options: { workers: 4 }
          },
          {
            loader: 'babel-loader',
            options: { cacheDirectory: true }
          }
        ]
      }
    ]
  }
}

2. Using with DLL Plugin:

// webpack.dll.js
module.exports = {
  entry: { vendor: ['react', 'lodash'] },
  plugins: [
    new webpack.DllPlugin({
      name: '[name]',
      path: path.join(__dirname, '[name]-manifest.json')
    })
  ]
};

// webpack.config.js
new webpack.DllReferencePlugin({
  manifest: require('./vendor-manifest.json')
})