Request frequency limits and anti-brushing

Request Rate Limiting and Anti-Scraping

Koa2, as a lightweight Node.js framework, requires effective request rate limiting and anti-scraping measures to ensure service stability. High-frequency requests can exhaust server resources, while malicious scraping may lead to data security issues. Properly designed rate-limiting strategies can effectively balance user experience and system load.

Basic Rate Limiting Principles

The token bucket algorithm is a common rate-limiting implementation. Each user maintains a token bucket that is replenished at a fixed rate. Requests consume tokens, and requests are denied when no tokens are available. The sliding window algorithm provides more precise tracking of requests within a time unit.

const Redis = require('ioredis')
const redis = new Redis()

async function rateLimit(key, limit, duration) {
  const current = await redis.incr(key)
  if (current === 1) {
    await redis.expire(key, duration)
  }
  return current > limit
}

This code implements a simple counter using Redis. The key is typically a user ID or IP, limit is the maximum allowed requests within the time window, and duration is the length of the time window (in seconds). The function returns true when the counter exceeds the threshold.

Koa2 Middleware Implementation

In Koa2, rate limiting can be uniformly handled via middleware. The following example uses the koa-ratelimit library to implement IP-based rate limiting:

const ratelimit = require('koa-ratelimit')
const Redis = require('ioredis')

app.use(ratelimit({
  driver: 'redis',
  db: new Redis(),
  duration: 60000,
  errorMessage: 'Request frequency too high',
  id: (ctx) => ctx.ip,
  headers: {
    remaining: 'Rate-Limit-Remaining',
    reset: 'Rate-Limit-Reset',
    total: 'Rate-Limit-Total'
  },
  max: 100,
  disableHeader: false
}))

This configuration enforces a limit of 100 requests per minute. Response headers include the remaining requests (Rate-Limit-Remaining) and reset time (Rate-Limit-Reset). When the limit is triggered, a 429 status code is returned.

Rate Limiting in Distributed Environments

Single-machine rate limiting fails in distributed systems, necessitating external storage for cluster-wide rate limiting. Redis's INCR and EXPIRE commands are a typical solution:

const redis = new Redis(6379, 'redis-cluster')

app.use(async (ctx, next) => {
  const key = `rate_limit:${ctx.ip}`
  const limit = 30
  const window = 60
  
  const current = await redis.get(key)
  if (current && parseInt(current) > limit) {
    ctx.status = 429
    ctx.body = { error: 'Request too frequent' }
    return
  }
  
  await redis.multi()
    .incr(key)
    .expire(key, window)
    .exec()
  
  await next()
})

Redis transactions ensure the counter and expiration time are set synchronously, avoiding race conditions. multi() starts the transaction, and exec() executes all commands.

Advanced Anti-Scraping Strategies

Basic rate limiting can be bypassed, requiring a combination of strategies:

1. Dynamic Thresholds: Adjust rate limits based on user behavior

function getDynamicLimit(userLevel) {
  const limits = { 'vip': 200, 'normal': 50, 'new': 20 }
  return limits[userLevel] || 30
}

2. Secondary Verification for Sensitive Operations: Require CAPTCHA for critical actions

app.use(async (ctx, next) => {
  if (ctx.path === '/transfer' && !ctx.session.captchaVerified) {
    ctx.status = 403
    ctx.body = { error: 'CAPTCHA verification required' }
    return
  }
  await next()
})

3. Behavioral Fingerprinting: Generate unique fingerprints from device information

function generateFingerprint(ctx) {
  const { headers, ip } = ctx
  return crypto
    .createHash('md5')
    .update(`${ip}-${headers['user-agent']}`)
    .digest('hex')
}

Anomaly Traffic Detection

Real-time analysis of request characteristics to identify abnormal traffic patterns:

const anomalyDetection = {
  history: new Map(),
  
  check(ctx) {
    const key = ctx.ip
    const now = Date.now()
    const record = this.history.get(key) || { count: 0, lastTime: now }
    
    // Calculate request interval anomaly score
    const interval = now - record.lastTime
    const anomalyScore = interval < 100 ? 1 : 0
    
    // Update record
    this.history.set(key, {
      count: record.count + 1,
      lastTime: now,
      score: (record.score || 0) * 0.9 + anomalyScore
    })
    
    return this.history.get(key).score > 0.8
  }
}

app.use(async (ctx, next) => {
  if (anomalyDetection.check(ctx)) {
    ctx.status = 429
    ctx.body = { error: 'Abnormal traffic detected' }
    return
  }
  await next()
})

This algorithm calculates an anomaly score based on request intervals, with sustained short-interval requests accumulating high scores.

Gradual Rollout and Circuit Breaking

New APIs should use gradual rollout strategies combined with circuit breakers to prevent cascading failures:

const circuitBreaker = {
  state: 'closed',
  failureCount: 0,
  threshold: 5,
  resetTimeout: 30000,
  
  check() {
    if (this.state === 'open') {
      const now = Date.now()
      if (now - this.lastFailure > this.resetTimeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Service circuit breaker active')
      }
    }
  },
  
  recordFailure() {
    this.failureCount++
    if (this.failureCount >= this.threshold) {
      this.state = 'open'
      this.lastFailure = Date.now()
    }
  },
  
  recordSuccess() {
    this.failureCount = 0
    if (this.state === 'half-open') {
      this.state = 'closed'
    }
  }
}

app.use(async (ctx, next) => {
  try {
    circuitBreaker.check()
    await next()
    circuitBreaker.recordSuccess()
  } catch (err) {
    circuitBreaker.recordFailure()
    throw err
  }
})

The circuit breaker has three states: closed (normal), open (tripped), and half-open (probing). Continuous failures exceeding the threshold trip the breaker, which enters a half-open state after a timeout to probe for recovery.

Logging and Monitoring Systems

Comprehensive monitoring is essential for rate-limiting systems:

const statsd = require('node-statsd')()
const logger = require('./logger')

app.use(async (ctx, next) => {
  const start = Date.now()
  
  try {
    await next()
    const duration = Date.now() - start
    
    // Record success metrics
    statsd.increment('requests.total')
    statsd.timing('response_time', duration)
    logger.info(`${ctx.method} ${ctx.url} - ${duration}ms`)
    
  } catch (err) {
    // Record error metrics
    statsd.increment('requests.errors')
    logger.error(`Request failed: ${err.stack}`)
    throw err
  }
})

Key metrics should include:

• QPS (Queries Per Second)
• Response time distribution
• Error rate
• Rate-limiting trigger count
• Abnormal request ratio

Client-Side Rate Limit Notifications

Good user experience requires clear rate-limiting feedback:

app.use(async (ctx, next) => {
  try {
    await next()
  } catch (err) {
    if (err.status === 429) {
      ctx.set('Retry-After', '60')
      ctx.body = {
        error: 'Request too frequent',
        detail: 'Rate limit: 100 requests per minute. Please try again later.',
        retryAfter: 60
      }
      return
    }
    throw err
  }
})

The Retry-After header informs clients of the retry wait time, while the response body provides detailed error information. Frontends can use this to display user-friendly messages:

axios.interceptors.response.use(null, error => {
  if (error.response.status === 429) {
    const retryAfter = error.response.headers['retry-after'] || 60
    showToast(`Operation too frequent. Please retry in ${retryAfter} seconds.`)
  }
  return Promise.reject(error)
})

Multi-Dimensional Rate Limiting

Combine different dimensions of rate limiting based on business needs:

// IP-based basic rate limiting
app.use(ratelimit({ id: ctx => ctx.ip, max: 100 }))

// Enhanced user-based rate limiting
app.use(async (ctx, next) => {
  if (ctx.state.user) {
    const key = `user_limit:${ctx.state.user.id}`
    const limit = getDynamicLimit(ctx.state.user.level)
    if (await rateLimit(key, limit, 60)) {
      ctx.status = 429
      return
    }
  }
  await next()
})

// Special limits for specific endpoints
app.use(async (ctx, next) => {
  if (ctx.path === '/api/sms') {
    const key = `sms_limit:${ctx.ip}`
    if (await rateLimit(key, 1, 60)) {
      ctx.status = 429
      return
    }
  }
  await next()
})

This layered rate-limiting system prevents global overload while applying stricter controls to sensitive endpoints. SMS verification APIs typically require separate limits to prevent SMS bombing attacks.