Mastering Rate Limiting in Distributed Systems

The Importance of Rate Limiting

In the world of microservices and public APIs, rate limiting is not just a feature—it's a necessity. It protects your infrastructure from abuse, ensures fair usage among tenants, and prevents cascading failures when dependent services get overwhelmed.

Imagine a scenario where a single user's script goes rogue and starts hitting your API 1000 times per second. Without rate limiting, this could saturate your database connections, crash your workers, and degrade performance for everyone else.

Common Algorithms

There are several standard algorithms for rate limiting, each with its own pros and cons:

• Token Bucket: Ideally suited for allowing short bursts of traffic while maintaining a steady average rate. Tokens are added to a bucket at a fixed rate, and requests consume tokens.
• Leaky Bucket: Processes requests at a constant rate, smoothing out bursts. Great for protecting databases that prefer a steady load.
• Fixed Window Counter: Simple to implement (e.g., reset counter every minute), but can suffer from "thundering herd" issues at the window boundaries.
• Sliding Window Log: The most accurate but resource-intensive, as it stores timestamps for every request.

Distributed Limiting with Redis

When running multiple instances of your service, local memory rate limiting isn't enough. You need a centralized store. Redis is perfect for this due to its atomic operations and speed.

Here's a simple example of a "Fixed Window" limiter using Redis `INCR` and `EXPIRE` in Go pseudo-code:

func RateLimit(userID string, limit int) bool {
    key := "rate_limit:" + userID + ":" + time.Now().Unix() / 60
    
    // Increment the counter
    count, _ := redisClient.Incr(key).Result()
    
    // Set expiry for 60 seconds if it's a new key
    if count == 1 {
        redisClient.Expire(key, 60*time.Second)
    }
    
    return count <= limit
}

For more advanced "Sliding Window" implementation, you might use Redis Sorted Sets (ZSET) to store timestamps and count elements within the last window.

Handling 429s Gracefully

If you are consuming APIs (like SnapAPI), you must handle HTTP 429 Too Many Requests errors correctly. Here is the golden rule: Respect the Retry-After header.

A robust client implementation should:

1. Check for status code 429.
2. Read the `Retry-After` header (usually in seconds).
3. Sleep for that duration (plus a small jitter).
4. Retry the request.

// Exponential Backoff Example
func makeRequest(url string) {
    baseDelay := 1 * time.Second
    maxRetries := 5
    
    for i := 0; i < maxRetries; i++ {
        resp, err := client.Get(url)
        if resp.StatusCode == 429 {
            // Wait and retry
            time.Sleep(baseDelay * Math.Pow(2, i))
            continue
        }
        return resp
    }
}

Building for Resilience

Rate limiting is a two-way street. As an API provider, you implement it to protect your system. As an API consumer, you handle it to ensure your application remains stable even when limits are hit.