Rate Limiter System Design: Algorithms, Architecture, Redis, and Scaling
A rate limiter controls how many API requests a client can make within a specific time window. It protects API servers from excessive traffic, prevents abuse, and helps ensure fair resource usage across users.
In a rate limiter system design, the main challenge is not simply counting requests. You need to choose the right rate limiting algorithm, decide where the limiter should run, determine how its state is stored, and make sure the same limit works correctly across multiple servers.
This guide explains the core algorithms, Redis-based architecture, distributed rate limiting, and the key design decisions you need for a production system.
What Is a Rate Limiter?
A rate limiter limits the number of requests a client can make during a defined period.
When a client sends a request, the limiter identifies the client using an API key, user ID, or IP address and checks the applicable rate limiting rules.
If the client is below the limit, the request is allowed. If the client exceeds the rate limit, the limiter rejects the request, usually with HTTP 429 Too Many Requests. The server can also return Retry-After to tell the client when it should try again. MDN: 429 Too Many Requests
Rate limiting is different from concurrency limiting. Rate limiting controls how frequently requests arrive, while concurrency limits control how many operations can execute at the same time. Microsoft: Throttling pattern
Why Rate Limiting Matters in System Design
Without rate limiting, a single client can generate excessive traffic and consume resources needed by legitimate users.
Consider an API that normally handles 5,000 requests per second. A sudden burst of traffic from a bot, retry loop, or faulty client can overwhelm the application, database, or third-party services.
A rate limiter provides an early control point that can:
- protect API servers from overload
- enforce quotas for users or API keys
- prevent abusive clients from consuming shared resources
- control expensive API operations
- maintain fair resource usage
The limiter should ideally reject unnecessary traffic before it reaches expensive backend components.
Rate Limiting Requirements and Design Goals
Before choosing an algorithm, define the limiting rules.
A typical API might have:
- 100 requests per minute per user
- 10 requests per second for an expensive endpoint
- a global limit for the entire service
- different limits for free and premium users
The limiter also needs to answer an important question: what identifies a client?
Depending on the system, the limit can be based on:
- user ID
- API key
- IP address
- tenant
- endpoint
- a combination of these values
For a production rate limiter system design, the main non-functional requirements are low latency, high availability, predictable behavior, and consistent enforcement across multiple servers.
Where Should a Rate Limiter Be Placed?
A rate limiter can run at different layers.
At the edge, unwanted traffic can be rejected before it reaches your infrastructure. At the API gateway, you can enforce API keys, users, tenants, and endpoint-specific quotas. Application-level rate limiter middleware can enforce rules that require business logic.
A common architecture is:
Client → API Gateway → Rate Limiter → API Servers → Database
The API gateway is often a good enforcement point because rejected requests can be stopped before consuming application resources.
If you want to understand the difference between an API gateway and a load balancer, see S4Stack’s API Gateway vs Load Balancer.
For larger systems, rate limiting can exist at multiple layers. An edge limiter can stop obvious abusive traffic while an application-level limiter enforces user-specific quotas.
Rate Limiting Algorithms
The most common rate limiting algorithms are:
| Algorithm | How it works | Main advantage | Main drawback |
|---|---|---|---|
| Token bucket | Requests consume tokens that refill at a fixed rate | Supports controlled bursts | Requires bucket state |
| Leaky bucket | Requests are processed at a fixed rate | Smooth traffic | Can introduce delay |
| Fixed window | Counts requests inside fixed intervals | Simple and efficient | Boundary bursts |
| Sliding window log | Stores request timestamps | Very accurate | Higher memory usage |
| Sliding window counter | Combines counts from adjacent windows | Good accuracy and memory efficiency | Approximate |
Token Bucket Algorithm
The token bucket algorithm maintains a bucket containing tokens.
Tokens are added at a fixed refill rate until the bucket size is reached. Each request consumes one token.
If the bucket contains enough tokens, the request is allowed. If there are no tokens available, the request is rejected or delayed.
The major advantage is that it supports short bursts of traffic while still enforcing an average rate.
For example, a bucket may have:
- bucket size: 100 tokens
- refill rate: 10 tokens per second
A client can make a short burst of up to 100 requests if enough tokens have accumulated, but the long-term rate remains controlled.
Amazon API Gateway uses token-bucket-based throttling. AWS API Gateway throttling
Leaky Bucket Algorithm
The leaky bucket algorithm processes requests at a relatively fixed outflow rate.
Incoming requests enter a queue, and the system processes them at the configured rate. If the queue becomes full, additional requests are dropped.
This is useful when the downstream service needs a smoother request flow rather than allowing large bursts.
The trade-off is that requests may be delayed, so it is less suitable when immediate responses are important.
Fixed Window Counter
The fixed window counter algorithm divides time into fixed intervals.
For example:
100 requests per minute.
The limiter increments a counter value for each request. When the minute ends, the counter resets.
It is simple, fast, and memory efficient, but it has a boundary problem.
A client could send 100 requests at 12:00:59 and another 100 at 12:01:01. The limiter sees 100 requests in each window even though 200 requests arrived within two seconds.
Sliding Window Log
The sliding window log algorithm stores the timestamp of every request.
When a request arrives, the limiter removes outdated timestamps, counts the remaining requests, and accepts the new request only when the count is below the allowed limit.
This provides accurate rolling-window behavior, but storing every timestamp increases memory usage.
Redis Sorted Sets are commonly used for this approach because entries can be ordered by timestamp. Redis Sorted Sets
Sliding Window Counter
The sliding window counter algorithm combines the current and previous window counts to estimate the number of requests inside the current rolling window.
It uses much less memory than a sliding-window log while avoiding much of the boundary problem associated with a fixed window counter.
For many APIs, it provides a practical balance between accuracy, performance, and memory usage.
How to Design a Rate Limiter
A basic rate limiter system design can follow this flow:
Client Request → Identify Client → Check Limit → Update State → Allow or Reject
Suppose the API allows 100 requests per minute per user.
When a request arrives:
- Identify the user or API key.
- Load the applicable rate limiting rules.
- Check the current counter or bucket.
- Decide whether the request is allowed.
- Update the state.
- Forward the request or return HTTP 429.
The limiter should perform this decision quickly because it sits directly in the request path.
For rejected requests, the API should return an appropriate HTTP status code and, where useful, Retry-After.
Redis-Based Rate Limiter Architecture
A local in-memory limiter works well on a single server, but it creates a problem when the application runs on multiple servers.
Imagine five API servers, each maintaining its own counter. If a user has a limit of 100 requests per minute, the user could potentially send 100 requests to each server and receive a much higher effective quota.
A shared Redis store solves this problem.
The architecture becomes:
API Gateway → Rate Limiter Middleware → Redis → API Servers
Every limiter instance checks the same shared state.
Redis provides structures useful for different algorithms. INCR and EXPIRE can implement fixed-window counters, while Sorted Sets can support sliding-window designs. Redis also supports Lua scripting to make multi-step operations atomic. Redis rate limiter documentation
Atomicity is important because a naïve read-check-write operation can create a race condition.
Suppose the current counter is 99 and the limit is 100. Two requests arrive simultaneously:
- Server A reads 99.
- Server B reads 99.
- Both decide the request is allowed.
- Both increment the counter.
The system has now allowed more requests than intended.
A Redis Lua script can combine the check and update into one atomic operation, preventing this type of synchronization issue. Redis rate limiter documentation
How Distributed Rate Limiting Works
A distributed rate limiter is required when several application or gateway instances must enforce the same quota.
The core problem is shared state.
If each server maintains its own counter, the actual limit depends on which server receives the request. A shared Redis-based limiter ensures that all instances evaluate requests against the same logical quota.
At larger scale, Redis itself can become a bottleneck. The state can therefore be distributed across Redis shards based on a client identifier such as user ID, API key, or tenant.
This creates another trade-off: stronger global consistency may require additional coordination and latency, while local enforcement is faster but may allow small differences between regions.
For systems already using data partitioning, the concepts discussed in S4Stack’s database sharding guide are also useful background for understanding distributed state.
How to Scale a Distributed Rate Limiter
The main scaling problem is usually the shared state store.
If Redis becomes overloaded, you can distribute rate-limit keys across multiple shards. You can also place lightweight local checks in front of Redis for extremely hot clients, provided the design can tolerate the resulting accuracy trade-offs.
Hot keys are another problem. A single API key, user, or tenant generating a huge number of requests can overload the Redis shard responsible for that key.
The identifier used for partitioning therefore matters. IP-based rate limiting can also create problems because thousands of legitimate users may share the same public IP through NAT or corporate networks.
Multi-region systems introduce another trade-off. A globally synchronized limit provides stronger enforcement but requires cross-region coordination. Regional limits provide lower latency and better independence but may temporarily exceed a strict global quota.
The goal is not perfect theoretical accuracy at any cost. The goal is a rate limiter that protects the system while adding very little latency.
What Happens When the Rate Limit Is Exceeded?
The most common response is:
HTTP/1.1 429 Too Many Requests
Retry-After: 30
The response tells the client that its rate limit has been reached.
Clients should use backoff instead of immediately retrying. Otherwise, repeated retries can create a retry storm and increase the load on an already stressed system. Microsoft: Retry Storm antipattern
For some workloads, you can use a queue instead of immediately rejecting requests. This is useful when the operation does not need to complete synchronously.
Common Rate Limiter Design Challenges
The most important production challenges are usually not the basic algorithms.
Race Conditions
Concurrent requests can update the same counter at the same time. Shared state therefore needs atomic operations.
Redis Failure
You must decide whether the limiter should fail open or fail closed.
Fail-open allows requests when Redis is unavailable but risks backend overload. Fail-closed protects the backend but can block legitimate users.
The correct choice depends on the system and business requirements.
Hot Keys
A single client generating excessive traffic can overload one Redis shard. Edge protection, hierarchical limits, or local short-lived caching can help.
Multiple Limits
A request may be subject to several rules:
- per-user limit
- per-IP limit
- per-endpoint limit
- per-tenant limit
- global API limit
The limiter needs a clear rule-precedence strategy.
Clock and Window Problems
Sliding-window implementations depend on timestamps. In a distributed environment, inconsistent clocks can produce unexpected results, so time handling should be designed carefully.
Rate Limiter Monitoring and Observability
A production limiter should monitor:
- allowed requests
- rejected requests
- limiter latency
- Redis errors
- hot keys
- requests by endpoint
- requests by client
- HTTP 429 rate
You should also compare limiter metrics with API latency, database load, CPU, memory, and downstream failures.
If the limiter rejects thousands of requests but the backend is almost idle, the limiting rules may be too aggressive. If backend latency is already increasing before rejection starts, the limits may be too loose.
Rate Limiter System Design Example
Consider a SaaS API with 1 million users and a default limit of 100 requests per minute per user.
A practical architecture is:
Client → API Gateway → Rate Limiter → Redis Cluster → API Servers → Database
The gateway identifies the user and endpoint. The limiter retrieves the applicable rule and performs an atomic Redis operation.
For a token bucket, Redis stores the token count and refill information. For a sliding-window implementation, Redis Sorted Sets can store request timestamps.
If the request is allowed, it reaches the application servers. If the quota is exceeded, the limiter immediately returns HTTP 429.
As traffic grows, gateway instances can scale horizontally and Redis keys can be distributed across shards.
[IMAGE PLACEHOLDER 4 — End-to-End Rate Limiter System Design]
Show Client → API Gateway → Distributed Rate Limiter → Redis Cluster → API Servers → Database, including the HTTP 429 rejection path and monitoring.
Conclusion
A good rate limiter system design is not simply a counter in Redis.
You need to choose the right rate limiting algorithm, identify the client correctly, place the limiter close to the traffic entry point, coordinate state across multiple servers, and make the decision atomic.
For burst-heavy APIs, the token bucket algorithm is often a strong choice. For smoother rolling limits, a sliding window approach can work better. Redis provides fast shared state, while Lua scripting can prevent race conditions during concurrent updates.
The best design depends on the traffic pattern, accuracy requirements, latency budget, and failure strategy.
Frequently Asked Questions
How does a rate limiter work?
A rate limiter controls how many requests a user or service can make within a specific time window to prevent abuse and overload.
How to build a rate limiter?
Choose an algorithm like Token Bucket or Sliding Window, then implement it using an in-memory store or Redis for distributed systems.
What is the purpose of a rate limiter?
It protects APIs and systems from excessive traffic, abuse, and resource exhaustion while ensuring fair access.
How do you design a rate limiter in LLD?
Define the rate-limiting strategy, request flow, storage, and interfaces, then use patterns such as Strategy to support multiple algorithms.
Is Kafka a rate limiter?
No. Kafka is a distributed event-streaming platform; it can help manage traffic but is not itself a rate-limiting mechanism.
