Rate Limiting Fundamentals: The Token Bucket Explained With a Front Desk
Rate and burst are two different limits, and the token bucket is the cheapest way to set both without lying to your users.
Picture a front desk with a stack of appointment cards. One new card gets set down every few minutes, and the stack tops out at some height because the desk is only so deep. If nobody's come in for an hour, there's a pile waiting and the next four people walk straight through. If the stack's empty, you wait for the next card.
That's a token bucket. That's the whole thing. And the reason it matters is that most people building an API only think about the average, so they write "100 requests per minute" in the docs and then get surprised when a legitimate customer sends 40 in three seconds and gets a wall of 429s.
In this post:
- Why a single "requests per minute" number breaks in practice
- How the token bucket separates refill rate from burst size
- How I use it in my own back office, including on things that aren't HTTP at all
- When a token bucket is the wrong tool
1. Why it breaks
Real traffic is lumpy. Nobody sends you a smooth, evenly spaced stream of requests, because the world doesn't work that way: a page loads and fires six calls at once, a sync job wakes up and pushes everything it buffered overnight, a customer taps the submit button three times because the spinner looked frozen. The average over a minute can be perfectly reasonable while the shape of it is a cliff.
So you reach for the obvious thing, a counter that resets. Fixed window. Count requests in the current minute, reject past 100, zero it out when the clock ticks. It's easy and it's wrong in a specific way: a client can send 100 at 11:00:59 and 100 more at 11:01:00 and you just took 200 requests in two seconds while your dashboard swears nobody exceeded the limit. The window boundary is a free lunch and clients find it, usually by accident.
The other failure is the opposite. You get strict, you enforce a hard minimum gap between requests, and now your well-behaved client that wants to send five calls to render one screen gets throttled for being normal. You've punished burstiness that costs you nothing, because the bucket had room and you refused to use it.
Here's my opinion, and it's mild criticism of a very common practice: publishing one number as your rate limit is a documentation bug. "1000 requests per hour" tells me nothing about whether I can send ten in a row. Two numbers, rate and burst, and suddenly every integrator knows exactly how to build against you.
2. The mechanism
A token bucket has two parameters and one piece of state. The parameters: refill rate, how fast tokens drip in, and capacity, how many tokens the bucket can hold. The state: how many tokens are in there right now. Every request costs a token. If there's a token, you spend it and the request goes through. If there isn't, you reject or you queue.
Capacity is your burst. Refill rate is your sustained throughput. They are independent dials and that's the entire point. The token bucket algorithm as described on Wikipedia goes further into the networking lineage, where it was used to shape packet flows, but the shape of it is identical whether you're policing bytes on a wire or calls to a quoting endpoint.
TOKEN BUCKET
refill: 1 token every 6s ───────┐
(sustained rate) │
▼
┌────────────────────┐
│ ● ● ● ● ● │ capacity = 10
│ │ (BURST)
└─────────┬──────────┘
│ overflow tokens
│ are discarded ──► (wasted)
│
request arrives ─────────┤
│
┌─────────────┴──────────────┐
│ │
tokens > 0? tokens == 0?
│ │
spend 1 token reject (429) or queue
ALLOW with Retry-After
│
▼
downstream work
IDLE PERIOD BURST
─────────── ─────
t=0 ●●●●●●●●●● 10 saved up
t=1 ●●●●● 5 requests fire instantly, allowed
t=2 ● 4 more fire, allowed, bucket near empty
t=3 (empty) next request waits ~6s for refill
Look at the bottom block: the saved tokens are what pay for the burst, and once they're spent the client drops to the refill rate. Nothing else changed.
A few properties fall out of this for free. Idle clients accumulate credit, up to capacity and no further, so you can't bank a month of quiet and then detonate. Steady-state throughput is exactly the refill rate, no window boundary to exploit, because there's no window. And the state is tiny: a token count and a last-updated timestamp, which means you don't actually store a timer, you just compute how many tokens should have arrived since you last looked and clamp to capacity. Lazy refill. That's the implementation trick that makes this cheap enough to run per customer, per endpoint, per whatever.
Worth knowing: the token bucket's cousin, the leaky bucket, smooths output to a constant rate and tolerates no burst at all. Both are useful. They answer different questions. If you want a hard ceiling on how fast work leaves your system, leaky. If you want to permit honest bursts while capping the average, token.
3. How I run it
My own company is a California field-services outfit and I built the back office that runs it, so I'm not theorizing here, I've had to pick numbers and live with them.
The obvious place is outbound. Anything my system calls that I don't own gets a bucket in front of it, because the vendor has their own limits and I'd rather shape my traffic than discover theirs during a busy morning. The refill rate sits comfortably under whatever they publish, the capacity is sized to the biggest legitimate clump I actually produce, which for me is usually a batch of follow-ups going out together after a job closes. When the bucket empties, the caller doesn't fail, it waits. Outbound rate limiting with a queue behind it is a politeness feature, not an error path.
The less obvious place, and the one I'd push harder on, is that half of my buckets have nothing to do with HTTP. The phone assistant has a bucket on how many times it will attempt to reach the same person in a day, because an assistant that doesn't get tired is an assistant that will absolutely harass a customer on your behalf. Follow-up messaging has a bucket per contact, with a small capacity so a legitimate back-and-forth flows naturally and a slow refill so nobody gets drip-fed into the spam folder. Field update notifications have one. Any loop that can run without a human watching it gets a token bucket, and I treat that as a rule rather than a judgment call, because the failure mode of an unbounded automated loop is embarrassment in front of a customer, and that costs more than a rejected request ever will.
Per-key, not global. One noisy integration shouldn't starve everybody, so the bucket lives keyed to the identity doing the work: the customer, the contact, the vendor connection. Global buckets are a last-resort fuse, not your primary control.
And when I reject, I say why and I say when. A limit response with no indication of how long to wait is just a mystery, and mysteries turn into retry storms, which is how a rate limiter becomes the cause of the outage it was supposed to prevent. Tell the client the ceiling, tell it the reset, let it behave.
4. Tradeoffs
The token bucket's generosity is also its risk: capacity is, by definition, permission to exceed your sustained rate for a moment, so if your downstream genuinely cannot survive a spike, say a database that falls over at a certain concurrency, then a burst allowance is a loaded gun and you want a leaky bucket or a straight concurrency limiter instead. Distributed enforcement is the other honest cost. The moment you run more than one instance you need the token count somewhere shared, and then you're paying a round trip per request or accepting sloppy approximation by splitting the bucket across nodes, neither of which is free. And none of this protects you from a single expensive request: one token for a trivial read and one token for a report that scans everything is a fiction, so either weight the cost per operation or accept that your limiter is measuring the wrong thing. If your traffic is already smooth and your real problem is a handful of brutal queries, rate limiting isn't your fix.
Takeaway
- Set two dials, refill rate for the sustained ceiling and capacity for the burst, and publish both.
- Store a token count and a timestamp, compute the refill lazily, and key the bucket to the identity doing the work.
- Put a bucket on every automated loop that can reach a human, not just on your HTTP endpoints.
One line: a rate limit without a burst number is only half a limit.
Jesse