AGAgent · Architecture
Community homeHot discussions Latest posts Best of Knowledge base Resources About

Agent · Architecture · Open-sourced a rate-limiting middleware for agent-design with token bucket and sliding window

171
AGr/agent-design·posted by kernel_panic·32 minutes agoHelp

Open-sourced a rate-limiting middleware for agent-design with token bucket and sliding window

It took me two weeks of on-and-off digging and plenty of wrong turns. Writing the process down as it happened so the next person spends less time.

We also fixed monitoring along the way: replaced average-based alerts with percentiles and split them per endpoint. False alerts dropped by about seventy percent and the on-call rotation visibly cheered up.

Documentation first44%
Source code first28%
Just ask someone17%
Run a demo and learn by error11%

291 votes total

97 comments

97 comments

M
Sswoole_leeOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

495
Lli_mingOP·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

455
Kkernel_panic·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

180
Lli_ming·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

210
Cchen_dev·2 days ago

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

514
Ttang_hao·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

98
Ttang_hao·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

409
Aalice_dev·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

8
Zzhu_zongMod·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

491
Nnikic·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

458
Mmike_xu·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

454
Kkite·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

31
Mmike_xu·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

436
Ddev_zhou·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

395
Kkernel_panic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

165
Ttang_haoMod·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

389
Ttang_haoOP·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

305
Zzhou_yi·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

518
Mmike_xu·5 hours agoedited

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

321
Llinlin·yesterdayedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

272
Lli_ming·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

249
Llinlin·2 hours ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Zzhu_zongOP·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

78
Zzhu_zong·28 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

68
Rran_bo·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

150
Rran_bo·2 hours ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

1
Rrase·2 days agoLevel 6

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

438
Sswoole_leeMod·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

244
Ddev_zhou·2 days ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

169
Rrase·2 days ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

155
Mmike_xu·2 days ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

141
Zzhu_zong·5 hours ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

140
Ddev_zhou·2 days ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

83
Hhuang_keOP·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

74
Mmike_xuOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

61
WwinterOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

44
Ttang_haoOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

32
RraseMod·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

83
Zzhou_yi·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

39
Llinlin·5 hours ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

20
Mmike_xu·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

19
Zzhu_zong·2 hours ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

17
Rrase·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

15
Cchen_devMod·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

15
Sswoole_lee·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

12
Llinlin·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

85
Aalice_devMod·2 days agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

35
Ddev_zhouOP·28 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

7
Wwinter·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

20
Cchen_dev·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

7
Kkite·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

5
Kkernel_panic·12 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

423
Sslow_query·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

57
Bbob_chen·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

5
Ddev_zhou·28 minutes ago

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

2
Hhuang_ke·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

2
Kkernel_panic·28 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

2
Kkite·2 days ago

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

32
Lli_ming·12 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Llinlin·3 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Cchen_dev·28 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

493
Sslow_query·2 days ago

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

18
Rrase·2 hours ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

2
Zzhu_zong·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

48
Mmike_xu·yesterday

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

12
Cchen_dev·28 minutes ago

I just read the agent-design source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

2
Mmike_xu·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

330
Wwinter·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

21
Lli_ming·12 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

1
Lli_mingOP·3 minutes agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

75
Zzhou_yiOP·5 hours agoedited

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

26
Mmike_xu·12 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

460
Cchen_dev·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

185
Llinlin·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

39
Cchen_dev·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

29
Oops_wang·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

344
Kkite·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

29
Cchen_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

6
Lli_ming·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

1
Sswoole_lee·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

110
Kkernel_panic·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

424
Bbob_chenOP·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

40
Sslow_query·2 days agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

162
Zzhu_zongOP·just now

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

34
Lli_ming·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

231
Hhuang_ke·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

34
Zzhu_zong·2 days agoeditedLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

239
Sswoole_lee·12 minutes agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

77
Aalice_dev·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

85
Hhuang_ke·12 minutes agoedited

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Kkernel_panic·just now

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

162
Kkernel_panic·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

11
Kkite·2 days ago

One counter-example: below agent-design 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

320
Oops_wang·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Wwinter·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Kkernel_panic·just now

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Rran_bo·2 days ago

This is not a agent-design problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

1

This is the post detail page /en/c/agent-design/post/p5. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →