CTCLI Tools · Career
Community homeHot discussions Latest posts Best of Knowledge base Resources About

CLI Tools · Career · Why this cli-tools-career pattern gets slow in production: a source-level explanation

372
CTr/cli-tools-career·posted by kernel_panic·just nowPostmortem

Why this cli-tools-career pattern gets slow in production: a source-level explanation

It took me two weeks of on-and-off digging and plenty of wrong turns. Writing the process down as it happened so the next person spends less time.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not cli-tools-career itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

The first thing was to collapse the variables. We were changing config and upgrading the version at the same time, and afterwards nobody could say which change caused what. We rolled back to moving one variable at a time, re-ran three times, and only then did the curve settle. Tedious, but not skippable.

On trade-offs, my view is this: if nobody on the team owns this area long-term, do not introduce a second mechanism. With two coexistence you first have to work out which one is even in play when things break, and that costs far more than the performance you saved.

169 comments

169 comments

· first 120 loaded
M
Hhuang_ke·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

498
Aalice_dev·2 days ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

494
Rran_bo·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

494
Wwinter·2 days ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

493
Bbob_chen·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

473
Bbob_chen·just now

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

433
Kkite·1 hour ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

243
Kkernel_panic·28 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

138
WwinterOPMod·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

15
Zzhou_yi·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

235
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

34
Zzhu_zong·yesterdayedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

10
Hhuang_keOP·5 hours agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

340
Rran_bo·5 hours agoeditedLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

158
Mmike_xu·5 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Zzhu_zong·2 days agoeditedLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

179
Cchen_dev·2 days ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

190
WwinterOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

259
Wwinter·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

13
Lli_ming·12 minutes agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

458
Aalice_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

90
Lli_ming·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

447
Rrase·3 minutes agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

439
Rran_bo·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

378
Lli_mingOP·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

221
Llinlin·just now

Saved. I am reworking this area this week — this saves a lot of wrong turns.

270
Wwinter·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

462
LlinlinMod·28 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

1
Oops_wang·2 days ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

15
Cchen_dev·28 minutes ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

251
Wwinter·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

33
Ddev_zhou·2 days agoedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

251
Zzhou_yi·2 days agoedited

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

391
Zzhou_yi·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

28
Mmike_xu·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

56
WwinterOP·2 days ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

498
Ddev_zhou·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

221
Cchen_dev·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

162
Nnikic·3 minutes ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

210
Ttang_haoOP·28 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

395
Zzhou_yi·2 hours ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

102
Kkite·just now

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

83
Rrase·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

233
Wwinter·2 days ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

120
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

68
Zzhu_zong·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

201
Kkite·2 days agoedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

194
Kkite·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

187
Zzhu_zong·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

177
Sslow_query·28 minutes agoedited

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

141
Wwinter·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

137
Sslow_query·5 hours ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

2
Rran_bo·2 days ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

141
Ttang_hao·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

108
Kkite·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

97
Oops_wang·5 hours ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

87
Mmike_xu·3 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

178
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

293
Aalice_dev·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

304
Kkite·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

121
Rrase·2 hours ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

135
Rrase·just now

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

427
Hhuang_ke·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Rrase·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

15
Rrase·2 hours ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

416
Wwinter·2 days agoLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

299
Oops_wang·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Kkernel_panic·1 hour ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

79
RraseMod·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

77
Llinlin·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

76
Lli_mingOP·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Ttang_hao·3 minutes agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

53
Sswoole_lee·5 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

44
Oops_wangOP·2 days ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

5
Ttang_haoOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Ddev_zhou·3 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

215
Llinlin·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

11
Rrase·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

317
Sslow_query·2 days ago

One counter-example: below cli-tools-career 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

98
Sslow_query·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

42
Kkite·3 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

8
Mmike_xu·28 minutes ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

230
Cchen_dev·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

349
LlinlinOP·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

93
Sslow_query·2 days agoLevel 6

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

488
Llinlin·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

41
Nnikic·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

19
Sslow_query·3 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

5
Kkernel_panic·3 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

24
Llinlin·2 days agoLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

1
Kkite·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

6
Ddev_zhou·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

364
Kkite·just now

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

48
Aalice_dev·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

38
Aalice_dev·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

286
Wwinter·28 minutes ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

24
KkiteOP·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

486
Lli_ming·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

34
Bbob_chen·28 minutes agoedited

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

423
Llinlin·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

3
Mmike_xu·2 days ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

31
Rran_boMod·1 hour ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

19
Nnikic·5 hours ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

19
Cchen_dev·3 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

17
Llinlin·3 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

17
Kkernel_panic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

13
Nnikic·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

27
Lli_ming·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

6
Ddev_zhou·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

107
WwinterMod·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

51
Llinlin·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

65
Oops_wang·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

6
Zzhou_yi·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

5
Sswoole_lee·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

1
Mmike_xu·just now

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

278
Kkernel_panic·5 hours ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

4
Kkite·1 hour ago

This is not a cli-tools-career problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

3
Sslow_query·2 days ago

I just read the cli-tools-career source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

2
Zzhou_yi·just now

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

1
Ttang_hao·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

14

This is the post detail page /en/c/cli-tools-career/post/p9. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →