CAClean Architecture · Chinese
Community homeHot discussions Latest posts Best of Knowledge base Resources About

Clean Architecture · Chinese · War story: clean-architecture-zh timezone handling almost cost us a full day of billing

465
CAr/clean-architecture-zh·posted by tang_hao·yesterdayPostmortem

War story: clean-architecture-zh timezone handling almost cost us a full day of billing

Most clean-architecture-zh articles stop at "how to use it" and never cover "when not to use it". This is an attempt at the second half.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not clean-architecture-zh itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

One last trap: in container environments remember to adjust the memory-related parameters in step. Otherwise the host limit and the process expectation disagree, and the symptom is intermittent, unreproducible failure.

Nnews.ycombinator.comExternal link · opens in a new tab
161 comments

161 comments

· first 120 loaded
M
Nnikic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

475
Aalice_dev·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

460
Bbob_chen·12 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

428
Rran_bo·just nowedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

28
RraseOP·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

45
Rran_bo·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

37
Ttang_hao·just now

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

413
Kkernel_panic·just now

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

406
Aalice_dev·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

345
Bbob_chenOP·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

274
Sslow_query·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

331
Oops_wang·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

288
Rran_bo·2 hours ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

243
Bbob_chen·just now

Saved. I am reworking this area this week — this saves a lot of wrong turns.

234
Zzhou_yi·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

7
Mmike_xu·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

514
Ddev_zhou·just nowedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

212
Sslow_query·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

18
Zzhu_zong·28 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

4
Wwinter·2 days agoedited

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

144
Zzhu_zong·3 minutes ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

99
Lli_ming·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

215
Zzhu_zong·yesterday

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

188
Bbob_chenOP·28 minutes ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

325
Zzhou_yi·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

23
Zzhou_yi·3 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

134
Wwinter·2 days agoeditedLevel 6

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

479
Bbob_chenOP·2 days agoeditedLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

13
Bbob_chen·2 days agoedited

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

6
Kkernel_panic·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

16
Zzhu_zong·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

92
Bbob_chen·12 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

82
Hhuang_keMod·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

77
Lli_ming·12 minutes ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

72
Sswoole_lee·12 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

69
Aalice_dev·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

64
Zzhu_zong·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

62
Cchen_dev·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

58
Hhuang_ke·12 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

55
Lli_ming·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

106
Sswoole_lee·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

390
Wwinter·just now

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

126
Zzhu_zong·2 days agoedited

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

107
Rrase·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

467
Nnikic·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

86
Wwinter·12 minutes ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

17
Sslow_query·2 days agoedited

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

146
Nnikic·just nowedited

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

295
Ddev_zhouOP·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

38
Rran_bo·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

41
Aalice_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

469
Mmike_xuOP·5 hours agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

109
Oops_wang·2 days agoLevel 6

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

20
Ddev_zhou·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

2
Nnikic·3 minutes agoLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

65
Hhuang_ke·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

212
Rran_bo·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

518
Mmike_xuOP·28 minutes ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Zzhu_zong·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

365
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

363
Rrase·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

54
Ddev_zhouOP·1 hour ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Kkernel_panic·28 minutes agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

147
Rran_bo·2 hours ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

18
Rrase·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

104
Lli_mingOP·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

502
Zzhou_yi·just now

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

75
Kkernel_panic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

437
Hhuang_keOP·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

157
Nnikic·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

246
Aalice_dev·2 days agoLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

10
Bbob_chenOP·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

21
Mmike_xu·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

2
Zzhu_zong·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

457
Kkite·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

43
Ddev_zhou·28 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

31
Lli_ming·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

19
WwinterMod·3 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

16
Ddev_zhou·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

14
Lli_ming·2 hours ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

30
Oops_wang·yesterdayedited

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

393
Lli_ming·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

11
Ttang_hao·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

9
Ttang_hao·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

131
Sslow_query·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

390
Sswoole_lee·12 minutes ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

5
Zzhou_yi·1 hour ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

510
Oops_wang·2 hours ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

370
Kkite·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

60
Bbob_chen·3 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

5
Mmike_xu·5 hours ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

31
Ttang_haoMod·1 hour ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

169
RraseOP·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

53
Llinlin·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

140
Kkite·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

65
Kkite·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

145
Wwinter·12 minutes agoLevel 6

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

1
Aalice_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

24
Ttang_hao·just now

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

64
Zzhou_yi·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

454
Llinlin·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

447
Hhuang_ke·2 days agoLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

7
Wwinter·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

313
Nnikic·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

134
Kkernel_panic·2 days agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

131
Cchen_dev·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

103
Hhuang_ke·2 days agoLevel 6

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

58
Zzhou_yi·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

3
Ddev_zhou·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Llinlin·2 days agoeditedLevel 6

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

2
Zzhu_zong·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Ddev_zhou·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

1
Bbob_chen·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

5
Nnikic·28 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

4
Zzhu_zong·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

4
Hhuang_ke·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

3
Aalice_dev·2 days agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

3
Zzhou_yi·28 minutes agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

2
Zzhu_zong·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Rran_bo·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

1

This is the post detail page /en/c/clean-architecture-zh/post/p7. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →