Clean Architecture · Chinese · clean-architecture-zh logging: what separates "readable" from "usable"

401
CAr/clean-architecture-zh·posted by kernel_panic·yesterdayTranslation

clean-architecture-zh logging: what separates "readable" from "usable"

Most clean-architecture-zh articles stop at "how to use it" and never cover "when not to use it". This is an attempt at the second half.

The first thing was to collapse the variables. We were changing config and upgrading the version at the same time, and afterwards nobody could say which change caused what. We rolled back to moving one variable at a time, re-ran three times, and only then did the curve settle. Tedious, but not skippable.

Self-host — full control44%
Managed service — less work28%
Hybrid: core self-hosted17%
Not decided yet11%

682 votes total

225 comments

225 comments

· first 120 loaded
M
Hhuang_ke·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

498
Sswoole_lee·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

479
Sswoole_lee·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

471
Lli_ming·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

413
Kkite·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

460
Cchen_devOP·1 hour ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

381
Rrase·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

434
Wwinter·1 hour ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

429
Kkite·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

405
Llinlin·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

398
Zzhou_yi·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

267
Oops_wang·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

42
Zzhu_zong·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

497
Nnikic·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

371
Bbob_chen·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

134
Kkernel_panic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

1
Ttang_hao·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

397
Bbob_chenOP·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

311
Bbob_chen·3 minutes ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

364
Ddev_zhou·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

323
Llinlin·1 hour ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

341
Kkite·12 minutes ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

168
Rran_bo·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

377
Oops_wang·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

178
Lli_mingOP·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

503
Kkite·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

6
Llinlin·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

489
Ttang_hao·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

319
Oops_wang·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

83
Bbob_chen·2 hours ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

51
Zzhou_yi·2 hours ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

3
Ddev_zhou·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

454
Lli_ming·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

270
Sswoole_lee·2 hours ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

2
Kkite·yesterday

Saved. I am reworking this area this week — this saves a lot of wrong turns.

490
Kkite·yesterday

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

80
Sslow_query·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

286
Wwinter·2 days agoeditedLevel 6

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

301
Wwinter·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

158
Kkernel_panicOP·2 days agoLevel 6

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

375
Wwinter·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

44
Ttang_hao·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

2
Sslow_query·5 hours agoLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

114
Mmike_xu·2 days agoedited

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

36
KkiteMod·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

16
Bbob_chen·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

7
Oops_wang·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

27
Hhuang_ke·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

300
Oops_wang·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

12
Wwinter·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Lli_ming·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

241
Zzhu_zong·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

218
Rrase·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

104
Zzhu_zong·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

196
Aalice_dev·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

192
Sswoole_lee·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

187
Nnikic·1 hour ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

470
Nnikic·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

158
WwinterMod·28 minutes agoedited

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

145
LlinlinMod·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

354
Aalice_dev·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

158
Bbob_chen·28 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

39
RraseOP·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

39
Mmike_xu·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

213
Ttang_hao·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

37
Kkernel_panic·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

219
Zzhu_zong·2 days agoLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

176
Zzhou_yi·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

2
Kkernel_panic·2 days agoLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

137
Zzhu_zong·2 days agoLevel 6

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

4
Bbob_chen·2 days agoLevel 6

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Rran_boOP·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

37
Ddev_zhou·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

5
Sslow_queryOP·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

137
Kkite·12 minutes agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

495
Hhuang_ke·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

403
Nnikic·2 days agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

63
Zzhu_zong·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

1
Ddev_zhou·1 hour ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

26
Wwinter·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

316
Cchen_dev·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

79
Ttang_hao·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

198
Aalice_dev·3 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

136
LlinlinOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

66
Kkernel_panicOP·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

2
Ddev_zhouMod·2 days ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

134
Ttang_hao·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

99
KkiteOP·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

13
Oops_wangMod·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

86
Aalice_devOP·just now

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

15
Zzhu_zongOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Kkite·12 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

123
Kkite·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

20
Aalice_devOP·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Cchen_dev·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

53
Kkite·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

52
Rran_bo·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

48
Sslow_query·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

19
Aalice_dev·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

334
Ttang_hao·2 days ago

I just read the clean-architecture-zh source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

17
Mmike_xu·3 minutes ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

16
Rran_bo·3 minutes ago

This is not a clean-architecture-zh problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

9
Nnikic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

259
Rrase·2 days ago

One counter-example: below clean-architecture-zh 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

6
Oops_wang·2 hours ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

4
Zzhou_yiOP·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

480
Rrase·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

28
Sslow_query·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

208
Rrase·2 days agoeditedLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

27
Mmike_xu·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

330
Nnikic·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

23
Rran_bo·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

213
Rrase·just now

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

6
Rrase·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

6
Hhuang_ke·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

4
Wwinter·2 days agoedited

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

1
Lli_ming·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Ttang_hao·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

1
Nnikic·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

284
Bbob_chen·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

1

This is the post detail page /en/c/clean-architecture-zh/post/p1. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →