UbUbuntu Community
Community homeReleases Service deployment System tuning

Ubuntu Community · Interview notes: how I answered a question about the Ubuntu concurrency model

1.6K
Ubr/ubuntu·posted by huang_ke·3 hours agoOpen source

Interview notes: how I answered a question about the Ubuntu concurrency model

It took me two weeks of on-and-off digging and plenty of wrong turns. Writing the process down as it happened so the next person spends less time.

The first thing was to collapse the variables. We were changing config and upgrading the version at the same time, and afterwards nobody could say which change caused what. We rolled back to moving one variable at a time, re-ran three times, and only then did the curve settle. Tedious, but not skippable.

Worth noting: the official docs do cover this, just in a very inconspicuous spot. I only found it reading the source comments, where the author explains the reasoning — roughly "so that it degrades into predictable behaviour in extreme cases".

Bblog.rust-lang.orgExternal link · opens in a new tab
408 comments

408 comments

· first 120 loaded
M
Wwinter·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

514
Aalice_dev·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

431
Sslow_query·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

391
Ddev_zhouMod·1 hour ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

428
Aalice_dev·2 hours agoedited

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

423
Zzhu_zong·1 hour ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

5
Sswoole_lee·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

420
Nnikic·12 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

330
Bbob_chen·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

327
Rrase·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

3
Sswoole_lee·28 minutes agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

303
RraseOP·12 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

350
Sswoole_lee·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

303
LlinlinOP·1 hour ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

2
Mmike_xu·2 days agoedited

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

82
Aalice_dev·3 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

11
Bbob_chen·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

492
Kkernel_panic·2 days agoLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

44
Nnikic·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

7
Llinlin·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

35
LlinlinOP·1 hour ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

1
Llinlin·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

435
Kkernel_panic·3 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

99
Zzhou_yi·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

54
Nnikic·just now

Saved. I am reworking this area this week — this saves a lot of wrong turns.

25
Llinlin·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

388
Sswoole_lee·28 minutes ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

111
Bbob_chen·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

155
Oops_wang·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

220
Wwinter·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

55
Aalice_devOP·just nowLevel 6

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

12
Cchen_devOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

3
Lli_ming·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

1
Wwinter·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

44
Rran_bo·2 hours ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

162
Zzhou_yiOP·2 days agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

327
Aalice_dev·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

6
Nnikic·yesterday

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

70
Nnikic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

481
Oops_wang·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

472
Kkite·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

341
Zzhou_yiOP·2 days agoeditedLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

226
Mmike_xuOP·2 days agoLevel 6

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

168
Mmike_xu·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Hhuang_ke·just nowLevel 6

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

90
Sswoole_lee·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

450
Ttang_hao·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

110
Oops_wang·2 days agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

446
Hhuang_ke·yesterdayLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

435
Llinlin·2 days agoLevel 6

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

51
Rrase·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

322
Aalice_dev·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

45
Mmike_xuOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

94
WwinterOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

217
Rrase·12 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

24
Cchen_dev·2 hours ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

201
Rran_bo·3 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

140
Oops_wang·5 hours ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

306
Wwinter·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

66
Rran_bo·2 days agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

4
Rran_bo·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

200
Ddev_zhou·2 days ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

20
Sslow_query·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

139
Sswoole_leeOP·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

78
Nnikic·3 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

130
Rrase·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

130
Llinlin·yesterdayedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

418
Zzhu_zongOP·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

59
Oops_wang·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

106
Kkite·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

365
Bbob_chenOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

45
Oops_wangOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

10
Sswoole_lee·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

104
Zzhou_yi·2 days ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

97
Sswoole_lee·2 days ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

95
Zzhou_yi·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

139
Llinlin·1 hour ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

93
Rrase·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

318
Wwinter·1 hour ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
Hhuang_ke·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

216
Ttang_hao·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

258
Hhuang_ke·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

110
Ddev_zhouOP·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

6
Ddev_zhou·28 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

1
Lli_ming·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

74
NnikicOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

6
Wwinter·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

61
Kkite·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

59
Hhuang_ke·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

57
Sslow_query·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

46
Sslow_queryOP·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Mmike_xu·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

45
Rrase·just now

Saved. I am reworking this area this week — this saves a lot of wrong turns.

1
Rran_bo·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

43
LlinlinMod·2 days agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

31
Nnikic·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

24
Wwinter·1 hour ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

21
Hhuang_ke·5 hours ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

21
Ddev_zhouMod·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

20
Ddev_zhou·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

451
Rrase·just now

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

14
Kkernel_panic·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

509
Oops_wangOP·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

357
Rrase·1 hour ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

8
Nnikic·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

321
Zzhu_zong·yesterday

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

7
Oops_wang·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

169
Wwinter·2 days ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

7
Rran_bo·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

3
Zzhou_yi·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

1
Sslow_queryOP·2 days ago

This is not a Ubuntu problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

446
Cchen_dev·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

7
LlinlinMod·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Aalice_dev·2 days agoedited

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Zzhou_yi·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

296
Lli_ming·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Kkernel_panic·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Ttang_hao·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

95
Oops_wang·2 days ago

I just read the Ubuntu source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Ttang_hao·2 days ago

One counter-example: below Ubuntu 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

1

This is the post detail page /en/c/ubuntu/post/p12. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →