ECEdge Computing · Resources
Community homeHot discussions Latest posts Best of Knowledge base Resources About

Edge Computing · Resources · Two weeks with edge-computing-resources: what feels right and what drives me up the wall

1.7K
ECr/edge-computing-resources·posted by kite·2 hours agoAnnouncement

Two weeks with edge-computing-resources: what feels right and what drives me up the wall

Some background first. Our setup is edge-computing-resources plus three downstream services, seven figures of daily requests, peaking around nine in the evening.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not edge-computing-resources itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

On trade-offs, my view is this: if nobody on the team owns this area long-term, do not introduce a second mechanism. With two coexistence you first have to work out which one is even in play when things break, and that costs far more than the performance you saved.

Worth noting: the official docs do cover this, just in a very inconspicuous spot. I only found it reading the source comments, where the author explains the reasoning — roughly "so that it degrades into predictable behaviour in extreme cases".

We also fixed monitoring along the way: replaced average-based alerts with percentiles and split them per endpoint. False alerts dropped by about seventy percent and the on-call rotation visibly cheered up.

625 comments

625 comments

· first 120 loaded
M
Ddev_zhou·2 days ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

463
Mmike_xuOP·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

379
Sslow_query·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

436
Kkite·12 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

150
WwinterOP·2 days ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

347
Bbob_chen·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

391
Cchen_dev·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

379
Wwinter·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

362
Oops_wangOPMod·3 minutes ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

297
Sswoole_lee·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

345
Ddev_zhou·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

334
Aalice_devOP·2 hours ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

250
Mmike_xu·just now

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

304
Lli_ming·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

16
Hhuang_ke·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

2
Wwinter·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

251
Sslow_query·2 days ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

238
Sslow_query·2 days ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

220
Wwinter·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

219
Sslow_query·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

209
Rrase·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

196
Sswoole_leeOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

130
Rrase·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

455
Sslow_query·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

178
Zzhu_zong·12 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

176
Sslow_queryOP·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

170
Sswoole_lee·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

158
Ddev_zhou·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

119
Aalice_dev·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

151
Oops_wang·12 minutes ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Cchen_dev·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

39
Cchen_dev·2 hours ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

151
Nnikic·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

123
Kkernel_panic·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

215
Aalice_dev·2 hours ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

149
Sswoole_lee·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

133
Sslow_query·2 hours ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

9
Sswoole_lee·28 minutes agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

18
Aalice_dev·2 days ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

469
Mmike_xu·2 days ago

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

473
Ttang_hao·12 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

64
Oops_wangMod·2 days ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

46
Hhuang_keMod·2 days agoLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

220
Oops_wang·2 days agoLevel 6

I just read the edge-computing-resources source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

24
Llinlin·just nowLevel 6

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

8
Rran_boMod·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

33
Bbob_chen·2 days agoLevel 6

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

33
Aalice_dev·3 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

11
Sslow_query·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

110
Mmike_xu·2 days ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

1
Sslow_query·just now

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

18
Kkite·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

9
Kkernel_panic·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

265
Ddev_zhouMod·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

333
Hhuang_ke·2 days agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Cchen_dev·yesterday

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

126
Lli_ming·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

123
Sslow_queryMod·28 minutes agoedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

119
Llinlin·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

117
Zzhu_zong·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

109
Rrase·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

105
Lli_ming·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

81
Kkite·just now

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

72
Sswoole_lee·just nowedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

234
Sslow_query·28 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

178
Hhuang_ke·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

220
Llinlin·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

434
Sswoole_lee·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

332
NnikicOP·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

42
Aalice_dev·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

189
Nnikic·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

492
Zzhou_yiOP·1 hour agoedited

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

321
Hhuang_keOPMod·2 days ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

99
Lli_ming·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

348
Sslow_query·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

8
Nnikic·3 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

130
Oops_wang·28 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

5
Rran_boOP·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

140
Kkite·12 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

8
Hhuang_ke·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

68
Mmike_xuOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

3
Rrase·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

53
Sslow_query·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

46
Rran_bo·2 days ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

43
Sswoole_lee·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

32
Bbob_chenOP·2 days ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

312
Bbob_chen·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

27
Mmike_xuOP·1 hour ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

47
Llinlin·2 days ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

44
Bbob_chen·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

19
Zzhou_yi·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

134
Zzhou_yi·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

15
Ddev_zhou·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

13
Ddev_zhou·2 days ago

This is not a edge-computing-resources problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

257
Zzhu_zong·12 minutes ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

12
Llinlin·just now

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

6
Hhuang_ke·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

462
Oops_wang·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

314
Zzhu_zong·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

200
Sswoole_lee·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

20
Zzhou_yi·2 hours agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

4
Aalice_devMod·2 days agoLevel 6

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

266
Kkernel_panic·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

96
Aalice_dev·2 days ago

One counter-example: below edge-computing-resources 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

367
Kkernel_panic·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

123
Rran_bo·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

57
Sslow_query·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

229
Nnikic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

5
Llinlin·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

14
Lli_ming·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

302
Nnikic·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

45
Ddev_zhou·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

17
Oops_wang·1 hour ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

91
Sswoole_lee·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

4
Wwinter·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

3
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

402
Mmike_xuMod·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Wwinter·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

219
Wwinter·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

2
Lli_ming·28 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

2

This is the post detail page /en/c/edge-computing-resources/post/p11. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →