Vue · Remote · I upgraded our production vue-remote and hit these 11 landmines

1.5K
VUr/vue-remote·posted by winter·yesterdayHelp

I upgraded our production vue-remote and hit these 11 landmines

Most vue-remote articles stop at "how to use it" and never cover "when not to use it". This is an attempt at the second half.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not vue-remote itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

Image placeholder · object storage in production
POST /api/uploads → CDN origin pull
296 comments

296 comments

· first 120 loaded
M
Sswoole_lee·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

518
Sswoole_lee·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

503
Aalice_dev·2 hours ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

468
LlinlinOP·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

388
Rrase·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

440
Wwinter·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

434
Cchen_dev·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

408
Sslow_query·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

375
KkiteOP·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

369
Mmike_xuOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

367
Cchen_devMod·2 days ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

411
Llinlin·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

20
Sswoole_lee·2 days agoedited

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

411
Wwinter·2 days ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

406
Wwinter·1 hour ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

393
Ttang_hao·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

1
Zzhou_yi·3 minutes agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

63
Kkernel_panic·2 days agoedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

9
Aalice_dev·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

380
Zzhu_zong·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

371
Lli_mingOP·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

308
Oops_wang·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

107
LlinlinOP·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

488
WwinterOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

10
Ddev_zhou·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

363
Rrase·12 minutes agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

316
Zzhu_zongMod·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

494
Sswoole_lee·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

295
Zzhou_yi·1 hour agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

154
Llinlin·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

239
Sslow_query·2 hours agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

52
Rrase·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

256
Sswoole_lee·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

110
Bbob_chen·12 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

334
Hhuang_ke·5 hours ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

269
Kkernel_panic·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

269
Bbob_chen·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

119
Rran_bo·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

22
Cchen_dev·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

492
Ddev_zhou·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

5
Rrase·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

3
Nnikic·28 minutes agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

27
Lli_ming·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

21
Rran_bo·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

6
Mmike_xu·2 days ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

362
WwinterOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

220
Cchen_dev·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

345
Nnikic·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

335
Zzhu_zong·2 days ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

20
Ttang_hao·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

287
Hhuang_ke·28 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

281
Oops_wang·3 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

64
Ddev_zhou·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

3
Oops_wang·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

246
Llinlin·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

199
Wwinter·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

477
Llinlin·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

199
Kkernel_panic·2 days ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

198
Kkite·2 days ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

185
Ttang_haoMod·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

158
Hhuang_keMod·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

148
Sslow_query·5 hours ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

5
Aalice_dev·2 hours ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

145
Zzhu_zong·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

98
Aalice_devOP·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

520
Kkite·3 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

88
Nnikic·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

75
Llinlin·3 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

74
Rrase·2 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Aalice_dev·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

244
Cchen_dev·1 hour ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

176
Llinlin·28 minutes ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

94
Ddev_zhouOP·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

60
Rran_boOP·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

174
Bbob_chen·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

163
Lli_mingOP·2 days agoLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

141
Sslow_queryOP·1 hour ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

10
Cchen_devOP·1 hour ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

4
Aalice_dev·yesterdayedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

60
Ddev_zhouMod·5 hours ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Wwinter·2 days agoedited

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

236
Zzhou_yi·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

249
Ttang_hao·2 days ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

3
Oops_wang·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

2
Kkernel_panicOP·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

299
Sslow_query·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

353
Ddev_zhou·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

265
Sswoole_leeOP·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

135
Ddev_zhou·2 days agoedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

1
Aalice_dev·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

507
Aalice_dev·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

154
Kkite·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

146
Zzhou_yi·3 minutes agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

284
Wwinter·2 days agoedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

113
Sswoole_lee·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

57
Bbob_chen·2 days agoedited

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

57
Rran_bo·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

34
Kkernel_panic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

54
Kkernel_panic·just now

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

54
Sslow_query·2 days ago

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

28
Wwinter·2 days ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

347
Bbob_chen·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

329
Zzhou_yi·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

28
Ttang_hao·1 hour ago

I just read the vue-remote source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

27
Mmike_xu·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

25
Lli_ming·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

21
Ttang_hao·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

14
Cchen_dev·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

13
Cchen_dev·just now

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

11
Mmike_xu·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

131
Lli_ming·yesterday

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

10
Wwinter·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

13
Rrase·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

302
Mmike_xu·5 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

4
Oops_wang·just now

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

2
Hhuang_ke·2 days ago

This is not a vue-remote problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

406
Oops_wangOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

5
Mmike_xu·12 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

1
Llinlin·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

1
Zzhou_yi·2 days agoedited

One counter-example: below vue-remote 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

1

This is the post detail page /en/c/vue-remote/post/p2. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →