LLLlamaindex · Toolchain
Community homeHot discussions Latest posts Best of Knowledge base Resources About

Llamaindex · Toolchain · Two weeks with llamaindex-tools: what feels right and what drives me up the wall

889
LLr/llamaindex-tools·posted by linlin·3 days agoPostmortem

Two weeks with llamaindex-tools: what feels right and what drives me up the wall

Short version: llamaindex-tools needs almost no tuning at small and medium scale — the point where it starts to hurt is much further out than most people assume. Full measurements below.

The first thing was to collapse the variables. We were changing config and upgrading the version at the same time, and afterwards nobody could say which change caused what. We rolled back to moving one variable at a time, re-ran three times, and only then did the curve settle. Tedious, but not skippable.

-- The query that broke: a full scan over 20M rows.
-- A composite index took P99 from 1.8s down to 42ms.
SELECT id, title, created_at
  FROM posts
 WHERE community_id = ?
   AND status = 1
 ORDER BY score DESC
 LIMIT 20;

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not llamaindex-tools itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

Worth noting: the official docs do cover this, just in a very inconspicuous spot. I only found it reading the source comments, where the author explains the reasoning — roughly "so that it degrades into predictable behaviour in extreme cases".

370 comments

370 comments

· first 120 loaded
M
Kkernel_panic·5 hours ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

520
Wwinter·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

520
Rran_bo·2 days ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

487
Rrase·1 hour agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

433
Kkite·2 days ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

4
Zzhu_zong·2 days ago

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

425
Kkite·2 days ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

421
Llinlin·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

413
Ttang_hao·2 days ago

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

365
Zzhu_zong·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

361
Rrase·yesterday

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

346
Aalice_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

280
Ddev_zhou·12 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

48
Rran_bo·28 minutes agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

71
Sswoole_leeOP·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

2
Zzhu_zong·2 days agoedited

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

510
Zzhu_zongOPMod·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

247
Bbob_chenOP·2 days ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

243
Ddev_zhou·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

276
Nnikic·2 days agoedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

260
Nnikic·2 days ago

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

259
Wwinter·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

246
Bbob_chen·12 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

242
Mmike_xu·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

234
Bbob_chen·2 hours ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

72
Sswoole_lee·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

1
Kkite·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

226
KkiteMod·5 hours agoedited

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

224
Rrase·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

197
Wwinter·2 days ago

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

187
Lli_ming·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

182
Zzhou_yi·2 hours agoedited

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

19
Oops_wang·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

164
Kkite·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

129
Zzhu_zong·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

124
Nnikic·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

111
WwinterOP·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

25
Ddev_zhou·2 days ago

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

105
Llinlin·2 days agoedited

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

90
Ddev_zhou·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

84
Nnikic·yesterday

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

344
Zzhu_zong·12 minutes agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

408
Zzhu_zong·5 hours agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

444
Hhuang_ke·2 days ago

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

224
WwinterOP·28 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

110
WwinterMod·2 days agoLevel 6

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

2
KkiteMod·yesterday

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

4
Ttang_haoOP·2 days ago

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

271
Llinlin·28 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

145
Sswoole_lee·2 days agoLevel 6

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

14
Zzhu_zong·2 days agoedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

312
Llinlin·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

508
Llinlin·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

277
Ddev_zhou·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

404
Mmike_xu·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

19
LlinlinOP·just now

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

6
Ddev_zhou·2 days ago

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

43
Aalice_dev·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

14
Kkite·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

19
Zzhu_zong·2 days ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

159
Kkernel_panic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

1
Ddev_zhouOP·just now

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

18
Aalice_devOP·2 days agoedited

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

10
Sswoole_lee·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

62
Bbob_chenMod·5 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

460
Ttang_hao·12 minutes ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

84
Mmike_xuOP·28 minutes agoedited

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

277
Rran_bo·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

48
Hhuang_ke·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

3
Rran_bo·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

61
Hhuang_ke·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

62
Cchen_devMod·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

85
Oops_wangMod·12 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

6
Wwinter·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

85
Sslow_query·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

2
Ttang_haoOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

12
Ttang_hao·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

61
Cchen_dev·12 minutes ago

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

46
Sswoole_lee·just now

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

128
Cchen_dev·28 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

471
Sslow_query·yesterdayedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

352
Zzhu_zong·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

98
Nnikic·2 hours agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

174
Sswoole_lee·2 days agoLevel 6

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

17
Zzhou_yi·2 days agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

7
Mmike_xu·2 days agoLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

1
Zzhou_yi·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

88
Llinlin·2 days agoLevel 6

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

386
Hhuang_ke·28 minutes agoLevel 6

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

284
Bbob_chen·12 minutes agoLevel 6

Saved. I am reworking this area this week — this saves a lot of wrong turns.

88
Llinlin·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

2
Wwinter·3 minutes ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

68
Kkite·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

118
Sslow_query·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

144
Ddev_zhou·2 days agoLevel 6

I just read the llamaindex-tools source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

484
Hhuang_ke·2 days ago

This is not a llamaindex-tools problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

71
Cchen_dev·2 days ago

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

123
Bbob_chen·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

349
Llinlin·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

461
Zzhu_zong·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

114
Lli_ming·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

94
Lli_ming·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

104
Sswoole_lee·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

8
Mmike_xu·yesterday

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

40
Cchen_devOP·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

290
Sslow_query·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

36
Ddev_zhou·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

33
Bbob_chen·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

28
Oops_wang·2 days agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

17
Rrase·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

16
Ddev_zhou·yesterday

Saved. I am reworking this area this week — this saves a lot of wrong turns.

12
Kkite·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

3
Cchen_dev·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

319
Rrase·just now

One counter-example: below llamaindex-tools 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

10
Rran_bo·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

474
Sswoole_lee·just now

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

6
Ddev_zhou·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

6
Cchen_dev·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

2
Zzhu_zong·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

1

This is the post detail page /en/c/llamaindex-tools/post/p3. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →