OWASP · Open source · Hiring: remote owasp-opensource engineer (full-time, long-term)

1.2K
OWr/owasp-opensource·posted by nikic·just nowAnnouncement

Hiring: remote owasp-opensource engineer (full-time, long-term)

Short version: owasp-opensource needs almost no tuning at small and medium scale — the point where it starts to hurt is much further out than most people assume. Full measurements below.

Order of investigation, by return on effort: 1. Check downstream latency first — usually it is not your problem 2. Then pool hit rate and wait-queue length 3. Only then GC and allocation 4. Suspect the framework last

On trade-offs, my view is this: if nobody on the team owns this area long-term, do not introduce a second mechanism. With two coexistence you first have to work out which one is even in play when things break, and that costs far more than the performance you saved.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not owasp-opensource itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

One last trap: in container environments remember to adjust the memory-related parameters in step. Otherwise the host limit and the process expectation disagree, and the symptom is intermittent, unreproducible failure.

702 comments

702 comments

· first 120 loaded
M
Aalice_dev·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

519
Ttang_hao·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

515
Rran_bo·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

513
Rran_boOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

407
Llinlin·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

458
Kkite·12 minutes agoedited

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

443
Zzhou_yi·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

403
Oops_wangOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

339
Bbob_chenMod·12 minutes ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

326
Zzhou_yi·28 minutes ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Sslow_query·3 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

417
Kkite·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

320
Mmike_xuOP·2 hours ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

55
Zzhou_yi·5 hours ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

217
Bbob_chen·2 days agoLevel 6

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

340
Lli_ming·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

126
Nnikic·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

77
Rrase·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

27
Rrase·2 days agoLevel 6

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

465
Oops_wang·2 days agoLevel 6

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

185
Rran_bo·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

227
Zzhou_yi·2 hours ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

160
Sslow_query·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

334
Kkite·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

205
Sswoole_lee·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

11
Sslow_query·2 days ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

146
Aalice_dev·2 days agoeditedLevel 6

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

123
Rrase·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

7
Bbob_chenOP·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

346
Hhuang_ke·2 days agoLevel 6

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

1
Lli_ming·28 minutes ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

90
RraseOP·1 hour ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

12
Sswoole_lee·just now

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

333
Sswoole_leeOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

197
Aalice_dev·2 days agoLevel 6

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

69
Cchen_dev·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

165
Aalice_devOP·2 days agoLevel 6

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

265
Lli_ming·2 hours agoeditedLevel 6

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

265
Sslow_queryOP·2 hours agoLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

13
Ddev_zhou·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

52
Llinlin·1 hour ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

1
Sswoole_lee·2 days ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

308
Sswoole_lee·just nowLevel 6

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

2
Sswoole_lee·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

215
Sslow_query·2 days agoeditedLevel 6

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

51
Hhuang_ke·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

5
Sslow_query·2 days ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

5
Zzhou_yi·2 days ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

279
Kkernel_panic·2 days ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

322
Hhuang_keOP·28 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

254
Ddev_zhouOP·2 days agoedited

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

418
Rran_boMod·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

190
Rrase·2 days agoedited

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

145
Sslow_query·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

303
Nnikic·yesterday

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

289
Aalice_dev·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

276
Sslow_queryOP·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

491
Ddev_zhou·2 days agoedited

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

320
Sswoole_lee·2 days ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

1
Nnikic·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

241
Llinlin·28 minutes ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

235
Sswoole_lee·3 minutes ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

11
Kkite·2 days ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

10
Sswoole_lee·5 hours ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

234
Cchen_dev·12 minutes ago

We have run this in production for two years without hitting it. That said, we never reached this scale, so our experience is not really evidence here.

225
Hhuang_ke·2 days ago

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

223
Zzhu_zongOP·2 hours ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

147
Cchen_devOP·2 hours ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

131
Bbob_chenMod·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

228
Cchen_dev·12 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

28
Aalice_dev·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

19
Kkernel_panic·5 hours ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

185
Zzhu_zongOP·28 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

107
Sswoole_lee·2 days agoedited

Saved. I am reworking this area this week — this saves a lot of wrong turns.

133
Bbob_chen·28 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

130
Sslow_query·2 days ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

126
Hhuang_ke·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

120
Ttang_hao·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

382
Zzhu_zong·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

38
Sslow_query·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

114
Ddev_zhou·1 hour ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

78
Mmike_xu·28 minutes ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

70
Ttang_haoOP·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

1
KkiteOP·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

1
Lli_ming·yesterdayedited

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

55
Hhuang_ke·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

48
Wwinter·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

43
Ttang_hao·2 days ago

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

27
Ttang_hao·2 days ago

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

25
RraseOP·2 days ago

Can you give a minimal reproduction? I ran it locally for ten minutes and could not reproduce on macOS with the latest version.

232
Bbob_chen·2 days agoedited

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

20
Llinlin·12 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

19
Sswoole_lee·2 days agoedited

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

124
Oops_wang·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

168
Oops_wang·2 days ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

36
Mmike_xu·28 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

20
Rrase·2 days ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

226
Bbob_chen·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

2
Ddev_zhou·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

1
Cchen_devOP·2 days ago

This is not a owasp-opensource problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

10
Kkite·2 days agoedited

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

17
Lli_ming·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

16
LlinlinOP·2 days ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

288
Rrase·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

122
Cchen_dev·2 days ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

395
Cchen_devMod·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

172
Lli_ming·yesterday

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

163
Zzhu_zongMod·just now

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

112
Lli_ming·2 days ago

I just read the owasp-opensource source — the author actually explains the reasoning in a comment, roughly "so that it degrades into predictable behaviour in extreme cases".

16
Ttang_hao·2 days ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

15
Cchen_dev·2 days ago

Sharing our numbers, 8 cores 16GB, same scenario:

| Concurrency | P50 | P99 |
|---|---|---|
| 200 | 12ms | 88ms |
| 500 | 31ms | 340ms |

P99 clearly collapses at 500 concurrency, which lines up with your knee point.

14
Kkite·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

311
Zzhu_zong·1 hour ago

I see point 3 differently. The trade-off depends on your read/write ratio: read-heavy with little writing means caching actually widens the inconsistency window.

9
Bbob_chen·2 days ago

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

25
Cchen_dev·2 days ago

One counter-example: below owasp-opensource 7.4 the semantics of that code are different, so do not copy it verbatim. We got burned in staging and rolled back once.

2
Rrase·12 minutes ago

Thanks for sharing real numbers — far more useful than the articles that only cover concepts.

4
Kkernel_panic·12 minutes agoedited

Agreeing with the above. One addition: with this option enabled the GC count in your metrics doubles, so adjust the alert threshold at the same time or it will keep firing.

1
Aalice_dev·2 hours ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

76
Ttang_haoMod·2 days ago

There is actually a simpler fix that needs no architecture change: move this check up to the gateway and the problem disappears. The cost is one extra lookup at the gateway.

27
Kkernel_panic·2 days ago

This matches what we see in production. We only hit it past 3k QPS; the earlier load tests showed nothing — the test traffic was too clean, with no long-tail requests.

1

This is the post detail page /en/c/owasp-opensource/post/p1. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →