KAKafka · English
Community homeHot discussions Latest posts Best of Knowledge base Resources About

Kafka · English · This kafka-en bug hid for three years before anyone reported it

4
KAr/kafka-en·posted by ran_bo·32 minutes agoInternals

This kafka-en bug hid for three years before anyone reported it

Some background first. Our setup is kafka-en plus three downstream services, seven figures of daily requests, peaking around nine in the evening.

What genuinely surprised me was the tail. The average looked great while P99 jumped by an order of magnitude past some threshold. The cause was not kafka-en itself but our upstream connection reuse — the load test traffic was too clean and hid the long-tail requests.

Image placeholder · object storage in production
POST /api/uploads → CDN origin pull
3 comments

3 comments

M
KkiteOP·3 minutes ago

Has anyone run a controlled experiment? I did, reducing it to a single variable, and the difference was 4% — within noise. So I suspect the main cause is something else.

295
Oops_wang·just now

This is not a kafka-en problem, it is a usage problem. The docs say this API is not thread-safe and you must lock around it yourself.

7
Rran_bo·12 minutes ago

Worth learning from this debugging approach. We went straight at the logs and took a much longer route.

3

This is the post detail page /en/c/kafka-en/post/p12. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →