Polars · Beginners · polars-newbie under load on an 8-core 16GB box — full benchmark scripts included

3
POr/polars-newbie·posted by bob_chen·6 hours agoOpen source

polars-newbie under load on an 8-core 16GB box — full benchmark scripts included

It took me two weeks of on-and-off digging and plenty of wrong turns. Writing the process down as it happened so the next person spends less time.

The first thing was to collapse the variables. We were changing config and upgrading the version at the same time, and afterwards nobody could say which change caused what. We rolled back to moving one variable at a time, re-ran three times, and only then did the curve settle. Tedious, but not skippable.

Self-host — full control44%
Managed service — less work28%
Hybrid: core self-hosted17%
Not decided yet11%

5 votes total

2 comments

2 comments

M
Cchen_dev·3 minutes ago

A question: what changes in a container with a 512Mi memory limit? That is how we run it in production.

46
Ttang_hao·3 minutes ago

Saved. I am reworking this area this week — this saves a lot of wrong turns.

1

This is the post detail page /en/c/polars-newbie/post/p5. Posts and comments are generated deterministically from a seeded PRNG, so the same post always renders the same content and the link can be shared, reloaded and indexed. In production this page reads MySQL for the post, Redis for hot-post caching, and fetches the whole comment tree in a single query on the path column.

See the database schema →