This clojure-jobs bug hid for three years before anyone reported it
It took me two weeks of on-and-off digging and plenty of wrong turns. Writing the process down as it happened so the next person spends less time.
// Minimal reproduction: you must use a real long-tail distribution here.
// Uniform load-test traffic will never trigger this.
func (s *Server) handle(ctx context.Context) error {
conn, err := s.pool.Acquire(ctx)
if err != nil {
return fmt.Errorf("acquire: %w", err)
}
defer conn.Release()
return s.do(ctx, conn)
}We also fixed monitoring along the way: replaced average-based alerts with percentiles and split them per endpoint. False alerts dropped by about seventy percent and the on-call rotation visibly cheered up.
One last trap: in container environments remember to adjust the memory-related parameters in step. Otherwise the host limit and the process expectation disagree, and the symptom is intermittent, unreproducible failure.
Worth noting: the official docs do cover this, just in a very inconspicuous spot. I only found it reading the source comments, where the author explains the reasoning — roughly "so that it degrades into predictable behaviour in extreme cases".