David Backeus

We run 7x SolidQueue workers with 150 threads each to reach 1000+ concurrent requests to OpenAI. However what we found when pushing this is that OpenAI’s response times start tanking, even though we’re within our rate limits (we’re at the highest documented tier for limits), so scaling above this isn’t helpful. Perhaps the most interesting thing about this is that we monkey patched solid queue to stop preventing us from running with fewer database connections than worker threads (we run with a pool size of 15). We did this since we didn’t fancy complicating our infrastructure by deploying pgbouncer.