LOL your last post got helpful answers until you were super rude and an admin deleted it.
this post was submitted on 03 Dec 2023
-6 points (28.6% liked)
General Programming Discussion
7809 readers
4 users here now
A general programming discussion community.
Rules:
- Be civil.
- Please start discussions that spark conversation
Other communities
Systems
Functional Programming
Also related
founded 5 years ago
MODERATORS
Third time’s the charm!
I knew that you'd like it.
ZOG yes. Until you came in it.
Try asking this question 1 million times
That's what I'm doing.
Maybe producer consumer?
Producer spits out all the messages to send out onto a message queue, fifo or whatever suits u.
Parrallelizable consumers (think deployed containers) listen to queue and execute request, get response and save it
Scale consumer count up or down as you need to deal with ratelimits
What question have you answered?
Yes, I think the second. You have a pool of 100 http clients and a queue of one million requests and a queue to accept the responses as the clients complete, and a little machine that waits for capacity in the client queue to send the next request until there are no more requests. If the response is important to this process, your machine is also pulling from the response queue as available and computing whatever it needs from that, for example to decide whether to abort the rest of the requests. Any other use of the responses can be handled outside this loop.
The other way would work fine, but I think it's actually slightly more complicated and slower because you now have a queue of 10000 batches of 100 requests each and the machine has to watch for all one hundred clients to complete before sending off the next batch. Otherwise, it's the same situation.
Yeap