Update: After some research, I've learned that throttle has the capability to drop excess function invocations, making it unsuitable for my needs. I am still seeking an idiomatic solution to process every item in a queue at an appropriate pace without losing any items.
I'm currently developing a node application that interacts with an API having rate limits. The rate at which I can create calls surpasses the rate at which I can send them. I aim to consume a queue of calls while ensuring they are processed at the right speed and none are overlooked. To illustrate my issue, here is a small TypeScript test I have created:
import * as _ from "lodash";
let start = new Date().getTime();
function doLog(s: string) {
let elapsed = new Date().getTime() - start;
console.log(`${s} ${elapsed}`);
}
let array = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'];
let throttled = _.throttle(doLog, 100);
array.forEach(s => throttled(s));
My expectation was to see output similar to:
a 2
b 101
c 203
d 302
e 405
f 502
g 603
h 706
i 804
j 902
However, what I actually observe is:
a 2
j 101
Here are some peculiar observations I've made:
- With a throttle of 100ms, the size of the array doesn't seem to matter: only the first and last elements are printed, regardless of having 2 elements or 20.
- Using a throttle of 1ms, I see the printing of 3-6 front elements from the array and the last element.