I've been trying to wrap my head around how to port a "simple" processing stream context from Scala/Akka to Typescript equivalent that includes parallel processing and asynchronicity via promises.
Assuming a straight forward stream, here some pseudo code from Akka with some simple comments.
Source.from(GooglePubSubSubscription) // read from
.filter(validateMessage)
.map(convertMessagePayload)
.mapAsync(4)(callOtherApi) // the API, max 4 concurrent calls active at all times
.map(acknowledgeMessageOnPubSub) // on success of third party API ack message on pubsub
The problem with frameworks like async or p-limit is that the workload is not known in advance, as it is a continuous stream. So needs additional work around error/retry handling and managing input/output.
I tried rxjs, where I failed ultimately due to back pressure support across splitting/joining the stream. And I could find find a stage that had a limited concurrency option.
I ended up writing a minimal implementation on top of async's queue using the callbacks for drain, etc. but /feels/ to me that this should be a solved problem and there is an existing library in JS that can model these kind of streams.
Any recommendations?