I am using project reactor flux to read data from DB and then publish to a messaging system.
The flux consists of many steps, i.e.
// we have a collection of things initially in the sink
mainFlux = sink.asFlux()
.filter(this::checkForEmptyDataEnvelopeCollection)
// do other transformations/checks here
.flatMap(Flux::fromIterable) // parallelize
.map(...) // some prep work
.map(mainPublisher::publish) // publishing happens here
.retryWhen( ... ) // publish to a different/error destination
.subscribe(... ) // report completion here
The backpressure signal comes from the publisher (or from retry/error publisher).
What would be the best way to control backpressure for the whole pipeline?
The publishing is async, it returns CompletableFuture and I want to apply it if I have a certain number of futures in flight and I want to apply it to the whole pipeline - I want to stop getting things from the initial sink.
Do I have to have some kind of global state outside of flux?