Spark streaming has a state, but how should pass between the blue/green environments? I'm on Databricks, which allows me to create clones of tables, so the new environment gets snapshots of the table as seen by the production. After the deployment, they will work on the same data tables, and some of them are ingested by streaming code. It would be nice if the current streaming status is somehow "cloned" like the tables. Does this make sense? The lack of a state forces the new environment to execute some full "reload", and this is something I like to avoid.
I wonder if someone already faced this problem and how he/she solved that. Any help is appreciated.