Apache Flink Checkpoining (Manually put a value into RocksDB Checkpoint and retrieve during recovery or Restart)

Question

We have a scenario where we have to persist/save some value into the checkpoint and retrieve it back during failure recovery/application restart.

We followed a few things like ValueState, ValueStateDescriptor still not working. https://github.com/realtime-storage-engine/flink-spillable-statebackend/blob/master/flink-spillable-benchmark/src/main/java/org/apache/flink/spillable/benchmark/WordCount.java

https://towardsdatascience.com/heres-how-flink-stores-your-state-7b37fbb60e1a https://github.com/king/flink-state-cache/blob/master/examples/src/main/java/com/king/flink/state/Example.java

We can't externalize it to a DB as it may cause some performance issues. Any lead to this will be helpful using checkpoint. How to put and get back from a Checkpoint?

Answer 1

All of your managed application state is automatically written into Flink checkpoints (and savepoints). This includes

• keyed state (ValueState, ListState, MapState, etc)
• operator state (ListState, BroadcastState, etc)
• timers

This state is automatically restored during recovery, and can optionally be restored during manual restarts.

The Flink Operations Playground shows how to work with checkpoints and savepoints, and lets you observe their behavior during failure/recovery and restarts/rescaling.

If you want to read from a checkpoint yourself, that's what the State Processor API is for. Here's an example.