We're building a Flink app that consumes events from different Kafka topics. This app uses the bounded out of order watermark strategy on the source. During normal execution everything works as expected and we do not get any late arriving data (based on watermarks), but on checkpoints/ savepoints restores we're getting late arriving events, no matter how much we increase the out of order bound. Did anyone ever encounter this situation?
Flink marks source late arriving events
38 Views Asked by Dan At
1
There are 1 best solutions below
Related Questions in APACHE-FLINK
- Fine grained resource mangement and heap memory in flink task slot
- Does parallel flink tasks affect each other if they are unioned at the end?
- I am facing issue with ParquetFileWriting n hdfs in flink where parquet file size is around 382 KB . I want the parquet file in MB
- Apache Flink (AWS) does not recognize saved temporary function
- Flink 1.19 error Cannot determine simple type name "com"
- Unsupported options found for 'hudi'
- Flink 1.18 register custom API endpoint handler
- Flink Stuck on Broadcast
- Blunder about RichCoFlatMapFunction in flink 1.17.2 according to the official leanring guide
- Is there a way to store & retrieve a window's state in flink
- puzzled with flink window state
- Flink 1.15.2 OOM issue due to RocksDB
- How to create custom metrics with labels (python SDK + Flink Runner)
- flink-rpc-akka-loader - Security Vulnerability Issues
- I am new to Apache Flink and getting error FileNotFoundError: [WinError 2] at in_streaming_mode() The system cannot find the file specified
Related Questions in FLINK-STREAMING
- Fine grained resource mangement and heap memory in flink task slot
- Flink 1.19 error Cannot determine simple type name "com"
- Getting FlinkRuntime Exception during oracle exactly once jdbc sink
- Is there a way to store & retrieve a window's state in flink
- puzzled with flink window state
- Flink 1.15.2 OOM issue due to RocksDB
- If I emit an event from an operator after holding it in state for certain duration will the downstream operator accept it if it is past the watermark?
- How to write to Kafka Topic(Or to a file) from a Flink Stream
- Flink marks source late arriving events
- Why is flink UI not showing the right numbers?
- Union of bounded and unbounded streams in flink
- gRPC Connection Cancelled with "Multiplexer Hanging Up" Error in PyFlink
- Delta Lake as ingress for Flink Stateful Functions
- implement custom partitioning with windowAll()
- implementation of RoundRobin partitioning in Apache Flink
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
Watermarks are not checkpointed, so after recovery, the watermarks have to be re-established based on the events processed after the checkpoint. If one or more sources are more-or-less idle at that point in time, this could explain why the behavior is so different compared to the situation before the restart.