Flink- force Checkpoint

Question

• Currently, flink application is configured and implemented to create avro files on every checkpoint.

• Is is possible to force the flink application to create avro file on-demand, instead of configurable time interval.

• Is there any REST APIs or any other java implementation and configuration to force checkpoint.

Environment

• Flink version - 1.15.4
• Jdk 8

Answer 1

Is is possible to force the flink application to create avro file on-demand, instead of configurable time interval.

Assuming you are using StreamingFileSink.forBulkFormat to produce in avro format, you can implement custom CheckpointRollingPolicy and checkpoint on processing time or on a specific event:

public final class CustomCheckpointRollingPolicy<IN, BucketID>
        extends CheckpointRollingPolicy<IN, BucketID> {

    private static final long serialVersionUID = 1L;

    @Override
    public boolean shouldRollOnEvent(PartFileInfo<BucketID> partFileState, IN element) {
        return false;
    }

    @Override
    public boolean shouldRollOnProcessingTime(
            PartFileInfo<BucketID> partFileState, long currentTime) {
        return false;
    }
}
...
StreamingFileSink
      .forBulkFormat(outputBasePath, AvroWriters.forGenericRecord(schema))
      .withRollingPolicy(new CustomCheckpointRollingPolicy())
      .build()

Is there any REST APIs or any other java implementation and configuration to force checkpoint.

Yes, you can trigger savepoint without cancel, which will trigger checkpoint. The corresponding REST API endpoint is /jobs/:jobid/savepoints. See REST API #jobs-jobid-savepoints section for details

UPD: It's possible to trigger checkpoint via a dedicated /jobs/:jobid/checkpoints POST endpoint. See REST API #jobs-jobid-checkpoints-1 section for details.