Kafka Message loss when Kafkaserver goes down

90 Views Asked by At

I have a scenario that when Kafka server or Zookeeper both goes down, but the producer application trying to send a message to Kafka broker.

When the Kafka broker or Zookeeper comes online, messages is getting loss, and i see the producer delivery result offset has Offset: -1001 and Partition [Any]

Below are my configuration to reproduce

new ProducerConfig
{
    BootstrapServers = <BROKET_END_POINT>,
    MessageMaxBytes = 20971520,
    MessageCopyMaxBytes = 20971520,
    LogConnectionClose = false,    
    BrokerAddressFamily = BrokerAddressFamily.V4,
    ConnectionsMaxIdleMs = 0,
    //https://github.com/Azure/azure-event-hubs-for-kafka/issues/139
    SocketKeepaliveEnable = true,
    MetadataMaxAgeMs = 180000,
    //https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations
    RequestTimeoutMs = 60000,
    MessageTimeoutMs = 0,
    MessageSendMaxRetries = int.MaxValue,
    RetryBackoffMs = 100,
    EnableIdempotence = true,
    Acks = Acks.All,
    BatchSize = 2000000,
    LingerMs = 0,
    CompressionType = CompressionType.Gzip
    SocketNagleDisable = true
};

I used the above setting to produce the message using Confluent kafka nuget package version 1.8.2 from .Net code.

Appreciate your inputs in advance

0

There are 0 best solutions below