I have a scenario that when Kafka server or Zookeeper both goes down, but the producer application trying to send a message to Kafka broker.
When the Kafka broker or Zookeeper comes online, messages is getting loss, and i see the producer delivery result offset has Offset: -1001 and Partition [Any]
Below are my configuration to reproduce
new ProducerConfig
{
BootstrapServers = <BROKET_END_POINT>,
MessageMaxBytes = 20971520,
MessageCopyMaxBytes = 20971520,
LogConnectionClose = false,
BrokerAddressFamily = BrokerAddressFamily.V4,
ConnectionsMaxIdleMs = 0,
//https://github.com/Azure/azure-event-hubs-for-kafka/issues/139
SocketKeepaliveEnable = true,
MetadataMaxAgeMs = 180000,
//https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-configurations
RequestTimeoutMs = 60000,
MessageTimeoutMs = 0,
MessageSendMaxRetries = int.MaxValue,
RetryBackoffMs = 100,
EnableIdempotence = true,
Acks = Acks.All,
BatchSize = 2000000,
LingerMs = 0,
CompressionType = CompressionType.Gzip
SocketNagleDisable = true
};
I used the above setting to produce the message using Confluent kafka nuget package version 1.8.2 from .Net code.
Appreciate your inputs in advance