NServiceBus - NServiceBus.Persistence.CosmosDB.SagaUpdate.Conflict

Question

I am facing some issues with the NServiceBus. We are getting the below errors when we process multiple messages. I am using NServiceBus 7.7.0 version.

Below is the NServiceBus configuration.

UseNServiceBus(context =>
            {
                var appInsightsOptions = configuration.GetRequiredSection("AppInsights").Get<AppInsightsOptions>();
                NServiceBus.Logging.ILoggerFactory nservicebusLoggerFactory = new ExtensionsLoggerFactory(loggerFactory: new SerilogLoggerFactory());
                NServiceBus.Logging.LogManager.UseFactory(loggerFactory: nservicebusLoggerFactory);
                var endpointConfiguration = new EndpointConfiguration(endpointName);
                endpointConfiguration.EnableApplicationInsights(new TelemetryConfiguration(appInsightsOptions.InstrumentationKey));
                
                var transport = endpointConfiguration.UseTransport<AzureServiceBusTransport>();
                transport.ConnectionString(GetTransportConnectionString(configuration));
                //transport.Transactions(TransportTransactionMode.TransactionScope);
                endpointConfiguration.UseSerialization<NewtonsoftSerializer>();
                endpointConfiguration.UsePersistence<CosmosPersistence>()
                    .CosmosClient(new CosmosClient(cosmosConnectionString))
                    .DatabaseName(configuration.GetConnectionString("DATABASENAME"))
                    .DefaultContainer(containerName: "sagastore", partitionKeyPath: "/id")
                    .DisableContainerCreation();

                endpointConfiguration.LimitMessageProcessingConcurrencyTo(1);

                endpointConfiguration.SendFailedMessagesTo("error");
                endpointConfiguration.AuditProcessedMessagesTo("audit");
                endpointConfiguration.AuditSagaStateChanges(serviceControlQueue: "audit");
                endpointConfiguration.EnableInstallers();

                return endpointConfiguration;

            }

Answer 1

By default the NServiceBus.Persistence.CosmosDB implementation uses optimistic concurrency control. Depending on the "scatter-gather" nature of the saga optimistic concurrency might not be the best choice because it can lead to the effects you are seeing. When multiple messages try to update the same saga instance at the same time, one update wins and all others will have to roll back and retry. The saga concurrency behavior is explained in more details in the saga concurrency documentation with specific guidance for CosmosDB.

You can switch from optimistic concurrency to pessimistic concurrency by overriding the locking mode

var sagaPersistenceConfiguration = persistence.Sagas();
sagaPersistenceConfiguration.UsePessimisticLocking();

This is an endpoint level switch and pessimistic concurrency comes with its own sets of tradeoffs as explained in the internals section of the documentation. You can either globally switch the mode for the endpoint that saga is running, or think about isolating this specific saga out into its own dedicated endpoint.