Performance issue with ZooKeeper ID level locking - Need optimization guidance

Question

We are currently implementing domain ID level locking using Apache ZooKeeper to prevent concurrent modifications of domain objects by multiple actors. The locking mechanism is applied to objects based on their unique IDs.

lock object write (object id) ..update the object release the lock.

We've encountered a significant performance problem when applying the lock before each update operation. Profiling our application has revealed that the following methods are consuming an excessive amount of time:

It only takes 2 seconds for the business process to complete without zookeeper and over 80 seconds with zookeeper.

1. org.apache.curator.RetryLoop.callWithRetry () 48,855 ms (40.1%) 758 ms (1.5%)

2. org.apache.curator.framework.imps.CuratorFrameworkImpl.start () 16,795 ms (13.8%) 16,795 ms (34.1%)

3. com.company.coordination.permit.imps.PessimisticLock.release () 25,080 ms (20.6%) 2,034 ms (4.1%)

I have provided screenshots of the profiling results and our ZooKeeper configuration settings for reference.

I suspect that there might be an issue with my ZooKeeper configuration or the approach to implementing domain ID level locking.

We have tried updating to the latest Curator and Zookeeper and we have the same issue. Different environments linux vs windows is the same result. Unable to find the root cause of performance issue

Any assistance or suggestions from the community would be greatly appreciated. Thank you in advance!

ZooKeeper Configuration:

#Wed, 11 Oct 2023 11:22:57 +0100
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=xxx/zookeeper-3.8.2/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpHost=0.0.0.0
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true


dataLogDir=xxx/zookeeper-3.8.2/logs

maxClientCnxns=200

autopurge.snapRetainCount=5

autopurge.purgeInterval=24

admin.serverPort=9876

Profiling Snapshots: