I used three clients to write to three different databases and found that the writing speeds varied greatly

Question

I used three clients to write to three different databases and found that the writing speeds varied greatly.

I start a cnosdb cluster, 3 meta 3 tskv:

cnosdb-meta run --config /etc/config_8901.toml
cnosdb-meta run --config /etc/config_8911.toml
cnosdb-meta run --config /etc/config_8921.toml
cnosdb run --config /etc/config_8902.toml
cnosdb run --config /etc/config_8912.toml
cnosdb run --config /etc/config_8922.toml

and create 3 databases in a same tenant, as follow:

create database benchmark001 with ttl '9999999' shard 9;
create database benchmark002 with ttl '9999999' shard 9;
create database benchmark003 with ttl '9999999' shard 9;

and then, I use the tsdb-comparisons tool load data;

nohup cat /data/cnosdb--iot-seed-123-scale-100-2023-15d.gz| gunzip |load_cnosdb --batch-size 1000 --workers 100 --db-name benchmark001 --urls http://tskv:21001 > log/benchmark001.log &
nohup cat /data/cnosdb--iot-seed-123-scale-100-2023-15d.gz| gunzip |load_cnosdb --batch-size 100 --workers 100 --db-name benchmark002 --urls http://tskv:21001 > log/benchmark002.log &
nohup cat /data/cnosdb--iot-seed-123-scale-100-2023-15d.gz| gunzip |load_cnosdb --batch-size 15000 --workers 25 --db-name benchmark003 --urls http://tskv:21001 > log/benchmark003.log &

When I check the log, I find that the writing speeds varied greatly, as follow:

time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1694593550,460334.31,4.603544E+06,460334.31,58197.46,5.820000E+05,58197.46
1694593560,505418.28,9.657586E+06,482875.49,63901.78,1.221000E+06,61049.52
1694593570,426405.05,1.392190E+07,464051.33,53896.67,1.760000E+06,58665.15
1694593580,445295.87,1.837483E+07,459362.58,56300.34,2.323000E+06,58073.96
1694593590,447851.91,2.285339E+07,457060.46,56599.51,2.889000E+06,57779.07
^C
[root@xxx]# tail -f log/benchmark002.log 
time,per. metric/s,metric total,overall metric/s,per. row/s,row total,overall row/s
1694593550,1292059.40,1.293298E+07,1292059.40,163343.47,1.635000E+06,163343.47
1694593560,1507096.46,2.800396E+07,1399526.49,190499.79,3.540000E+06,176915.13
1694593570,1293654.22,4.093685E+07,1364253.65,163546.11,5.175000E+06,172461.05
1694593580,1400032.03,5.493751E+07,1373196.89,176995.74,6.945000E+06,173594.55
1694593590,1423748.82,6.917450E+07,1383305.58,180006.24,8.745000E+06,174876.68
1694593600,1411798.90,8.329355E+07,1388054.24,178486.59,1.053000E+07,175478.30
1694593610,1352479.28,9.681788E+07,1382972.79,171005.89,1.224000E+07,174839.47
1694593620,1470933.21,1.115301E+08,1393968.77,185963.17,1.410000E+07,176230.05
1694593630,1293557.09,1.244626E+08,1382815.45,163539.65,1.573500E+07,174820.45
1694593640,1281551.04,1.372783E+08,1372689.55,161997.64,1.735500E+07,173538.24
1694593650,1471194.48,1.519910E+08,1381644.43,185989.77,1.921500E+07,174670.18
^C
[root@xxx]# tail -f log/benchmark003.log 
1694593560,505418.28,9.657586E+06,482875.49,63901.78,1.221000E+06,61049.52
1694593570,426405.05,1.392190E+07,464051.33,53896.67,1.760000E+06,58665.15
1694593580,445295.87,1.837483E+07,459362.58,56300.34,2.323000E+06,58073.96
1694593590,447851.91,2.285339E+07,457060.46,56599.51,2.889000E+06,57779.07
1694593600,458105.56,2.743460E+07,457234.65,57898.07,3.468000E+06,57798.91
1694593610,440548.65,3.183998E+07,454851.02,55701.33,4.025000E+06,57499.26
1694593620,465247.21,3.649228E+07,456150.49,58802.12,4.613000E+06,57662.11
1694593630,404946.43,4.054174E+07,450461.19,51200.13,5.125000E+06,56944.12
1694593640,390041.60,4.444203E+07,444419.44,49301.57,5.618000E+06,56179.89
1694593650,462704.80,4.906916E+07,446081.77,58498.91,6.203000E+06,56390.72

I don't know why there is such a big gap, what is the reason?