SolrCloud delete document only delete on leader replica not the rest replica

Question

I setup solrcloud using solr helm and try to delete one/all documents from a collection. But it turns out that only leader replica deleted them as expected. All others remain the same

Solrcloud version: 9.3 Collection config: name: IBpets, numShards: 1, replicationFactor: 3 perReplicaState: true

Below is the leader collection of the collection(solr2-solrcloud-1) solrcloud graph

Below is the way I did the delete: solrcloud k8s svc and portfoward command

solrcloud admin GUI

Only solr2-solrcloud-1 got everything deleted. All other two replica still have all documents

cpu and memory is good

SolrCloud Setup: I have 1635 collections in total, among which only 135 collections have data and all others are empty.

3 solr pods are installed on 3 separate AWS machine: m6a.2xlarge 3 zookeeper pods are intalled on 3 separate AWS machine: m6a.large

solrOptions: javaMemory: "-Xms12g -Xmx12g" javaOpts: "-Djute.maxbuffer=50000000"

zookeeper: jvmFlags: "-Djute.maxbuffer=50000000 -Xms4096m -Xmx4096m -verbose:gc -Xloggc:$ZOO_LOG_DIR/zookeeper_gc.log"

I also tried to delete single document both through admin GUI and curl command. But neither works. Still leader deletes everthing. curl -X POST http://localhost:58983/solr/IBpets/update?softCommit=true -H 'Content-Type: application/json' -d ' { "delete": { "query": "ib_uuid:(\"0eadfae078f744e6b30bf2448e08f672\")" } }