Mongodb - updating a collection with >100,000 documents in migration

Question

Following migration code seems to be getting stuck while updating this huge collection.

@ChangeSet(order = "010", id = "add_tenant_to_product", author = "pn")
fun addTenantToProduct(mongoTemplate: MongoTemplate) {
    log.info("Adding tenants to products")
    val query = Query()
    val update = Update()
    update.set("tenants", arrayOf("shared"))

    mongoTemplate.updateMulti(query, update, Product::class.java)
}

@ChangeSet(order = "011", id = "add_tenant_to_product_version", author = "pn")
fun addTenantToProductVersion(mongoTemplate: MongoTemplate) {
    log.info("Adding tenants to product-version")
    val query = Query()
    val update = Update()
    ...

The last few lines of the application log indicate the changeset getting invoked but subsequent one not getting invoked.

2021-05-11 08:00:29.385  INFO [product-service,,,] 1 --- [           main] c.a.a.mongock.changelog.Migration        : Adding tenants to products
2021-05-11 08:00:29.576  INFO [product-service,,,] 1 --- [           main] org.mongodb.driver.connection            : Opened connection

The Spring actuator health probe fails because of this and the container keeps on restarting. Is there a better API to use which will be faster?

Note:

1. version used: 'com.github.cloudyrock:mongock:1.12'
2. below is the configuration

@Profile("!tenant")
@Configuration
class MongockConfig : AbstractMongoConfiguration() {

    @Value("\${spring.data.mongodb.uri}")
    private lateinit var uri: String

    @Value("\${spring.data.mongodb.database}")
    private lateinit var database: String

    override fun mongoClient(): MongoClient {
        return MongoClient(MongoClientURI(uri))
    }

    override fun getDatabaseName(): String {
        return database
    }

    @Bean
    fun mongock(): Mongock {
        return MongockBuilder(mongoClient(), databaseName, "com.somepackage.mongock.changelog")
            .setMongoTemplate(mongoTemplate())
            .setLockQuickConfig()
            .build()
    }
}

Here are the logs

---

2021-05-10 10:59:33.406  INFO [product-service,,,] 1 --- [           main] org.mongodb.driver.connection            : Opened connection [connectionId{localValue:16, serverValue:26359638}] to dummy-clusterr-39649.servers.mongodirector.com:27017
2021-05-10 10:59:33.525  INFO [product-service,,,] 1 --- [           main] c.github.cloudyrock.mongock.LockChecker  : Mongbee trying to acquire the lock
2021-05-10 10:59:33.574  INFO [product-service,,,] 1 --- [           main] c.github.cloudyrock.mongock.LockChecker  : Lock is taken by other process until: Mon May 10 11:01:03 GMT 2021
2021-05-10 10:59:33.574  INFO [product-service,,,] 1 --- [           main] c.github.cloudyrock.mongock.LockChecker  : Mongock is going to sleep to wait for the lock:  90265 ms(1 minutes)
2021-05-10 11:01:03.839  INFO [product-service,,,] 1 --- [           main] c.github.cloudyrock.mongock.LockChecker  : Mongbee trying to acquire the lock
2021-05-10 11:01:03.844  INFO [product-service,,,] 1 --- [           main] c.github.cloudyrock.mongock.LockChecker  : Mongbee acquired the lock until: Mon May 10 11:04:03 GMT 2021
2021-05-10 11:01:03.844  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : Mongock starting the data migration sequence..
2021-05-10 11:01:03.982  INFO [product-service,,,] 1 --- [           main] org.reflections.Reflections              : Reflections took 60 ms to scan 1 urls, producing 2 keys and 3 values 
2021-05-10 11:01:04.038  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@7abd0e71 pass over
2021-05-10 11:01:04.046  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@6635f36 pass over
2021-05-10 11:01:04.047  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@61352a80 pass over
2021-05-10 11:01:04.052  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@7576ed14 pass over
2021-05-10 11:01:04.054  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@b0302179 pass over
2021-05-10 11:01:04.068  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@e6caade0 pass over
2021-05-10 11:01:04.071  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@2688129a pass over
2021-05-10 11:01:04.073  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@e46af677 pass over
2021-05-10 11:01:04.075  INFO [product-service,,,] 1 --- [           main] com.github.cloudyrock.mongock.Mongock    : com.github.cloudyrock.mongock.ChangeEntry@d788b92f pass over
2021-05-10 11:01:04.077  INFO [product-service,,,] 1 --- [           main] c.a.a.mongock.changelog.Migration        : Adding organisations to questions
2021-05-10 11:01:04.158  INFO [product-service,,,] 1 --- [           main] org.mongodb.driver.connection            : Opened connection [connectionId{localValue:17, serverValue:26359725}] to dummy-clusterr-39649.servers.mongodirector.com:27017

Answer 1

can you provide a bit more information, such as mongock version you are using, log trace(what you show is not enough fo diagnostic), mongock configuration, etc?

On the other hand, you should use MongockTemplate instead MongoTemplate directly. It's the same API, you wont miss anything. Actually using a MongoTemplate directly should throw an exception.

For more information, please check Mongock's documenation