Auto incrementing an indexed field in mongodb when there are multiple concurrent requests

Question

I am trying to auto increment an indexed field in mongodb whenever there is an insertion happens, I read many posts on SO this and mongoose-auto-increment, but I am not getting how are they working Because consider below scenario

Suppose I want to auto increment a field counter in my collection and currently the first record already exist whose counter value is 1, now suppose there are three concurrent inserts happens in the database now as counter value is 1 so all of them must be trying to set counter 2. But as we know know among these three whoever will get the first lock will successfully set its counter as 2, but what about other two operations because now when they will acquire lock they will also try to set counter value as 2 but as 2 is already taken so I guess mongoose will give error duplicate key error.

Can anyone please tell me how does above two posts solves the concurreny problem for auto-incrementing an indexed field in mongodb.

I know I am missing some conecpt but what ??

Thanks.

Answer 1

I encounter the same problem so I ended up building my own increment handling concurrency and it was quite easy! Bottom line, the fast answer, I used a try catch loop while I save the document to catch the duplicated key error on my incremented field. Here is how I emplemented this on mongoose and in my controller/service/model architecture:

First, I need to store the auto increment, it won't be a big collection since I will never have more than a dozen concerned collections in a project, so I don't even need special indexes or whatever:

counter.model.js

// requires modules blabla...

// The mongoose schema for the counter collection
const CounterSchema = new Schema({
  // entity describes the concerned collection.field
  entity: {
    type: {
      collection: { type: String, required: true },
      field: { type: String, required: true }
    }, required: true, unique: true
  },
  // the actual counter for the collection.field
  counter: { type: Number, required: true }
});

// The mongoose-based function to query the counter
async function nextCount(collection, field){
  let entityCounter = await CounterModel.findOne({
    entity: { collection, field })
    
  let counter = entityCounter.counter + 1
  entityCounter.counter = counter;
  await entityCounter.save();
  return counter
}

// mongoose boiler plate  
CounterSchema.statics.nextCount = nextCount;
const CounterModel = mongoose.model("counter", Counterschema)

module.exports = CounterModel

Then, I made a service to use my counter model. We also use the service to format the auto-increment as needed. For example, accountancy wants that all client number starts with "411" ans adds a 5 figures id, so client number 1 actually will be 41100001

counter.service.js

// requires modules blabla ....

class CounterService {
  constructor(){}

  static nextCount = async(collection, field, prefix, len){
    // Gets next counter from db
    const counter = await CounterModel.nextCount(collection, field)

    // Formats the counter as requested in params
    let counterString = `${counter}`;    
    while (counterString.length < len) {
      counterString = `0${counterString}`;}

    return `${prefix}${counterString}`;
  }
}
module.exports = CounterService

Then here is where we handle the concurrency: in the client model (I won't put here all the client model file but only what need for the explanation). Let's assume we have the client collection with the "num" field that needs the auto increment as described before:

client.model.js

// ...
const ClientSchema = new Schema({
  firstName: ...
  lastName: ...
  num: { type: String, required: true, unique: true }
})

async function addClient(clientToAdd){
  
  let newClient;

  let genuineCounter = false
  while(!genuineCounter){
    try{
      // Gets next increment from counter service
      clientToAdd.num = await CounterService.nextcount("client","num","411",5)
      
      newClient = new ClientModel(clientToAdd)
      await newClient.save();

      // if code reaches this point, no concurrency problem we end the loop
      genuineCounter = true
    } catch(error) {
      // If num is duplicated, an error is catched
      // we must ensure that the duplicated error comes from num field:
      // 11000 is the mongoDB returned error for duplicate unique index
      // and we check the duplicated field (could be other unique field!)
      if (error instanceof MongoError 
          && error.code === 11000
          && Object.keys(error.keyValue)[0] === "num")
         genuineCounter = false
      // For any other duplicated field or error we throw the error
      else throw error 
      }
    }
  
  return newClient 
  }
}

And here we go ! If two users query the counter at the same time, second one will keep querying the counter until the key is not a duplicate.

A small bonus to test it: creates the small moodule to easily fake delay where you want:

file delay.helper.js

const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
exports.delay = delay;

// use anywhere after module import with:
// await delay(5000)

Then import this module into the counter module and fake some delay between counter query and counter save:

counter.model.js

// previous described file and nextCount function with the use of delay()
const delay = require("./delay.helper")

async function nextCount(collection, field) {
  let entityCounter = await CounterModel.findOne(...)
   await delay(5000)
   ...
   await entityCounter.save()
}

Then, from your front-end project, or your api end point, have two identical tabs and send 2 queries in a row:

• Let's say the actual counter in db is 12
• Query A reads counter in db = 12
• Query A waits for 5 seconds
• Query B reads counter in db = still 12
• Query A increments and stores new client with num = 411000013; stores counter = 13
• Query B increments and tries to store new client, 411000013 already exists, error catched and tries again
• Query B reads counter in db = 13
• Query B waits for 5 seconds, then increments and store new client with num = 411000014 and stores also new counter = 14