Singleton Lock for Azure Webjob's TimerTrigger accross multiple regions

Question

I have a timer-triggered WebJob deployed across multiple regions and it is getting triggering concurrently from all regions on given scheduled time, how do I make sure that only one instance of the job runs at a time? I tried applying Singleton attribute and "is_singleton": true but still it is triggering from all regions.

Is there any other way to achieve this? This link says that Singleton attribute no longer works for this purpose and also I don't see any lock file created in the azure blob storage. If it's true how do we implement this to ensure only one region is triggered from multiple regions? Or if there is any other inbuilt way of achieving this with WebJob SDK that would be really helpful to me.

Program.cs:

var builder = new HostBuilder();
builder
    .ConfigureWebJobs((context, b) =>
    {
        b.AddAzureStorageCoreServices();
    });
var host = builder.Build();
using (host)
{
    var jobHost = host.Services.GetService(typeof(IJobHost)) as JobHost;
    await host.StartAsync().ConfigureAwait(false);
    await jobHost.CallAsync("Run").ConfigureAwait(false);
    await host.StopAsync().ConfigureAwait(false);
}

Function.cs:

[Singleton]
[NoAutomaticTrigger]
public async Task Run()
{
}

settings.job:

{
  "schedule": "0 */5 * * * *",
  "is_singleton": true
}

NuGet package:

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions" Version="4.0.1" />

Answer 1

So it sounds like you want each region to run, but just not at the same time.

One simple solution would be to stagger them so they don't all run at the same time but you give each one a slightly different cron time, such as 10m apart from each other.

But supposing you couldn't predict how long they will run or you essentially want them to run much faster then the minimum 10m apart, and supposing they could all access a database, such a Azure Cosmos Mongo, then you could add a table and some simple logic which would essentially model a "locking" pattern.

In mongodb you can use the findOneAndUpdate function to do an atomic update on a well known document which will allow only one process to "lock" a document at a time.

The database in this case contains a collection (aka table) with a single document (aka row) that looks like this:

interface ILock {
  state: 'UNLOCKED' | 'LOCKED';
  lockedBy: null | string;
  lockedAt: null | Date;
}

psuedo code

while (true)
{
  // todo: add some kind of timeout here so it doesn't run forever.

  // `findOneAndUpdate` is atomic, if multiple processes
  // attempt to do this modification simultaneously
  // only one will succeed, the others will get an `undefined`
  // result to indicate the document was not found.
  var lock = await this.db.locks.findOneAndUpdate(
    { state: 'UNLOCKED' },
    {
      $set: {
        state: 'LOCKED',
        lockedBy: this.jobId,
        lockedAt: new Date()
      }
    }
  )

  if (lock) {
    try {
      // you have the lock, do your thing...
      await DoWork();
      // you are done, exit the loop.
      return;
    } finally {
      // don't forget to unlock!
      await this.db.locks.findOneAndUpdate(
        { state: 'LOCKED' },
        {
          $set: {
            state: 'UNLOCKED',
          }
        }
      )
    }
  } else {
    // you are not the one neo, take the blue pill...
    await sleep(3000)
  }
  
}