Get retry count inside Azure Durable Function activity

Question

I'm using an Azure Durable Function v4 (isolated) orchestrator that starts a couple of activities. There is a retry-policy that will retry a failed activity a certain amount of time. Inside the activity, I would like to get the number of retries that have been performed for that activity. Is this possible? If yes, how would I go about it?

This small class corresponds to my actual setup:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace Functions;

public class DemoDurableFunction
{
  [Function(nameof(StartOrchestrator))]
  public async Task<object> StartOrchestrator(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "start")] HttpRequestData req,
    [DurableClient] DurableTaskClient client)
  {
    var instanceId = await client.ScheduleNewOrchestrationInstanceAsync(nameof(RunOrchestrator), input: "");
    var response = client.CreateCheckStatusResponse(req, instanceId);

    return response;
  }

  [Function(nameof(RunOrchestrator))]
  public async Task RunOrchestrator([OrchestrationTrigger] TaskOrchestrationContext context)
  {
    var replaySafeLogger = context.CreateReplaySafeLogger(nameof(RunOrchestrator));
    var options = TaskOptions.FromRetryPolicy(new RetryPolicy(maxNumberOfAttempts: 2, firstRetryInterval: TimeSpan.FromSeconds(1)));

    var tasks = new List<Task<string>>
    {
      context.CallActivityAsync<string>(nameof(RunActivity), input: "Alice", options: options),
      context.CallActivityAsync<string>(nameof(RunActivity), input: "Bob", options: options),
      context.CallActivityAsync<string>(nameof(RunActivity), input: "Chuck", options: options)
    };

    try
    {
      await Task.WhenAll(tasks);
    }
    catch (Exception e)
    {
      replaySafeLogger.LogError("Something horrible happened: {Message}", e.Message);
    }

    foreach (var task in tasks)
    {
      if (task.IsFaulted)
      {
        replaySafeLogger.LogInformation(">>> Activity failed: {Message}", task.Exception?.Message);
      }
      else
      {
        replaySafeLogger.LogInformation(">>> Activity result: {Result}", task.Result);
      }
    }
  }

  [Function(nameof(RunActivity))]
  public async Task<string> RunActivity([ActivityTrigger] string name, FunctionContext executionContext)
  {
    // TODO: log the number of retries that have been performed for this specific activity (not any activity of the 3).

    await Task.Delay(500);

    if (name.StartsWith("\"C"))
    {
      throw new Exception($"No C-names allowed {executionContext.RetryContext}");
    }

    return $"DONE: {name}";
  }
}

where the Program is simply:

using Microsoft.Extensions.Hosting;

var host = new HostBuilder()
  .ConfigureFunctionsWorkerDefaults()
  .ConfigureLogging(_ => {})
  .Build();

host.Run();

The FunctionContext inside the activity has a RetryContext property, but that is always null.

Everything works as expected: 2 tasks are successfully processed, and the "Chuck" fails and retried a couple of times.

Answer 1

Hope you are doing gud

So if you need a approach without using static variables, you can try the Durable Task Framework's extensibility points. Specifically, you can create a custom implementation of the IOrchestrationService interface to capture the retry count for each activity. lemme put it down for you

public class RetryTrackingOrchestrationService : IOrchestrationService
{
    private readonly IOrchestrationService innerService;

    public RetryTrackingOrchestrationService(IOrchestrationService innerService)
    {
        this.innerService = innerService;
    }

    public async Task<TaskOrchestrationWorkItem> LockNextTaskOrchestrationWorkItemAsync(
        TimeSpan receiveTimeout,
        CancellationToken cancellationToken)
    {
        var workItem = await innerService.LockNextTaskOrchestrationWorkItemAsync(receiveTimeout, cancellationToken);

        // Attach the retry count information to the orchestration context
        if (workItem != null && workItem.OrchestrationRuntimeState != null)
        {
            workItem.OrchestrationRuntimeState.CustomStatus["RetryCount"] = 
                workItem.OrchestrationRuntimeState.CurrentRetryAttempt;
        }

        return workItem;
    }

    // Implement other IOrchestrationService methods by delegating to innerService
    // ...
}

You would then use this custom orchestration service when setting up your Durable Task Client.

var durableTaskClient = new DurableTaskClient(
    new RetryTrackingOrchestrationService(new TaskHubWorkerSettings(), new DurableHttpClientFactory()));

With this setup, the RetryCount information will be available in the CustomStatus of the OrchestrationRuntimeState. You can access it in your activity function

[Function(nameof(RunActivity))]
public async Task<string> RunActivity([ActivityTrigger] string name, FunctionContext executionContext)
{
    var retryCount = executionContext.GetInput<OrchestrationRuntimeStatus>().CustomStatus["RetryCount"];
    
    // Use retryCount as needed

    await Task.Delay(500);

    if (name.StartsWith("C"))
    {
        throw new Exception($"No C-names allowed, Retry Count: {retryCount}");
    }

    return $"Hi Man it is DONE: {name}";
}

Hope this helps you mate thanks much.