Efficiently working with On-Behalf Of access tokens in an ASP.NET Core application

Question

Note: this is a follow-up of Reusing a Polly retrial policy for multiple Refit endpoints without explicitly managing the HttpClient

When making Refit work with Polly and an Azure AD-based authentication (On Behalf Of flow), I realized that acquiring an OBO token can be very slow (>400ms). The code for acquiring an OBO token based on the current logger in the user access token is shown below:

public async Task<string> GetAccessToken(CancellationToken token)
{
    var adSettings = _azureAdOptions.Value;
    string[] scopes = new string[] { "https://foo.test.com/access_as_user" };

    string? httpAccessToken = _httpContextAccessor.HttpContext?.Request?.Headers[HeaderNames.Authorization]
        .ToString()
        ?.Replace("Bearer ", "");
    if (httpAccessToken == null)
        throw new ArgumentNullException("Failed to generate access token (OBO flow)");

    string cacheKey = "OboToken_" + httpAccessToken;
    string oboToken = await _cache.GetOrAddAsync(cacheKey, async () =>
    {
        IConfidentialClientApplication cca = GetConfidentialClientApplication(adSettings);

        var assertion = new UserAssertion(httpAccessToken);
        var result = await cca.AcquireTokenOnBehalfOf(scopes, assertion).ExecuteAsync(token);
        return result.AccessToken;
    },
    new MemoryCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(TokenCacheExpirationInMinutes) });

    return oboToken;
}

private IConfidentialClientApplication GetConfidentialClientApplication(AzureAdAuthOptions adSettings)
{
    var certMetadata = _azureAdOptions.Value.ClientCertificates[0];
    string certPath = certMetadata.CertificateDiskPath;
    _logger.LogInformation($"GetAccessToken certificate path = {certPath}");

    string certPassword = certMetadata.CertificatePassword;

    var certificate = new X509Certificate2(certPath, certPassword);
    _logger.LogInformation($"GetAccessToken certificate = {certificate}");

    var cca = ConfidentialClientApplicationBuilder
        .Create(adSettings.ClientId)
        .WithTenantId(adSettings.TenantId)
        .WithCertificate(certificate)
        // .WithClientSecret(adSettings.ClientSecret)
        .Build();
    return cca;
}

This seems to work fine (not tested in a production environment though). however, I feel that I am reinventing the wheel here as I managing the OBO token caching myself.

Currently, this flow is used by Refit configuration:

private static IServiceCollection ConfigureResilience(this IServiceCollection services)
{
    services
        .AddRefitClient(typeof(IBarIntegration), (sp) =>
        {
            var accessTokenHelperService = sp.GetRequiredService<IAccessTokenHelperService>();
            return new RefitSettings
            {
                AuthorizationHeaderValueGetter = () => accessTokenHelperService.GetAccessToken(default)
            };
        })
        .ConfigureHttpClient((sp, client) =>
        {
            var BarSettings = sp.GetRequiredService<IOptions<BarApiSettings>>();
            string baseUrl = BarSettings.Value.BaseUrl;
            client.BaseAddress = new Uri(baseUrl);
        })
        .AddPolicyHandler(Policy<HttpResponseMessage>
            .Handle<HttpRequestException>()
            .OrResult(x => x.StatusCode is >= HttpStatusCode.InternalServerError or HttpStatusCode.RequestTimeout)
            .WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(TimeSpan.FromSeconds(1), RetryPolicyMaxCount)));

    return services;
}

Are there any caveats with the current implementation? I am only interested in possible security, performance or "reinventing-the-wheel" issues.