I am using a Durable Functions Orchestration to perform 3 tasks (activities). One of the tasks copies blobs from one storage account to another. I instantiate my source and target BlobServiceClient objects in the following way:
TokenCredential creds = new DefaultAzureCredential();
_sourceServiceClient = new BlobServiceClient(new Uri($"https://{_sourceStorageAcctName}.blob.core.windows.net"), creds);
_targetServiceClient = new BlobServiceClient(_targetStorageAcctConnStr);
// default user delegation key expiration to 1 hour
_userDelegationKey = _sourceServiceClient.GetUserDelegationKey(null, DateTimeOffset.UtcNow.AddHours(1));
The job should never take more than 10 minutes to complete given the amount of blobs to copy, hence why I set an expiration date of 1 hour for the User Delegation Key. Once I have the list of blobs to copy, I loop through it and generate a SAS with Read permissions and expiration of 5 minutes, and then call:
targetBlobClient.StartCopyFromUri(sourceBlobSASURI)
This is repeated for every blob in the list. My Azure account and the Function's managed identity have been added to the "Storage Blob Data Contributor" role in the source storage account. Testing locally in Visual Studio works fine. When I deploy the function to Azure and run it manually it works fine.
I then schedule the function to run on a daily basis, but now the function fails to authenticate to the storage account with the message below:
Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:cf03b2fa-001e-0028-7aec-6e22a1000000
Time:2024-03-05T11:00:17.9880230Z
Status: 403 (Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.)
ErrorCode: CannotVerifyCopySource
Content:
<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>CannotVerifyCopySource</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:cf03b2fa-001e-0028-7aec-6e22a1000000
Time:2024-03-05T11:00:17.9880230Z</Message></Error>
Headers:
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: cf03b2fa-001e-0028-7aec-6e22a1000000
x-ms-client-request-id: d6e94e13-cb3e-44e8-a874-df40a9a9faa6
x-ms-version: 2023-08-03
x-ms-error-code: CannotVerifyCopySource
Date: Tue, 05 Mar 2024 11:00:17 GMT
Content-Length: 322
Content-Type: application/xml
\r
\r
at Azure.Storage.Blobs.BlobRestClient.StartCopyFromURL(String copySource, Nullable`1 timeout, IDictionary`2 metadata, Nullable`1 tier, Nullable`1 rehydratePriority, Nullable`1 sourceIfModifiedSince, Nullable`1 sourceIfUnmodifiedSince, String sourceIfMatch, String sourceIfNoneMatch, String sourceIfTags, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, String leaseId, String blobTagsString, Nullable`1 sealBlob, Nullable`1 immutabilityPolicyExpiry, Nullable`1 immutabilityPolicyMode, Nullable`1 legalHold, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUriInternal(Uri source, IDictionary`2 metadata, IDictionary`2 tags, Nullable`1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable`1 rehydratePriority, Nullable`1 sealBlob, BlobImmutabilityPolicy destinationImmutabilityPolicy, Nullable`1 legalHold, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartCopyFromUri(Uri source, IDictionary`2 metadata, Nullable`1 accessTier, BlobRequestConditions sourceConditions, BlobRequestConditions destinationConditions, Nullable`1 rehydratePriority, CancellationToken cancellationToken)
...
Even when running it manually after seeing the failure, it fails with the same message. However, if I restart the Function App and then run it once more, it works fine. I am struggling to figure out how come just restarting the Function App fixes the issue? It's not a permament fix however, as eventually the scheduled execution will hit the error again, typically within 24 hours.
I am using the "in-process" model and wonder if this has anything to do with it. Any insights into this behavior is much appreciated.
I figured out the issue here. The class containing my functions was declared as static, including the property that instantiated my _sourceServiceClient and _targetServiceClient objects. Hence, everything works fine on the first execution, but on subsequent executions, it reuses the static objects with the values assigned during the first execution. So after 1 hour, the user delegation key expires causing the error I posted.
By removing the static property of the class and its properties, a new user delegation key will be generated with every execution, solving the problem.
This also explains why re-starting the function app causes everything to work again, as the class is instantiated newly after 1st execution.