How to whitelist only limited IP to access blob storage

Question

I am trying to upload an zip file from remote system to blob storage using sas(shared access signature). I have enable "Enabled from selected virtual networks and IP addresses" in azure blob networking

ip

but I'm still getting error below error:

INFO: Scanning...
2024-03-06T05:07:06.9515419Z INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2024-03-06T05:07:07.0092305Z 
2024-03-06T05:07:07.0093607Z Job 9cab7bd7-2a3d-6f48-6e76-a24a64f1fb02 has started
2024-03-06T05:07:07.0094274Z Log file is located at: /home/vsts/.azcopy/9cab7bd7-2a3d-6f48-6e76-a24a64f1fb02.log
2024-03-06T05:07:07.0094668Z 
2024-03-06T05:07:07.2407944Z INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission PUT https://blobmarch24.blob.core.windows.net/blobstorage/473.zip
2024-03-06T05:07:07.2409377Z --------------------------------------------------------------------------------
2024-03-06T05:07:07.2409755Z RESPONSE 403: 403 This request is not authorized to perform this operation.
2024-03-06T05:07:07.2409966Z ERROR CODE: AuthorizationFailure
2024-03-06T05:07:07.2410395Z --------------------------------------------------------------------------------
2024-03-06T05:07:07.2414251Z <?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
2024-03-06T05:07:07.2414972Z RequestId:7aec8617-401e-0058-2184-6f10c0000000
2024-03-06T05:07:07.2415294Z Time:2024-03-06T05:07:07.2468643Z</Message></Error>
2024-03-06T05:07:07.2415734Z --------------------------------------------------------------------------------
2024-03-06T05:07:07.2415863Z 
2024-03-06T05:07:07.2434661Z panic: close of nil channel
2024-03-06T05:07:07.2435546Z 
2024-03-06T05:07:07.2436101Z goroutine 92 [running]:
2024-03-06T05:07:07.2437032Z github.com/Azure/azure-storage-azcopy/v10/ste.(*jobMgr).reportJobPartDoneHandler(0xc000069400)
2024-03-06T05:07:07.2437735Z    /home/vsts/work/1/s/ste/mgr-JobMgr.go:718 +0x30c
2024-03-06T05:07:07.2438530Z created by github.com/Azure/azure-storage-azcopy/v10/ste.NewJobMgr
2024-03-06T05:07:07.2498434Z    /home/vsts/work/1/s/ste/mgr-JobMgr.go:203 +0xcb4
2024-03-06T05:07:07.2498613Z 
2024-03-06T05:07:07.2595374Z ##[error]Bash exited with code '2'.

what I have done is I created service principal "blob data owner" from azure cloudshell

az ad sp create-for-rbac --role="Storage Blob Data Owner" --role --scopes="/subscriptions/<subscriptionID>" --name "blob owner"

below code to login into pipeline agents using credentials

az login --service-principal -t <tenantID> -u <clientID> -p <clientSecret>

rest of the command that ran in pipeline

#getting machine IP
machine_ip=$(curl ifconfig.me)
#getting date and time in format
expiry_time=$(date -u -d '+30 minutes' +%Y-%m-%dT%H:%MZ)

#creating a sas token
sas_token=$(az storage container generate-sas --account-name <blobname> --name <containername> --permissions dlwr --expiry $expiry_time --ip $machine_ip --output tsv )

#addding the IP to allowed IP in blob storage
az storage account network-rule add -g pipelineTest --account-name blobmarch24 --ip-address $machine_ip

lastly to upload file

azcopy cp $(Build.BuildId).zip "https://blobmarch24.blob.core.windows.net/blobstorage/?$sas_token"

it failed and gave error as shared earlier

Then I did the same for my azure cloudshell it was successfully uploaded to blob...

Answer 1

As per the error message, you are using linux agent for the pipeline, I can reproduce the same error with your script.

I fixed the script as below:

1. add accountkey parameter for sas_token generate.
2. add delay after ip added to storage account.

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      az login --service-principal -t $(tenantid) -u $(sp) -p $(cert)
      #getting machine IP
      machine_ip=$(curl ifconfig.me)
      #getting date and time in format
      expiry_time=$(date -u -d '+30 minutes' +%Y-%m-%dT%H:%MZ)

      echo $expiry_time

      #creating a sas token
      sas_token=$(az storage container generate-sas --account-name $(sa) --name $(container) --permissions dlrw --expiry $expiry_time --ip $machine_ip --account-key $(key) --output tsv )

      #check token value, can remove for security.
      echo $sas_token

      #addding the IP to allowed IP in blob storage
      az storage account network-rule add -g $(rg) --account-name $(sa) --ip-address $machine_ip 

      sleep 30s   # add delay here

      azcopy cp $(Build.BuildId).zip "https://yourstorageaccountname.blob.core.windows.net/containername/?$sas_token"

In addition, the service principal should grant contributor and storage Blob data Contributor on storage account.

It succeeds on my side:

If you still have the issue, please also check:

1. Make sure your storage account is NOT in same region with your storage account. If it's same region, for Microsoft-hosted agent, it will use internal network to contact storage account, setting Network whitelist won't work. It's a known limitation from storage account side. You need to create the storage account in a different region.

2. Make sure the agent ip is added into the network whitelist.