Getting azure blob metadata key values using azure data factory

Question

Is there a easy way to access azure blob metadata key values through data factory or any other easier solution when loading the data in to azure dataware house on fabric.

I tried using getmetadata component and looks like that is not doing the job

Answer 1

To get the metadata of the blob you need to use the Rest API Get Blob Metadata It will return you all the User-defined Metadata.

• To execute it you need to use web activity and setup connection as below.

base url : https://storageacc name.blob.core.windows.net/container/blob?comp=metadata&<SAS Token>
Authentication : Anonymous

• Select the newly created connection in web activity and method as GET.

Output:

In Response Headers you will get X-ms-meta-<key> : <value>

Answer 2

Just use a Web activity with the URL being your blob address:

Property	Value
URL	https://[your-account-name].blob.core.windows.net/[your-container-name]/[your-blob-name]
Method	GET
Authentication	System Assigned Managed Identity
Resource	https://storage.azure.com
Headers	x-ms-version : 2020-06-12

An example of how to configure it:

This will return the content in an ADFWebActivityResponseHeaders object, in the form of:

As you can see, all metadata keys are prefixed with x-ms-meta-. You can then access the metadata keys, eg in a Set Variable activity:

@activity('Web1').output.ADFWebActivityResponseHeaders['x-ms-meta-mymetadatakey']

I also did a split by double-quote (") and then a Filter activity so I could loop through them with a For Each loop.

Filter items: @split(string(activity('Web1').output.ADFWebActivityResponseHeaders),'"')

Filter condition: @startswith(item(), 'x-ms-meta-')