Unable to create api manager policy for new MuleSoft api in anypoint platform using the shell script and api endpoint

Question

I have a shell script, in which for every API manager policy I have created a function, which is responsible for creating API manager policy for a mulesoft API.

Step 1. policy attributes is defined in deploy-config-$env.properties:

    ex: 
       policy=client-id-enforcement
       *#attributes*
       policy.client-id-enforcement.assetversion=1.2.0

Step 2. The shell- client-id-enforcement() function reads the attributes mentioned in deploy-config-$env.properties, if no attributes are given, the function is initialized with the default attributes of the function.

Step 3. the default values of the attributes with the values fetched from deploy-config-$env.properties are updated

Step 4. a $policy-json file is created. Example:

       {
        "configurationData": {
         "property1": "value1",
         "property2": "value2"
         },
         "pointcutData": null,
         "policyTemplateId": "12345",
         "groupId": "uidhud65gfu9guyfwydt",
         "assetId": "client-id-enforcement",
         "assetversion": "1.2.0"
        }

Step 5. this $policy.json file is passed to the api endpoint for creating the policy: example:

        policy_result="$(curl --silent -x POST "https://anypoint.mulesoft.com/apimnager/api/v1/organizations/$orgId/envirenments/$envId/apis/$apiid/policies -H "Authorization:Bearer $accesss_token" -H 'content Type:application/json' -d @$policy.json)" 

the CICD pipeline used is gitlab, and the config file is .gitlab-ci.yml

Everything was working fine, but one day suddenly the api endpoint used for creating the policy started throwing below error:

policy_result: {"name": "badRequestError","message": "the policy to be created is missing at least one of the following properties related to the policy template: 'groupId, 'assetid','assetversion'."}

But I checked, there is no problem in policy template grouped,assetid and assetversion , all the values are populated correctly in the policy.json file

After digging in more , I got to know that policy.json file when it is created,it has only carriage return (CR) appended to it and I get the above mentioned error while creating the policy BUT when I add manully a newline and remove it save it and when I open the same file in notepad ++, the file contains the carriage return (CR) and newline feed(LF) as given below:

This file with CR and LF is in correct format and this same file is used and the policy is created from the end point.

This happening all of a sudden without doing any changes to the cicd pipeline.

I tried adding the CR and LF using sed command to all the lines of the policy.json file, still the file doesn't contain the LF, only CR is appended.

sed -i $'s/$/\r/' policy.json | sed -i $'s/$/\r/n/' policy.json

I tried other different way of adding the CR and LF using awk ,sed and tr, still, LF is not happening in the policy.json and the API endpoint is not able to create the policy.

is it possible that the policy.json file can get corrupted everytime, the file is generated?

can I get any help in its resolution?, my development work is getting affected due to this.

Answer 1

This could be related to your local git config setting that controls how line endings are managed. The setting is called "core.autocrlf". To view the current Git settings for this parameter,

git config --global core.autocrlf

The following is the behavior for various possible settings(true/false/input)

**core.autocrlf=false** - No conversion happens upon merge
**core.autocrlf=true** Upon merge, all CRLF line endings will be converted to LF. When the file is checkedout, it ensures the line endings(LF) are replaced with original CRLF. Recommended for windows
**core.autocrlf=input** Upon merge, all CRLF line endings will be converted to LF. But won't be converted back when the same file is checked out. Recommended for Linux/Unix/OS X

Edit: If I understand correctly, your core problem is the line endings of yaml file gets converted to Carriage Return(CR) no matter what you specify/convert manually. This suggests that the default behavior to auto convert line endings based on source OS has been overwritten. FYI, CR only line endings are only used by Legacy MacOS(MacOS 9 and earlier). Can you check if someone had checked in .gitattributes file in the root of your repository? The problematic behavior is expected if it was set the following way,

* text eol=cr

If this is the setting you see, you should change it to following,

* text=auto

It's possible that someone had set this specifically for Legacy MacOS