DEVHIDE
Login Or Sign up

Unable to read YAML with jinja2 variables in it with ruamel

441 Views Asked by At

I am trying to parse a file from the helm bundle which contains lots of jinja variables in it. When I try to read the file using ruamel.yaml python library it throws following exception:

----- Python Traceback -----
File "/Users/bhupesh.gupta/Projects/node-python-poc/yaml-read-file-script.py", line 16, in <module>
  data = list(yaml.load_all(input))
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/main.py", line 451, in load_all
  for d in self.load_all(fp):
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/main.py", line 461, in load_all
  yield constructor.get_data()
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 115, in get_data
  return self.construct_document(self.composer.get_node())
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 66, in get_node
  return self.compose_document()
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 99, in compose_document
  node = self.compose_node(None, None)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
  node = self.compose_mapping_node(anchor)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
  item_value = self.compose_node(node, item_key)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
  node = self.compose_mapping_node(anchor)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 223, in compose_mapping_node
  item_value = self.compose_node(node, item_key)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 143, in compose_node
  node = self.compose_mapping_node(anchor)
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/composer.py", line 216, in compose_mapping_node
  while not self.parser.check_event(MappingEndEvent):
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 146, in check_event
  self.current_event = self.state()
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 597, in parse_block_mapping_key
  if self.scanner.check_token(KeyToken):
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/scanner.py", line 1794, in check_token
  while self.need_more_tokens():
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/scanner.py", line 211, in need_more_tokens
  self.stale_possible_simple_keys()
File "/opt/homebrew/lib/python3.9/site-packages/ruamel/yaml/scanner.py", line 360, in stale_possible_simple_keys
  raise ScannerError(
ruamel.yaml.scanner.ScannerError: while scanning a simple key
in "<unicode string>", line 20, column 1:
  <<{ toYaml . | indent 4 }}
  ^ (line: 20)
could not find expected ':'
in "<unicode string>", line 21, column 2:
   #<{- end }}

Following is the file which I am trying to parse:

{{- $profileFilePath := printf "%s/%s%s" "dep_configs" .Values.global.networkFunction.profile ".yaml" -}}
{{- $isFileAvail := .Files.Glob $profileFilePath }}
{{- $nameprefixStr :=  . }}
{{- if $isFileAvail }}
{{- $profileValues := $.Files.Get $profileFilePath | fromYaml }}
{{- if not (empty .Values.global.k8sPlatform.namePrefix) }}
{{- $nameprefixStr =  printf "%s-enodeb-%s-%s-%s" (lower $.Values.global.k8sPlatform.namePrefix) (lower $.Values.global.env.VRAN_ENB_ID)  (lower $profileValues.networkFunction.nfType) (lower $.Values.global.env.VRAN_NF_ID) -}}
{{- else }}
{{- $nameprefixStr =  printf "enodeb-%s-%s-%s" (lower $.Values.global.env.VRAN_ENB_ID)  (lower $profileValues.networkFunction.nfType) (lower $.Values.global.env.VRAN_NF_ID) -}}
{{- end }}
---
apiVersion: v1
kind: Pod
metadata:
  name: {{ $nameprefixStr }}-pod
  namespace: {{ $.Release.Namespace }}
  labels:
    name: {{ $nameprefixStr }}-pod
{{- with $.Values.global.k8sPlatform.labels }}
{{ toYaml . | indent 4 }}
{{- end }}
  annotations:
    {{- /* ##### fill the annotation for K8S looping through nw list */}}
    {{- $k8slistString := "" }}
    {{- $releaseName := $.Release.Name }}
    {{- range $.Values.cnfNetworks.cnfcNwList }}
       {{- if or ( eq .cniPluginType "sriov")  (eq .cniPluginType "sriov-dpdk") }}
         {{- if not (empty $k8slistString) }}
          {{- $k8slistString = printf "%s," $k8slistString }}
         {{- end }}
         {{- $intfAnnotation := printf "%s-%s@%s" $releaseName .cnfcNwType .cnfcNwType }}
         {{- $k8slistString = printf "%s%s" $k8slistString $intfAnnotation }}
       {{- end }}
     {{- end }}
     {{- if not (empty $k8slistString) }}
     k8s.v1.cni.cncf.io/networks: {{ $k8slistString }}
     {{- end }}
     {{- /* ##### START fill the annotation for ROBIN case looping through nw list */}}
     {{- $robinvarlistglobal := list  }}
     {{- $varlisttemp := list  }}
     {{- $dictvar := dict }}
     {{- range $.Values.cnfNetworks.cnfcNwList }}
        {{- if eq .cniPluginType "robin" }}
         {{- if not (eq .cnfcNwType "fec") }}

            {{- if (ne .ippool "") }}

            {{- $dictlocal := dict "interface_name" .cnfcNwType "ippool" .ippool }}
            {{- if not (empty .ipam) }}
              {{- if not (empty .ipam.ip) }}
              {{- $_ := set $dictlocal "static_ips" .ipam.ip }}
              {{- end }}
              {{- if not (empty .ipam.mtu) }}
              {{- $_ := set $dictlocal "mtu" .ipam.mtu }}
              {{- end }}
            {{- end }}
            {{- $robinvarlistglobal = append $varlisttemp $dictlocal }}
            {{- $varlisttemp = $robinvarlistglobal }}
            {{- end }}
         {{- else }}
         {{- /* ##### ROBIN FEC case - fill the devices string */}}
         {{- $devicestr := .devicePool }}
     robin.io/devices: '{{- $devicestr}}'
         {{- end }}
       {{- end }}
     {{- end }}
     {{- /* ##### END fill the annotation for ROBIN case looping through nw list */}}

     {{- if not (empty $robinvarlistglobal) }}
     robin.io/networks: '{{- $robinvarlistglobal | toJson}}'
     {{- end }}
     {{- if not (empty $.Values.global.k8sPlatform.nodeSelector) }}
     {{- if not (empty $.Values.global.k8sPlatform.nodeSelector.rpoolName) }}
     robin.io/robinrpool: {{ $.Values.global.k8sPlatform.nodeSelector.rpoolName }}
     {{- end }}
     {{- end }}
     {{- range $.Values.global.cnfCorrelationInput }}
     cnfcName: {{ .cnfcName }}
     cnfCorrelationIds: {{ .cnfcCorrelationIds | toJson | quote }}
     {{- end }}
spec:
  {{- /* ##### Check if AR ,FEC is enabled ############# */}}
  {{- $ar := 0 }}
  {{- $fec := 0 }}
  {{- $sriov := 0 }}
  {{- $sriovResourceMap := dict }}
  {{- range $.Values.cnfNetworks.cnfcNwList }}
    {{- if and (eq .cnfcNwType "ar") (eq .cniPluginType "sriov") }}
      {{- $ar = (add1 $ar) }}
    {{- end }}
    {{- if and (eq .cnfcNwType "fec") (eq .cniPluginType "sriov") }}
      {{- $fec = (add1 $fec) }}
    {{- end }}
    {{- if (eq .cnfcNwType "sriov") }}
      {{- $sriov = (add1 $sriov) }}
    {{- end }}
  {{- end }}
  {{- /* ### Loop cnfclist for counting SRIOV interface based on resource nameto get resourcemap### */}}
  {{- range $.Values.cnfNetworks.cnfcNwList }}
     {{- if or (eq .cniPluginType "sriov") (eq .cniPluginType "sriov-dpdk") (eq .cniPluginType "sriov-fec") }}
          {{- if hasKey $sriovResourceMap .resourceName }}
              {{- $_ := set $sriovResourceMap .resourceName (add1 (get $sriovResourceMap .resourceName)) }}
          {{- /* ### End of hasKey ### */}}
          {{- else }}
              {{- $_ := set $sriovResourceMap .resourceName 1 }}
          {{- end }}
          {{- /* ### End of hasKey ### */}}
      {{- end }}
      {{- /* ### End of cnfclist for couting SRIOV interface ### */}}
  {{- end }}
  containers:
  - name: {{ $profileValues.networkFunction.containername }}
    image: {{ $.Values.global.image.repository }}/{{ $profileValues.image.imageName }}:{{ $profileValues.image.imageTag }}
    imagePullPolicy: {{ $profileValues.image.pullPolicy }}
    {{- /* ### introduce a sleep of 10 for initializing vpp before starting app -NEEDED FOR ROBIN ### */}}
    command: ["bash", "-c", "/opt/ani/helm/entrypoint.sh"]
    #command: ["bash","-c","while true; do sleep 1000; done"]
    envFrom:
    - configMapRef:
        name: {{ $nameprefixStr }}-configmap-env
    env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: MY_POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
    - name: MY_NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName
    - name: MY_POD_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    - name: MY_CPU_REQUEST
      valueFrom:
        resourceFieldRef:
          containerName: {{ .containerName }}
          resource: requests.cpu
    - name: MY_CPU_LIMIT
      valueFrom:
        resourceFieldRef:
          containerName: {{ .containerName }}
          resource: limits.cpu
    - name: MY_MEM_REQUEST
      valueFrom:
        resourceFieldRef:
          containerName: {{ .containerName }}
          resource: requests.memory
    - name: MY_MEM_LIMIT
      valueFrom:
        resourceFieldRef:
          containerName: {{ .containerName }}
          resource: limits.memory
    volumeMounts:
    #
    # Volumes (common for CU & DU) = entrypoint , memfs, dskfs, provconfig ,hugepages , inject files
    #  CU specific  - mainconfig
    #  DU specific - devices, ar specific
    #
    - mountPath: /opt/ani/helm/entrypoint.sh
      subPath: entrypoint
      name: entrypoint
    - mountPath: /opt/ani/helm/inject-files.sh
      subPath: inject-files
      name: entrypoint
    - mountPath: /memfs
      name: memfs
    - mountPath: /dskfs/
      name: dskfs-fm
    - mountPath: /prov-config
      name: prov-config
    - mountPath: /ipaddr-config
      name: ipaddr-config
    {{- if $.Values.global.networkFunction.main_config }}
    - mountPath: /main-config
      name: main-config
    {{- end }}
    - mountPath: /mnt/huge
      name: hugepage
      readOnly: false
    - name: devices
      mountPath: /sys/devices
      readOnly: false
    {{- if (eq $ar 1) }}
    - mountPath: /dev/kni
      name: kni
    {{- end }}
    {{- if not (empty $.Values.host_volumes) }}
    {{- if $.Values.host_volumes.inject_files.large }}
    {{- if $.Values.host_volumes.inject_files.large.hostpath }}
    - mountPath: /inject-files-large
      name: inject-files-large
    {{- end }}
    {{- end }}
    {{- if $.Values.host_volumes.inject_files.small }}
    {{- if $.Values.host_volumes.inject_files.small.files }}
    - mountPath: /inject-files-small
      name: inject-files-small
    {{- end }}
    {{- end }}
    {{- end }} # end inject_files
    resources:
      requests:
        {{- toYaml $profileValues.networkFunction.runtime.resources.requests | nindent 8 }}
        {{- range $sriovResourceName,$sriovResourceCount := $sriovResourceMap }}
              {{- printf "%s: %d" $sriovResourceName $sriovResourceCount | nindent 8 }}
        {{- end }}
      limits:
        {{- toYaml $profileValues.networkFunction.runtime.resources.limits | nindent 8 }}
        {{- range $sriovResourceName,$sriovResourceCount := $sriovResourceMap }}
              {{- printf "%s: %d" $sriovResourceName $sriovResourceCount | nindent 8 }}
        {{- end }}
    securityContext:
      {{- toYaml $profileValues.networkFunction.runtime.securityContext | nindent 6 }}
    livenessProbe:
      exec:
        command:
        - /bin/bash
        - -c
        - /opt/ani/scripts/{{ $profileValues.networkFunction.runtime.liveness.script }}
      initialDelaySeconds: {{ $profileValues.networkFunction.runtime.liveness.initialDelaySeconds }}
      periodSeconds: {{ $profileValues.networkFunction.runtime.liveness.periodSeconds }}
      failureThreshold: {{ $profileValues.networkFunction.runtime.liveness.failureThreshold }}
    lifecycle:
      preStop:
        exec:
          command: [/bin/sh,-c,/opt/ani/scripts/prestop_exec.sh]
  terminationGracePeriodSeconds: {{ $profileValues.networkFunction.runtime.terminationGracePeriodSeconds }}
  volumes:
  - name: memfs
    emptyDir:
      medium: Memory
      sizeLimit: {{ $profileValues.networkFunction.config.host_volumes.memfs_storage }}
  - name: dskfs-fm
    # Writes by application on a limited partition
    persistentVolumeClaim:
      claimName: {{ $nameprefixStr }}-pvc-dskfs
  - name: prov-config
    configMap:
      name: {{ $nameprefixStr }}-configmap-provini
      items:
      - key: prov.ini
        path: prov.ini
  - name: ipaddr-config
    configMap:
      name: enodeb-{{ $.Values.global.env.VRAN_ENB_ID }}-{{ lower $profileValues.networkFunction.nfType }}-{{ $.Values.global.env.VRAN_NF_ID }}-configmap-ipaddr
      items:
      - key: ipaddr.ini
        path: ip_addr.ini
  - name: entrypoint
    configMap:
      name: {{ $nameprefixStr }}-configmap-entrypoint
      defaultMode: 0777
  {{- if $.Values.global.networkFunction.main_config }}
  - name: main-config
    configMap:
      name: {{ $nameprefixStr }}-configmap-mainconfig
      items:
      - key: mainconfig
        path: config.xml
      - key: license
        path: license.xml
  {{- end }}
  - name: hugepage
    emptyDir:
      medium: HugePages
  - name: devices
    # Required by dpdk
    hostPath:
      path: /sys/devices
  {{- if (eq $ar 1) }}
  - name: kni
    # Required by kni
    hostPath:
      path: /dev/kni
      type: CharDevice
  {{- end }}
  {{- if not ( empty $.Values.host_volumes) }}
  {{- if $.Values.host_volumes.inject_files.large }}
  {{- if $.Values.host_volumes.inject_files.large.hostpath }}
  - name: inject-files-large
    # Used for file-injection. Lab Feature for single node clusters
    hostPath:
      path: {{ $.Values.host_volumes.inject_files.large.hostpath }}
  {{- end }}
  {{- end }}
  {{- if $.Values.host_volumes.inject_files.small }}
  - name: inject-files-small
    configMap:
      name: {{ $nameprefixStr }}-configmap-inject-small-files
      items:
      {{- range $index, $file:= $.Values.host_volumes.inject_files.small.files }}
        - key: {{ $file.source }}
          path: {{ $file.source }}
      {{- end }}
  {{- end }} # inject_files.small
  {{- end }} # inject_files
  {{- if not ( empty $.Values.global.env.SECGW_INTERNAL_DNS) }}
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - {{ $.Values.global.env.SECGW_INTERNAL_DNS }}
  {{- end }}
{{- else }}
{{- fail "Deployment file does not exist" }}
{{- end }}

It contains 2 YAML documents within single file. Even if I remove the first one it still cannot parse (just for testing) although I need to complete file to be parsed.

ruamel.yaml
Original Q&A
1

There are 1 best solutions below

5
On

There are multiple problems here, partly due to limitations in ruamel.yaml.jinja2.

The error you indicate is created by the {{ ... }} pattern, that occurs on a line of its own, in what seems to be a jinja2 construct starting with {{- with (which correctly gets commented out before loading).

I have not been able to find documentation for that. The with statement in jinja2 is different, and without proper documentation I hesitate to update the ruamel.yaml.jinja2 extension. It also doesn't 100% match the {{ with ....}} / {{ end }} of the go-templates (no dash after the curly braces.

You should pre-process the data to get the offending line out of the way, by commenting out all the lines between with and end (the latter at the same indentation level), but it would nice if this were documented, to give more confidence that is the right approach in general.

import sys
import io
import ruamel.yaml
from ruamel.yaml.split import split
from pathlib import Path

input_file = Path('input.jinja2')
output_file = Path('output.jinja2')

# wty = ('{{ toYaml', '#{{ toYaml')

def reverse(s):
    return s.replace(wty[1], wty[0])

yaml = ruamel.yaml.YAML(typ='jinja2')
yaml.preserve_quotes = True
yaml.width = 256
yaml.explicit_start = True

def comment_out_with(doc, entry):
    res = []
    end = None
    for line in doc.splitlines(True):
        if end:
            res.append('#<<<' + line)
            if line.startswith(end):
                end = None
            continue
        if line.rstrip().startswith(entry[0]):
            res.append('#<<<' + line)
            end = line.split(entry[0])[0] + entry[1]  # prepend the indentation
            continue
        res.append(line)
    return ''.join(res)

def reverse(s):
    s = s.replace('#<<<{{', '{{')
    s = s.replace('#<{', '{{')
    return s

# list of tuples (data, doc), if data is None, write doc as None cannot have comments
documents = []

for doc, line_nr in split(input_file):
    dd = doc.decode('utf-8')
    for with_entry in [('{{- with', '{{- end'), ('{{ with', '{{ end')]:
        if with_entry[0] in dd:
            dd = comment_out_with(dd, with_entry)
    data = yaml.load(dd)  # jinja2 cannot handle bytes yet
    documents.append((data, doc))

for data, doc in documents:
    if data is None:
        continue
    # make your changes to the YAML data here
    data['spec']['containers'][0]['env'][0]['name'] = 'my_node_name'

with output_file.open('w') as fpo:
    for data, doc in documents:
        if data is None:
            fpo.write(doc.decode('utf-8'))
        else:
            buf = io.StringIO()
            yaml.dump(data, buf)
            fpo.write(reverse(buf.getvalue()))

The "trick" with writing the original document is necessary because ruamel.yaml cannot attach comments to None (as it cannot be subclassed in a meaningful way).

A minor thing, that is probably not a issue is that most block style sequences have indentation of 2 with no offset, but the sequences that are the values for the keys items and nameservers close to the end of the file have an indentation of 4 with an offset of 2.

Flow style sequences ( [/bin/sh,-c,/opt/ani/scripts/prestop_exec.sh] ) get space after the comma.

Apart from these minor "standardisations" that should not affect loading of the converted template , the output file differs only in the programmatically updated "my_node_name"

Related Questions in RUAMEL.YAML

Trending Questions

Popular # Hahtags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json python-3.x ruby-on-rails .net sql-server swift django angular objective-c pandas excel

Popular Questions

Copyright © 2021 Jogjafile Inc.