Passing a file to rundeck and reading it in python

Question

Sorry if this question seems pointless, im a very new user to Rundeck. I have a python script that need a file to operate so i added a file option to my rundeck job. But for some reason i cant get the file to work:

Here is a sample python code:

import os
import sys

print(len(sys.argv))                  # Just to see the passed args
file_path = sys.argv[1].strip()
file_name = sys.argv[2].strip()
hsp = os.path.join(file_path, file_name)

# Testing all possible paths
print('file_path ', os.path.isfile(file_path), os.path.isfile(file_path)) 
print('file_name ', os.path.isfile(file_name), os.path.isfile(file_name))
print('hsp ', os.path.isfile(hsp), os.path.isfile(hsp))

My inline command:

 /opt/env/bin/python /opt/script/main.py ${file.file_1} ${file.file_1.fileName}

My Rundeck output:

['/opt/script/main.py', '/home/rundeck/var/upload/9a6835df-f6fd-4f75-8d25-0964b6be6d40', 'file_1.xlsx'']

history_path  False False
history_name  False False
hsp  False False

Rundeck Job Definition:

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: random_id
  loglevel: INFO
  name: project_name
  nodeFilterEditable: false
  nodefilters:
    dispatch:
      excludePrecedence: true
      keepgoing: false
      rankOrder: ascending
      successOnEmptyNodeFilter: false
      threadcount: '1'
    filter: XYXYX.com
  nodesSelectedByDefault: true
  options:
  - label: History File
    name: history_path
    required: true
    type: file
  - description: 'The second optional file' 
    label: Other File
    name: file2
    type: file
  - dateFormat: YYYY-MM-DD
    description: 'Start date. '
    isDate: true
    label: Start date
    name: start
    regex: ^\d{4}-\d{2}-\d{2}$
  - dateFormat: YYYY-MM-DD
    description: 'End date'
    isDate: true
    label: End date
    name: end
    regex: ^\d{4}-\d{2}-\d{2}$
  plugins:
    ExecutionLifecycle: null
  scheduleEnabled: true
  sequence:
    commands:
    - description: Execute the script
      exec: ' /opt/project_venv/project/bin/python /opt/projects/project_file/main.py
        ${file.history_path} ${file.history_path.fileName} ${option.method} ${option.exp_format}
        ${file.file2} ${file.file2.fileName} ${option.start} ${option.end}'
    keepgoing: false
    strategy: node-first
  uuid: uuid

Can someone please help me understand where im going wrong and what i can do to fix this (i wanna get the patha nd read the file into pandas!).

Thank you!!

Answer 1

To achieve that you need to use a script step in your Rundeck workflow (passing the job options in the "arguments" field) and use the file_path, file_name, and hsp without single quotes in the Python print statements.

Python code:

import os
import sys

print(len(sys.argv))                  # Just to see the passed args
file_path = sys.argv[1].strip()
file_name = sys.argv[2].strip()
hsp = os.path.join(file_path, file_name)

# Testing all possible paths
print(file_path , os.path.isfile(file_path), os.path.isfile(file_path)) 
print(file_name , os.path.isfile(file_name), os.path.isfile(file_name))
print(hsp, os.path.isfile(hsp), os.path.isfile(hsp))

And Rundeck Job Definition Example:

This job calls your python code using the script step, and passing the options values (path and file) on the arguments field.

- defaultTab: nodes
  description: ''
  executionEnabled: true
  id: 35a3cf93-a0d2-44ef-a462-f0d20b7a22a2
  loglevel: INFO
  name: PythonCode
  nodeFilterEditable: false
  options:
  - name: file_name
    type: file
  - name: file_path
    value: /home/user/Downloads/
  plugins:
    ExecutionLifecycle: {}
  scheduleEnabled: true
  sequence:
    commands:
    - exec: 'echo "path: ${option.file_path} file: ${file.file_name.fileName}"'
    - args: ${option.file_path} ${file.file_name.fileName}
      expandTokenInScriptFile: true
      fileExtension: .py
      interpreterArgsQuoted: false
      scriptInterpreter: /usr/bin/python3
      scriptfile: /home/user/Downloads/mycode.py
    keepgoing: false
    strategy: node-first
  uuid: 35a3cf93-a0d2-44ef-a462-f0d20b7a22a2

You can import this definition to test it. To do so: go to the Jobs page, click on the "Jobs Action" list button > Upload Definition, and upload the YAML file. Consider that the paths included in this example are fictional, you need to change them according to your environment.

And the execution result:

path: /home/user/Downloads/ file: myfile.txt
3
/home/user/Downloads/ False False
myfile.txt False False
/home/user/Downloads/myfile.txt True True

This solution has been tested on Rundeck 5.1.0, the latest at this moment.