Unable to run Elasticmap reduce pyspark job on AWS Datapipeline

49 Views Asked by At

Unable to run EMR activity on AWS Data pipeline. Added the job step and bootstrap action step to execute the spark job but it's not able to pick it up.

added job step spark-submit, --deploy-mode, cluster, s3://customer-arc/scripts/iam.py

added bootstrap step (to install modules while starting the cluster) s3://customer-arc/scripts/modules.sh

modeules.sh


# Non-standard and non-Amazon Machine Image Python modules:
sudo pip3 install -U \
  awscli            \
  boto3             \
  dynamodb_json     \
  pandas            \
  awswrangler       \

Please help me to configure thisAWS data pipeline parameteres in the AWS data pipeline, I am unable to figure out what else to put in steps and bootstrap action space in the AWS data pipeline.

0

There are 0 best solutions below