I am executing pyspark script on EMR serverless and i am using AWS step function for creating EMR application and submitting job in it. I want to know how to add logging option in AWS step function configuration while creating EMR application or submitting job in it so that i can see log files from spark job run in cloudwatch or in s3. this is my current block from StartJobRun state from AWS step function configuration.
"EMR Serverless StartJobRun": { "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startJobRun", "Parameters": { "ApplicationId.$": "$.ApplicationId", "ExecutionRoleArn": "arn:aws:iam:/demo-emrserverless-iam-role", "JobDriver": { "SparkSubmit": { "EntryPoint": "s3://demo/test.py", "EntryPointArguments.$":[], "SparkSubmitParameters": "--conf } },
i tried to add "JobConfiguration" but it is not accepting JobConfiguration