How to pass or allow spark-operator pod in eks to access AWS resources like S3

25 Views Asked by At

I have installed spark-operator on eks in private aws account (doesn't NAT to access internet/ limited access can't opensource images). Followed aws official document emr-on-eks spark-operator. However I encountered with s3 access denied error in driver pod, after couple of settings on eks cluster by adding add-on for pod and creating service account with below permissions

{
        "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
        "Effect": "Allow",
        "Principal": {
            "Service": "pods.eks.amazonaws.com"
        },
        "Action": [
            "sts:AssumeRole",
            "sts:TagSession"
        ]
    }

Now, I ended up getting below error.

Exception in thread "main" java.io.IOException: Class class software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider does not implement AWSCredentialsProvider

Any suggestions to create spark-operator on eks cluster. Also what should I need to investigate/ allow cluster to create aws credentials or access to aws resources instead of aws credentials in pod or eks. Note: I can able to access s3 without aws credentials in eks cluster but not in pod.

0

There are 0 best solutions below