I have installed spark-operator on eks in private aws account (doesn't NAT to access internet/ limited access can't opensource images). Followed aws official document emr-on-eks spark-operator. However I encountered with s3 access denied error in driver pod, after couple of settings on eks cluster by adding add-on for pod and creating service account with below permissions
{
"Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
]
}
Now, I ended up getting below error.
Exception in thread "main" java.io.IOException: Class class software.amazon.awssdk.auth.credentials.ContainerCredentialsProvider does not implement AWSCredentialsProvider
Any suggestions to create spark-operator on eks cluster. Also what should I need to investigate/ allow cluster to create aws credentials or access to aws resources instead of aws credentials in pod or eks. Note: I can able to access s3 without aws credentials in eks cluster but not in pod.