Launching Spark Task via AWS Lambda

Question

I am running Spark container on AWS lambda and unable to launch the spark job successfully. This approach is based on SoAL architecture (Spark on AWS Lambda https://github.com/aws-samples/spark-on-aws-lambda) but our environment in purely Java.

The container used to deploy the lambda has java class which uses SparkLauncher to start the spark task. SparkLauncher is used as below this basically trigger the jar file which has spark task bundled.

Apache Spark version 3.3.1 Java 11

     Map<String, String> env = new HashMap<>();
                env.put("SPARK_PRINT_LAUNCH_COMMAND", "1");

                // Set up SparkLauncher
                SparkLauncher launcher = new SparkLauncher(env)
                        .setVerbose(true)
                        .setSparkHome(SPARK_HOME)
                        .setAppResource(jarFile.getAbsolutePath()) // Path to the downloaded jar
                        .setMainClass(taskInput.getEntryClass())
                        .addAppArgs(taskInput.getJarArgs())
                        .setAppName(APP_NAME)
                        .setConf("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
                        .setConf("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
                        .setConf("spark.hadoop.fs.s3a.endpoint", "s3.amazonaws.com")
                        .setConf("spark.hadoop.fs.s3a.path.style.access", "true")
                        .setConf("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.auth.InstanceProfileCredentialsProvider")
                        .setConf("spark.driver.memory", "5g")
                        .setConf("spark.executor.memory", "5g")
                        .redirectError(ProcessBuilder.Redirect.INHERIT)
                        .redirectOutput(ProcessBuilder.Redirect.INHERIT);

 SparkAppHandle handle = launcher.startApplication();
                handle.addListener(new CustomSparkListener());

                System.out.println("Started Spark Application.....");

                // Poll until application gets submitted
                while (handle.getAppId() == null) {
                    System.out.println("Waiting for application to be submitted: status=" + handle.getState());

                    if(handle.getState().name().equals("FAILED")) {
                        if(handle.getError().isPresent()) {
                            handle.getError().get().printStackTrace();
                        }

                        System.out.println("ERROR");

                        break;
                    }
                    else{
                        System.out.println("NO ERROR");
                    }
                    Thread.sleep(60000L);
                }

As seen above I have Spark App Handler registered, when lambda is triggered I get to see the Failed state but nothing get printed as part of handle.getError().get().printStackTrace();

This has blocked me to know the reason for launch failure.

Docker file used

FROM public.ecr.aws/lambda/java:11

ARG SPARK_VERSION=3.4.2
ARG HADOOP_VERSION=3
ARG HADOOP_AWS_VERSION=3.3.4
ARG SPARK_HOME=/opt/spark
ARG AWS_SDK_VERSION=1.12.262
ARG DELTA_FRAMEWORK_VERSION=2.4.0
ARG JAVA_HOME=/var/lang

ENV PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin
ENV SPARK_LOCAL_IP 0.0.0.0
ENV AWS_REGION="xx"
ENV AWS_ROLE_ARN=""

# Install dependencies
RUN yum -y upgrade
RUN yum -y install wget
RUN yum -y install tar
RUN yum -y install gzip
RUN yum -y install procps

# Download and extract Spark
RUN wget -qO- https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}.tgz  \
    | tar xz -C /opt && \
      mv /opt/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_VERSION} ${SPARK_HOME}

RUN wget -q https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar -P ${SPARK_HOME}/jars/
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_AWS_VERSION}/hadoop-aws-${HADOOP_AWS_VERSION}.jar -P ${SPARK_HOME}/jars
RUN wget -q https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-common/${HADOOP_AWS_VERSION}/hadoop-common-${HADOOP_AWS_VERSION}.jar -P ${SPARK_HOME}/jars
RUN wget -q https://repo1.maven.org/maven2/io/delta/delta-core_2.12/${DELTA_FRAMEWORK_VERSION}/delta-core_2.12-${DELTA_FRAMEWORK_VERSION}.jar -P ${SPARK_HOME}/jars/

COPY target/classes ${LAMBDA_TASK_ROOT}
COPY target/dependency/* ${LAMBDA_TASK_ROOT}/lib/

CMD [ "com.xxx.sparkonlambda.SparkOnLambdaHandler::handleRequest" ]

I tried to run the container on emulator provided by AWS the Spark Task got launched successfully.

I am expecting that if it is working on emulator then it should work in AWS lambda environment as well. Also how can I get to know the reason for FAILED state ?