I am on a Windows, these are the relevant values:

(base) C:\Users\User>echo %JAVA_HOME%
C:\Users\User\Downloads\OpenJDK8U-jdk_x64_windows_hotspot_8u392b08\jdk8u392-b08

(base) C:\Users\User>echo %PYSPARK_SUBMIT_ARGS%
--jars C:\spark-3.5.0-bin-hadoop3\jars\graphframes-0.8.3-spark3.5-s_2.12.jar pyspark-shell

(base) C:\Users\User>echo %SPARK_HOME%
C:\spark-3.5.0-bin-hadoop3

(base) C:\Users\User>pyspark --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.0
      /_/

Using Scala version 2.12.18, OpenJDK 64-Bit Server VM, 1.8.0_392
Branch HEAD
Compiled by user ubuntu on 2023-09-09T01:53:20Z
Revision ce5ddad990373636e94071e7cef2f31021add07b
Url https://github.com/apache/spark
Type --help for more information.

(base) C:\Users\User>java -version
openjdk version "1.8.0_392"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_392-b08)
OpenJDK 64-Bit Server VM (Temurin)(build 25.392-b08, mixed mode)

(base) C:\Users\User>javac -version
javac 1.8.0_392

the error starts when trying to create a new SparkSession using the Session Builder :

PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.
Code :  spark = SparkSession.builder.appName("YourAppName").getOrCreate()

Please help!

I have tried setting up Pyspark and graphframes on colab and it works fine but i cant seem to set it up on my local system. What am i doing wrong?

0

There are 0 best solutions below