raise Exception: Java gateway process exited before sending its port number

  apache-spark, pyspark, python, windows, yarn

I have a problem, when use yarn-client. With master=local[*] all right!
I use Java 8 (jre8u202), spark on yarn – 2.4.0, on my computer – 2.3.2 (becouse 2.4.0 version on my computer do not work and i have problem: Python worker failed to connect back), pySpark – 2.3.2, Scala – 2.12.0, hadoop – 2.6. Also, i write the code on spyder on windows.
I use environment variables, as following: JAVA_HOME, HADOOP_HOME, SCALA_HOME, SPARK_HOME, PYSPARK_DRIVER_PYTHON=’jupyter’, PYSPARK_PYTHON=’python’, SPARK_DRIVER_MEMORY=10g.
I have the I saw a lot of different solutions for this problem, but solutions don’t work.
My code:

conf=SparkConf()
conf.setMaster('yarn-client')
#conf.setMaster('local[*]')
conf.setAppName('PythonSparkintegrationexample4')
sc=SparkContext(conf=conf)
sc.setCheckpointDir("/Downloads/Logs")
file=sc.textFile(file_dir) 
words = file.flatMap(lambda line: line.split(" ")) 
wordCounts = words.countByValue()
for word, count in wordCounts.items():
    print("{} : {}".format(word, count)) 
sc.stop()

My error:


  File "C:ProgramDataAnaconda3libsite-packagesspyder_kernelscustomizespydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "C:ProgramDataAnaconda3libsite-packagesspyder_kernelscustomizespydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/SUBD-PodpruzhnikovIA/Desktop/test_spark_6.py", line 86, in <module>
    sc=SparkContext(conf=conf)

  File "C:ProgramDataAnaconda3libsite-packagespysparkcontext.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)

  File "C:ProgramDataAnaconda3libsite-packagespysparkcontext.py", line 300, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)

  File "C:ProgramDataAnaconda3libsite-packagespysparkjava_gateway.py", line 93, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")

Exception: Java gateway process exited before sending its port number

I will be very grateful for any ideas, because I have been sitting with this problem for several days.

Source: Windows Questions

LEAVE A COMMENT