Category : apache-spark

C:UsersHP>spark-shell Exception in thread "main" java.lang.ExceptionInInitializerError at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:54) at org.apache.spark.internal.config.package$.<init>(package.scala:1006) at org.apache.spark.internal.config.package$.<clinit>(package.scala) at org.apache.spark.deploy.SparkSubmitArguments.$anonfun$loadEnvironmentArguments$3(SparkSubmitArguments.scala:157) at scala.Option.orElse(Option.scala:447) at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:157) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:115) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:990) at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:990) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:85) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @7bedc48a at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357) at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297) ..

Read more

Installed spark (spark-3.1.1-bin-hadoop2.7) downloaded java jdk to "C:Program FilesJavajdk1.8.0_281" downloaded anaconda downloaded winutils.exe to C:sparkspark-3.1.1-bin-hadoop2.7bin 5.I have set the Windows Environment variables to, SPARK_HOME=C:sparkspark-3.1.1-bin-hadoop2.7 HADOOP_HOME=C:sparkspark-3.1.1-bin-hadoop2.7 PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook JAVA_HOME="C:Program FilesJavajdk1.8.0_281" (I have tried the suggestion of changing the above to c:Progra~1Javajdk1.8.0_281 and that doesnt work either) I am able to launch jupyter notebook. I then copy ..

Read more

I am facing the below error to access hive from local windows using winutils. The below steps are followed to access hive. Can some one help on solving my problem. ERROR Facing: Exception in thread "main" org.apache.spark.sql.AnalysisException: java.lang.RuntimeException: java.io.FileNotFoundException: File /tmp/hive does not exist; Step 1: created a folder as below and placed winutils.exe file. ..

Read more

In my spark job, each executor will initialize a c++ library which can segment a text line into word list. When two or more executors allocated on the same machine, initialization of the c++ library multiple times will cause core dump. public class TextSegment implements Serializable { private SparkSession spark = SparkSession.builder() .appName("TextSegment") .getOrCreate(); public ..

Read more