Is it possible to read a file using SparkSession object of Scala language on Windows?

  apache-spark, file, scala, windows

I’ve been trying to read from a .csv file on many ways, utilizing SparkContext object. I found it possible through scala.io.Source.fromFile function, but I want to use spark object. Everytime I run function textfile for org.apache.spark.SparkContext I get the same error:

scala>     sparkSession.read.csv("file://C:Users184229Desktopbigdata.csv")


21/12/29 16:47:32 WARN streaming.FileStreamSink: Error while looking for metadata directory.

java.lang.UnsupportedOperationException: Not implemented by the DistributedFileSystem FileSystem implementation
.....

As it’s mentioned in the title I run the code on Windows in IntelliJ

Source: Windows Questions

LEAVE A COMMENT