Unable to read text files on Windows machine with Pyspark. (Analysis Exception) with a file:// prefix for my file name

  apache-spark, pyspark, python, windows

Error:

AnalysisException: Path does not exist: file:/D:/sk –
Add/Main/2020/Main_2012-01.txt;

Code1 with error:

for i in os.listdir():
   if j.endswith('.txt'):
        print(i)
        df= spark.read.text(i)

Code2 with same err:

path=r"D:sk - AddMain20"
for i in os.listdir():
   if j.endswith('.txt'):
        print(i)
        df= spark.read.text(path+''+i)

Code3 without error:

df1=spark.read.text(r'D:sk - AddMain20Main_2020-12.txt')

Why is it adding file:/ prefix for my file name and causing the error.
In the 3rd case, it doesn’t. How do I resolve this?

Source: Windows Questions

LEAVE A COMMENT