What is the best method for reading files from a Windows shared folder using a DAG on a remote Apache Airflow server?
My current process needs to be executed a remote Airflow server, but it needs to read and write files (which the client and my process update) that are stored on a shared net Windows folder.
- Airflow is installed on remote Linux server (same network)
- Windows folders are just standard UNC paths where people have access based on their NT ID. These users are saving files which I need to retrieve.
- The files formats are .csv, .xls and .xlsx, which I plan to convert as Dataframes in my Airflow process
- Best if I can save my user and password within an Airflow connection to access it with a conn_id
Source: Windows Questions