Read files from Windows shared folder using Airflow on remote server?

  airflow, etl, remote-server, shared-directory, windows

What is the best method for reading files from a Windows shared folder using a DAG on a remote Apache Airflow server?

My current process needs to be executed a remote Airflow server, but it needs to read and write files (which the client and my process update) that are stored on a shared net Windows folder.

  • Airflow is installed on remote Linux server (same network)
  • Windows folders are just standard UNC paths where people have access based on their NT ID. These users are saving files which I need to retrieve.
  • The files formats are .csv, .xls and .xlsx, which I plan to convert as Dataframes in my Airflow process
  • Best if I can save my user and password within an Airflow connection to access it with a conn_id

Source: Windows Questions

LEAVE A COMMENT