Reading parquet file is slower in c++ than in python

  apache-arrow, c++, parquet, pyarrow, python

I have written code to read the same parquet file using c++ and using python. The time taken to read the file is much less for python than in c++, but as generally we know, execution in c++ is faster than in python. I have attached the code here –

#include <arrow/api.h>
#include <parquet/arrow/reader.h>
#include <arrow/filesystem/localfs.h>
#include <chrono>
#include <iostream>

int main(){
   // ...
   arrow::Status st;
   arrow::MemoryPool* pool = arrow::default_memory_pool();
   arrow::fs::LocalFileSystem file_system;
   std::shared_ptr<arrow::io::RandomAccessFile> input = file_system.OpenInputFile("data.parquet").ValueOrDie();

   // Open Parquet file reader
   std::unique_ptr<parquet::arrow::FileReader> arrow_reader;
   st = parquet::arrow::OpenFile(input, pool, &arrow_reader);
   if (!st.ok()) {
      // Handle error instantiating file reader...
   }

   // Read entire file as a single Arrow table
   std::shared_ptr<arrow::Table> table;
   auto t1 = std::chrono::high_resolution_clock::now();
   st = arrow_reader->ReadTable(&table);
   auto t2 = std::chrono::high_resolution_clock::now();
   if (!st.ok()) {
      // Handle error reading Parquet data...
   }
   else{
       auto ms_int = std::chrono::duration_cast<std::chrono::milliseconds> (t2 - t1);
       std::cout << "Time taken to read parquet file is : " << ms_int.count() << "msn";
   }
}

The code i used in python is –

#!/usr/bin/env python3
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
import time

start_time = time.time()

table = pq.read_table('data.parquet')

end_time = time.time()

print("Time taken to read parquet is : ",(end_time - start_time)*1000, "ms")

On running the c++ code for a file of size about 87mb, the output for c++ is –

Time taken to read parquet file is : 186ms

While for python the output is –

Time taken to read parquet is : 108.66141319274902 ms

Why there is such a difference in time of execution for the function read_table in c++ and python ?

Source: Windows Questions C++

LEAVE A COMMENT