I would like to obtain a std::shared_ptr<arrow::RecordBatch> from an std::shared_ptr<arrow::Table> as std::shared_ptr<arrow:Table> table = … auto rb = std::RecordBatch::Make(table->schema(), table->num_rows(), table->columns()).ValueorDie(); However the compiler complains that there’s no known conversion from ‘const vector<shared_ptr<arrow::ChunkedArray>>’ to ‘vector<shared_ptr<arrow::Array>>’ since the table->columns() of course returns vector<shared_ptr<arrow::ChunkedArray>>. I can’t seem to convert the arrow::ChunkedArray into arrow::Array. I’ve poured over the ..
Category : apache-arrow
#include <pybind11/pybind11.h> #include <Python.h> #include <iostream> #include <arrow/python/pyarrow.h> #include <arrow/table.h> // Convert pyarrow table to native C++ object and print its contents void print_table(PyObject* py_table) { // convert pyobject to table auto status = arrow::py::unwrap_table(py_table); if (!status.ok()) { std::cout << "Error converting pyarrow table to arrow table" << std::endl; return; } std::shared_ptr<arrow::Table> table = status.ValueOrDie(); ..
I’m trying to pass a pyarrow table to c++ via pybind11. In this example I’m simply trying to print the number of rows of a pyarrow table passed from python. #include <pybind11/pybind11.h> #include <Python.h> #include <iostream> #include <arrow/python/pyarrow.h> // Convert pyarrow table to native C++ object and print its contents void print_table(PyObject* py_table) { // ..
I started to use Apache arrow. I installed it via vcpkg. Everything was ok until I updated the installed arrow to 6.0.1. I installed it using following command from powershell console: vcpkg install arrow:x64-windows-static It was installed successfully and I saw the success message (somehow): The package arrow provides CMake targets: find_package(arrow CONFIG REQUIRED) target_link_libraries(main ..
I have a scenario where I am working with temporal data in Apache Arrow and am using compute functions to extract date/time components like so: auto year = arrow::compute::CallFunction("year", {array}); auto month = arrow::compute::CallFunction("month", {array}); auto day = arrow::compute::CallFunction("day", {array}); … While this works, I have to manage three separate Datums. I would ideally like ..
Is there any way to create a dynamic container of arrow::ArrayBuilder objects? Here is an example int main(int argc, char** argv) { std::size_t rowCount = 5; arrow::MemoryPool* pool = arrow::default_memory_pool(); std::vector<arrow::Int64Builder> builders; for (std::size_t i = 0; i < 2; i++) { arrow::Int64Builder tmp(pool); tmp.Reserve(rowCount); builders.push_back(tmp); } return 0; } This yields error: variable ‘arrow::Int64Builder ..
In am looping over the elements of an Arrow Array and trying to apply a compute function to each scalar that will tell me the year, month, day, etc… of each element. The code looks something like this: arrow::NumericArray<arrow::Date32Type> array = {…} for (int64_t i = 0; i < array.length(); i++) { arrow::Result<std::shared_ptr<arrow::Scalar>> result = ..
I’m trying to write an Apache Arrow table to a string. My big example has problems and I can’t get this little example to work. This one segfaults inside of Arrow in the WriteTable call. My bigger example doesn’t appear to serialize correctly. #include <arrow/api.h> #include <arrow/io/memory.h> #include <arrow/ipc/api.h> std::shared_ptr<arrow::Table> makeSimpleFakeArrowTable() { std::vector<std::shared_ptr<arrow::Field>> arrowFields; arrowFields.emplace_back(std::make_shared<arrow::Field>("Field1", ..
I have a special use case where I want to read only selected parquet files(list of files) from a directory on hdfs filesystem. I know I can use whole directory as dataset and then apply filter on data, but I want to read only selected list of files. How can I do that using arrow ..
I am new to apache arrow cpp api. I want to read multiple parquet files using apache arrow cpp api, similar to what is there in apache arrow using python api(as a table). However I don’t see any example of it. I know I can read a single parquet file using : arrow::Status st; arrow::MemoryPool* ..
Web Development Company
Recent Comments