Category : apache-arrow

I would like to obtain a std::shared_ptr<arrow::RecordBatch> from an std::shared_ptr<arrow::Table> as std::shared_ptr<arrow:Table> table = … auto rb = std::RecordBatch::Make(table->schema(), table->num_rows(), table->columns()).ValueorDie(); However the compiler complains that there’s no known conversion from ‘const vector<shared_ptr<arrow::ChunkedArray>>’ to ‘vector<shared_ptr<arrow::Array>>’ since the table->columns() of course returns vector<shared_ptr<arrow::ChunkedArray>>. I can’t seem to convert the arrow::ChunkedArray into arrow::Array. I’ve poured over the ..

Read more

#include <pybind11/pybind11.h> #include <Python.h> #include <iostream> #include <arrow/python/pyarrow.h> #include <arrow/table.h> // Convert pyarrow table to native C++ object and print its contents void print_table(PyObject* py_table) { // convert pyobject to table auto status = arrow::py::unwrap_table(py_table); if (!status.ok()) { std::cout << "Error converting pyarrow table to arrow table" << std::endl; return; } std::shared_ptr<arrow::Table> table = status.ValueOrDie(); ..

Read more

I’m trying to pass a pyarrow table to c++ via pybind11. In this example I’m simply trying to print the number of rows of a pyarrow table passed from python. #include <pybind11/pybind11.h> #include <Python.h> #include <iostream> #include <arrow/python/pyarrow.h> // Convert pyarrow table to native C++ object and print its contents void print_table(PyObject* py_table) { // ..

Read more

I started to use Apache arrow. I installed it via vcpkg. Everything was ok until I updated the installed arrow to 6.0.1. I installed it using following command from powershell console: vcpkg install arrow:x64-windows-static It was installed successfully and I saw the success message (somehow): The package arrow provides CMake targets: find_package(arrow CONFIG REQUIRED) target_link_libraries(main ..

Read more

I have a scenario where I am working with temporal data in Apache Arrow and am using compute functions to extract date/time components like so: auto year = arrow::compute::CallFunction("year", {array}); auto month = arrow::compute::CallFunction("month", {array}); auto day = arrow::compute::CallFunction("day", {array}); … While this works, I have to manage three separate Datums. I would ideally like ..

Read more

Is there any way to create a dynamic container of arrow::ArrayBuilder objects? Here is an example int main(int argc, char** argv) { std::size_t rowCount = 5; arrow::MemoryPool* pool = arrow::default_memory_pool(); std::vector<arrow::Int64Builder> builders; for (std::size_t i = 0; i < 2; i++) { arrow::Int64Builder tmp(pool); tmp.Reserve(rowCount); builders.push_back(tmp); } return 0; } This yields error: variable ‘arrow::Int64Builder ..

Read more

I’m trying to write an Apache Arrow table to a string. My big example has problems and I can’t get this little example to work. This one segfaults inside of Arrow in the WriteTable call. My bigger example doesn’t appear to serialize correctly. #include <arrow/api.h> #include <arrow/io/memory.h> #include <arrow/ipc/api.h> std::shared_ptr<arrow::Table> makeSimpleFakeArrowTable() { std::vector<std::shared_ptr<arrow::Field>> arrowFields; arrowFields.emplace_back(std::make_shared<arrow::Field>("Field1", ..

Read more