How to share instantiated objects defined by c++ between python?

  c++, python

background:
In large-scale deep learning training, I need to read the data from the HDFS machine. The interface used is the class defined in C++, which is used in python through pybind11. E.g:

// in c++
class Reader {
public:
   py::bytes read();
private:
   std::vector<int> indexs;
};
PYBIND11_MODULE(dataloader, m) {
    py::class_<Reader>(m, "Reader")
}
# in python
import Reader
reader = Reader()
'''do something with reader'''

Now, in order to do data shuffle, I store a lot of data in the c++reader class. When I use it in python, I need to use multiple GPUs and multi-process training. If each process has a reader instantiation object, this will Occupies a huge amount of memory. Now I hope that in a multi-GPU and multi-process scenario, multiple processes can share the same reader object to ensure that the memory footprint is relatively small. Is there any good way?

Source: Windows Questions C++

LEAVE A COMMENT