Callback in libaio for write/read in C++

  aio, c++, io, read-write

I have a file /tmp/test of size 64MB, and I am writing to it till end of file using block size of 4k using libaio library in c++. I have a queue depth of 16. So, I will store 16 requests in an iocbs array and once I get 16 requests, I will submit them together using io_submit. As soon as I submit the queue, I am starting another queue, so that the total number of jobs in the queue remains as 16. In order to achieve this, I have a main function foo that writes to the full file 1000 times, and as soon as the requests reach 16 (queue depth) I submit the queue, and I have a callback function that reads from the file /tmp/test and submits the request.

This is the callback function that I have currently:

void aio_callback(io_context_t ctx, struct iocb *iocb, long res, long res2)
{
   if(res2!=0)
   {
      fprintf(stderr, "error in aio event");
   }
   if(res!=iocb->u.c.nbytes)
   {
      fprintf(stderr, "incomplete io");
   }
   int fd_read = open("/tmp/test", O_RDONLY|O_DIRECT);
   void* read_buf=NULL;
   int buff_size = 4*1024;
   struct iocb *iocbs;
   posix_memalign(&read_buf, 512, buff_size);
   memset(read_buf, '{$content}', buff_size);
   io_prep_pread(iocb, fd_read, (void*)read_buf, buff_size, 0);
   iocbs = iocb;
   res = io_submit(ctx, maxio, &iocbs);
   if(res<0)
   {
     std::cout<<"callback io_submit_errorn";
     exit(-2);
   }
   close(fd_read);
}

struct iocb* iocbs[100];
struct io_event events[100];
int QueueDepth = 16;
void foo()
{
   //this is the main function that writes to a 64MB file till EOF using block size (buffer size) as 4k
  //I need to write to the full file 1000 times, so I have an outer loop running 1000 times
  int ops = 1000;
  int fd_write = open("/tmp/test", O_WRONLY|O_CREAT|O_DIRECT);
  while(ops--)
  {
    int count = 0;
    do
    {
       struct iocb *cb = (struct iocb*)malloc(sizeof(struct iocb));
       io_prep_pwrite(cb, fd_write, (void*)buffptr, buff_size, buff_size*(count) + (stride_length*count));
       io_set_callback(cb, aoi_callback);
       iocbs[curr_qd++] = cb; //curr_qd is the current queue depth
       count++;               //since file descriptor does not have implementation for eof we maintain count to shift the fd to current position
       if(curr_qd >= QueueDepth)
       {
          int res = io_submit(ctx, maxio, iocbs);
          curr_qd = 0; //since the queue has been submitted, current depth = 0
          int res_event = io_getevents(ctx, 1, maxio, events, NULL);
          if(res_event == res)
          {
              for(int i=0;i<QueueDepth;i++)              //line 1
                 aoi_callback(ctx, events[i].obj, events[i].res, events[i].res2);  //line 2
          } 
       }
    }while(count<(file_size/(buff_size + stride_length)));
  }
}

When I run the program, I am getting io_submit error with error code EAGAIN : Insufficient resources are available to the queue
When I comment the io callback function (i.e.) line 1 and line 2, the program works fine.

I was not able to find what exactly the EAGAIN error means or why it occurs. What is going wrong with my code?
I am a beginner with libaio library and programming using any aoi library. Would really appreciate any help or guidance.

Source: Windows Questions C++

LEAVE A COMMENT