All AruCo Tags are getting Rejected when I pass the image data from Unity

  aruco, c++, dll, opencv, unity3d

I am trying to detect AruCo markers from image data sent from Unity to OpenCV c++ dll.

Versions
OpenCV:4.5.0/4.5.1
Unity : 2020.1.17f1
Aruco dictionary : 6x6_250 family and AprilTags 36h11 family

The problems I am facing is:
i) when I use a camera and send each frame(as shown in code here), detectMarkers() function works but every tag gets rejected.
I have tried:
a) flipping the image only in Unity using System.Array.Reverse(rawImg);
–I have also tried flipping the image only in c++ using flip(imageCopy, imageCopy, -1) and then back after the image is processed;
–and in a combination of both in Unity and OpenCV 🙁 But I made sure the image which was feeding into detectMarkersfunction was proper(i.e not flipped) by imshow.
b) I have also changed from aruCo dictionary to April tags so that there is no version mismatch in the tags.
c) I also checked by loading the image using imread only in dll . This works and the tag gets detected.
d) In addition , I also tried sending single image data (using Texture2D as image type).Here the Unity crashes during cvtColor line.

So I am sure I am doing something wrong in transferring data from Unity to dll. Can anyone see where I am going wrong? Any help or suggestion would be appreciated. I have spent a lot of time on this already and going nowhere.

Part of Unity code snippet:

private WebCamTexture CamTexture;

 void Start()
 {
   CamTexture = new WebCamTexture();
   CamTexture.Play();
 }

 void Update()
 {
   if (CamTexture.isPlaying)
     {
       var rawImg = CamTexture.GetPixels32();
       MarkersDetection.detectAruco(ref ArucoID, ref arrayLength, ref rawImg, WebcamWidth, WebcamHeight);
     }
}

Part of OpenCV code snippet:

struct Color32
{
    uchar red;
    uchar green;
    uchar blue;
    uchar alpha;
};

extern "C" void __declspec(dllexport) __stdcall detectAruco(int& outArucoID,int& arraySize,Color32 **rawImg,int width,int height) {
cv::Mat imageCopy;
cv::Mat image(height, width, CV_8UC4, *rawImg);

cvtColor(image, imageCopy, COLOR_BGRA2BGR); 
cv::aruco::detectMarkers(imageCopy, arucoDictionary, arucoCorners, arucoIds, parameters, rejectedCandidates);
}

Source: Windows Questions C++

LEAVE A COMMENT