Deadlock in WPF: How to not block the GUI thread in HwndHost.BuildWindowCore?

  async-await, c++, multithreading, wpf

We have a large code base for processing and visualizing images in C++ with Qt. Now a user wants to build on that, but their code base is .NET Core 3.1 with WPF. We use PInvoke to interface the native code, and can very successfully join the projects. The few native widgets are embedded with Win32-in-WPF-HwndHost wrappers. Everything works pretty good and we’re quite happy about the mileage we’re getting!

There is just one blocker problem: The HwndHost method BuildWindowCore can sometimes hang in a deadlock. We could identify the root cause of the deadlock:

When BuildWindowCore calls into Qt to re-parent the native widget into the managed widget’s handle, we use a blocking call to ensure the re-parenting completed. However during re-parenting the widget, Qt sometimes calls DefWindowProc to pass unhandled window messages back to the parent WPF widget. Since the WPF thread is blocked with calling Qt, this is a circular blocking wait, ending in a deadlock.

While I can understand the problem, we are not knowledgeable enough about the WPF GUI thread to resolve the issue.

What we tried so far:

  • Make the call to Qt in the background (with await) but BuildWindowCore can not be an async method.
  • Move the re-parenting out of BuildWindowCore, and call it async later, but the HwndHost widget requires the re-parenting to take place in BuildWindowCore, or we get a WPF error that the native widget handle is not (yet) a child widget of the WPF widget.

I’m a bit at the end of my wit. We could make the call to native code non-blocking on C++ side, and poll for its completion in a loop in C#. But how would we give the control back to the WPF GUI thread while we poll for Qt to re-parent the widget?

In pseudo-code, I’m thinking about something like:

protected override HandleRef BuildWindowCore(HandleRef HWNDParent)

    while (!NativeCode.reParentingCompleted()) {
        // This method is async, is it clean to call it like this?

    return new HandleRef(this, NativeCode.getEmbeddedWidgetHandle());

My questions:

  • Would something like Dispatcher.Yield() in a polling loop help to keep WPF responsive?
  • How can we cleanly call this async method in the GUI thread, when BuildWindowCore can not be async?
  • Is this an "ok" solution, or are there better ways to not block the WPF GUI thread while calling HwndHost.BuildWindowCore()?

Source: Windows Questions C++