How to use Windows ConPTY API from a process which output has been redirected?

  c++, pty, winapi

I am working on an application which creates a Windows pseudoconsole (using the new ConPTY API).

It starts a PowerShell session (for test), and shows everything ConPTY sends from that session via the pseudoconsole, and everything works really well. Until I redirect the parent program output to anywhere.

Here’s my full code listing for reference.

#include <string>
#include <stdexcept>
#include <iostream>
#include <vector>

#include <Windows.h>

using namespace std::string_literals;

void throwLastError(int number)
{
    throw std::runtime_error("Error "s + std::to_string(number) + ": "s + std::to_string(GetLastError()));
}

STARTUPINFOEXW prepareStartupInformation(HPCON hPCon)
{
    STARTUPINFOEXW startupInfo{sizeof(STARTUPINFOEXW)};
    SIZE_T bytesRequired = 0;
    if (InitializeProcThreadAttributeList(nullptr, 1, 0, &bytesRequired))
        throw std::runtime_error("InitializeProcThreadAttributeList wasn't expected to succeed at that time.");

    const auto threadAttributeList = static_cast<LPPROC_THREAD_ATTRIBUTE_LIST>(calloc(bytesRequired, 1));
    startupInfo.lpAttributeList = threadAttributeList;
    
    if (!InitializeProcThreadAttributeList(threadAttributeList, 1, 0, &bytesRequired))
        throwLastError(4);

    if (!UpdateProcThreadAttribute(
            threadAttributeList,
            0,
            PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE,
            hPCon,
            sizeof(HPCON),
            nullptr,
            nullptr)
    )
        throwLastError(5);

    return startupInfo;
}

PROCESS_INFORMATION startProcess(STARTUPINFOEXW startupInfo, const std::wstring &commandLine)
{
    std::vector<wchar_t> cmdLineBuffer;
    for (auto x : commandLine)
        cmdLineBuffer.push_back(x);
    cmdLineBuffer.push_back(L'{$content}');
    
    PROCESS_INFORMATION processInfo{nullptr};
    if (!CreateProcessW(
        nullptr,
        cmdLineBuffer.data(),
        nullptr,
        nullptr,
        FALSE,
        EXTENDED_STARTUPINFO_PRESENT,
        nullptr,
        nullptr,
        &startupInfo.StartupInfo,
        &processInfo)
    )
        throwLastError(6);

    return processInfo;
}

int main()
{
    auto h = GetStdHandle(STD_OUTPUT_HANDLE);
    auto type = GetFileType(h);
    std::cout << "STD_OUTPUT_HANDLE " << h << " " << type << std::endl;
    
    try
    {
        HANDLE inPipeRead = nullptr, inPipeWrite = nullptr;
        if (!CreatePipe(&inPipeRead, &inPipeWrite, nullptr, 0))
            throwLastError(1);
    
        HANDLE outPipeRead = nullptr, outPipeWrite = nullptr;
        if (!CreatePipe(&outPipeRead, &outPipeWrite, nullptr, 0))
            throwLastError(2);
    
        HPCON hPCon = nullptr;
        if (CreatePseudoConsole(COORD { 80, 25 }, inPipeRead, outPipeWrite, 0, &hPCon) != S_OK)
            throwLastError(3);

        CloseHandle(inPipeRead);
        CloseHandle(outPipeWrite);
    
        auto startupInfo = prepareStartupInformation(hPCon);

        auto processInfo = startProcess(startupInfo, L"powershell.exe");
    
        std::cout << "PID: " << processInfo.dwProcessId << "nTID: " << processInfo.dwThreadId << "n";

        char buffer[1024];
        DWORD readBytes = 0;
        while (ReadFile(
            outPipeRead,
            buffer,
            sizeof buffer - 1,
            &readBytes,
            nullptr
        ))
        {
            buffer[readBytes] = 0;
            std::cout << "Read bytes: " << readBytes << "n";
            std::cout << buffer << "n";
        }
    }
    catch (const std::runtime_error &x)
    {
        std::cerr << x.what() << "n";
    }
}

So, if I start this program from terminal or from an IDE, it works (and tells how much data it has read from the PTY pipe). But if I start it with output redirected (e.g. myprogram.exe > C:Tempsomefile.txt, or even myprogram.exe | out-host in pwsh), then it stops working: the PowerShell then somehow inherits the stdin and stdout and doesn’t write anything to the PTY pipe.

For diagnostics, I’ve added output of GetFileType(GetStdHandle(STD_OUTPUT_HANDLE)), which helps to detect if the output has been redirected (since it’s not always obvious in practice). For pristine stdout, it should write 2, and other numbers for other output modes.

How can I overcome this issue? Is it possible for my program to work even if its own stdout/stderr were redirected somewhere? I thought one of the points of PTY was to create a separate isolated environment, which won’t interfere with the parent one.

Source: Windows Questions C++

LEAVE A COMMENT