Why does this boost::beast code hang inside a container but not outside of it in boost 1.75.0 but not boost 1.67.0?

  boost-beast, c++, containers

After upgrading from boost 1.67.0 to boost 1.75.0, we’re having problems with boost beast’s HTTP request library.

I have the following code which communicates with the Kubernetes API server to store a custom resource via the REST interface using boost beast:

#include <string>
#include <iostream>
#include <sstream>
#include <fstream> 
#include <boost/beast/core.hpp>
#include <boost/beast/version.hpp>
#include <boost/beast/http.hpp>
#include <boost/asio/connect.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <boost/asio/ssl/stream.hpp> 
#include <boost/asio/ssl/error.hpp>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>


namespace bip = boost::asio::ip;
namespace bhttp = boost::beast::http;
namespace ssl = boost::asio::ssl;

void postServiceEndpoint(std::string topicName, std::string url, 
                         std::string host, std::string port, std::string discoveryNamespace)
{
  boost::asio::io_context context;
  boost::asio::ip::tcp::resolver resolver(context);  
  ssl::context sslCtx({ssl::context::sslv23_client});
  boost::asio::ssl::stream<boost::beast::tcp_stream> sslStream(context, sslCtx);
  
 
  auto const results = resolver.resolve(host, port);
  SSL_set_tlsext_host_name(sslStream.native_handle(), host.c_str());
  boost::beast::get_lowest_layer(sslStream).connect(results);
  sslStream.handshake(ssl::stream_base::client);


  //Load the bearer token for authenticating with K8s... 
  std::ifstream t("/var/run/secrets/kubernetes.io/serviceaccount/token");
  std::string str((std::istreambuf_iterator<char>(t)), 
                  std::istreambuf_iterator<char>());
  std::string bearerToken = str; 

  std::string target = "/apis/sdsendpoints.net/v1/namespaces/" + discoveryNamespace + "sdsendpoints"; 
  //Because the endpoint hasn't been created yet, we cant use it in the target
  //string, but if we want to retrieve the endpoint later, we have to use its name
  //in the target string... Kubernetes's REST API be weird like that. 
  bhttp::request<bhttp::string_body> request(bhttp::verb::post, target, HTTPV1DOT1);
  std::cout << "DEBUG: Set host field. " << std::endl;
  request.set(bhttp::field::host, host);
  std::cout << "DEBUG: Set content type." << std::endl;
  request.set("Content-Type", "application/json");
  std::cout << "DEBUG: Set bearer token." << std::endl;
  request.set("Authorization", "Bearer " + bearerToken);
   
  boost::property_tree::ptree requestTree;
  requestTree.put("apiVersion", "sdsendpoints.net/v1");
  requestTree.put("kind", "SdsEndpoint");
  requestTree.put("metadata.name", topicName);
  requestTree.put("spec.endpointURL", url);
      
  std::stringstream jsonStream;
  boost::property_tree::write_json(jsonStream, requestTree);
  request.body() = jsonStream.str();
  request.prepare_payload();
  std::cout << "REQUEST: n" << request << std::endl;
  bhttp::write(sslStream, request);
  boost::beast::flat_buffer buffer;
  bhttp::response<bhttp::string_body> response;
  bhttp::read(sslStream, buffer, response);         
  if(response.result_int() >= 400)
  {
       std::cout << "Got failure on post endpoint: " << response.result_int() << ": " << response.result() << " : " << response.body() << std::endl;
  }
  //Cleanup the SSL socket...
  boost::system::error_code ec;
  sslStream.shutdown(ec);
  if(ec == boost::asio::error::eof)
  {
    //This is fine. I am okay with the events that are unfolding currently.
    ec.assign(0, ec.category());
  }
  if(ec)
  {
    std::cout << "Got error code: " << ec << " on socket cleanup in SSL shutdown" << std::endl;
  }
  sslStream.lowest_layer().shutdown(boost::asio::ip::tcp::socket::shutdown_both, ec);
  if(ec)
  { 
    std::cout << "Got error code: " << ec << " on socket cleanup in TCP socket shutdown." << std::endl;
  }

}

This code worked absolutely fine in boost 1.67.0, but upon upgrading to boost 1.75.0, we immediately found that the code would hang when the boost::beast request has the set() method called.

However, intriguingly, this behavior only occurs when the code is run inside of a container as part of a Kubernetes app.

Looking at our debug printouts, we only get to the "DEBUG: Set host field printout." When running the code outside of the container environment on the physical host machines, it runs fine and is able to successfully communicate across the network.

Originally, the container used the RHEL UBI minimal image 7.9. As part of our debugging efforts, we rebuilt it using the standard UBI image and found the same freezing behavior. Installing gdb into the container image and shelling into the running container to attach to the binary with the code reveals the following stack trace:

(gdb) bt
#0 0x0000000000ae0826 in name_string (this=<optimized out>) at /opt/common/boost/1.75.0/include/boost/beast/http/impl/fields.hpp:326
#1 operator() (this=<optimized out>, lhs=..., rhs=...) at /opt/common/boost/1.75.0/include/boost/beast/http/fields.hpp:140
#2 operator() (this=<optimized out>, key1=..., nonkey2=...) at /opt/common/boost/1.75.0/include/boost/intrusive/detail/tree_value_compare.hpp:175
#3 operator() (this=<optimized out>, t1=<optimized out>, t2=...) at /opt/common/boost/1.75.0/include/boost/intrusive/detail/key_nodeptr_comp.hpp:175
#4 lower_bound_loop (x=<optimized out>, y=<optimized out>, key=..., comp=...) at /opt/common/boost/include/intrusive/bstree_algorithms:2027
#5 lower_bound (header=<optimized out>, key=..., comp=...) at /opt/common/boost/1.75.0/include/boost/intrusive/bstree_algorithms.hpp:918
#6 lower_bound (this=<optimized out>, key=..., comp=...) at /opt/common/boost/1.75.0/include/boost/intrusive/bstree.hpp:333
#7 boost::beast::http::basic_fields<std::allocator<char> >::set_element (this=0x7fffec920490, e=...) at /opt/common/boost/1.75.0/include/boost/beast/http/impl/fields.hpp:1005
#8 0x00000000016a8cbc in postServiceEndpoint(std::string, std::string) ()

As far as we can tell from the gdb output, the boost::beast code is just sitting in a dead loop for some reason. The strangest part is that the code appears to be very innocuous as it’s just setting some fields inside the class. There’s no network code or any kind of weird multithreading that could lead to a race condition involved.

The code was compiled with intel icc 19.1.0.166 20191121 for both boost 1.67.0 and boost 1.75.0. The host environment is RHEL 7.9 (Maipo) as well, so the container and host are pretty much the same OS. Our container engine is CRIO 1.18.4 and we’re running kubernetes 1.21 (not that we think k8s has anything to do with the problem).

The only thing that has changed is the upgrading of boost versions, and if we revert the boost version the code runs fine again. However, because boost::json is only available starting in boost 1.75, we need to upgrade, so simply hanging back on the version isn’t really an option for us.

Has something in the API changed that might make the request’s set() method hang like this?

Source: Windows Questions C++

LEAVE A COMMENT