How can I read from a TAP device (via posix::stream_descriptor) simultaneously with Boost.Asio?

Question

My program should simultaneously read packets from a generated TAP device and process them. For this I use the tuntap library from LaKabane together with Boost.Asio's posix::stream_descriptor. However, since I am acting as a client and not a server, there is no option to accept packets asynchronously. The proviorical solution I have chosen is to read asynchronously again and again. However, there are two major problems with this:

1. There could be a stack overflow, as the same function is called "infinitely" often.
2. The function does not accept the packets fast enough. I have tested this with sudo ping -f ff02::1%test.

The following is my code so far:

#include <iostream>
#include <cstdlib>
#include <boost/asio.hpp>
#include <unistd.h>
#include "tun_tap.hpp"

void handle_packet([[maybe_unused]] const boost::system::error_code& error, [[maybe_unused]] std::size_t bytes_transferred, [[maybe_unused]] const std::array<char, 1520>& buffer)
{
    if (error)
    {
        std::clog << "Error in handle_packet: " << error.message() << std::endl;
        return;
    }
    std::clog << "Received packet of size: " << bytes_transferred << std::endl;
    std::clog << std::flush;

    // To something with the packet
    sleep(5);
}

void start(boost::asio::posix::stream_descriptor& tap_device)
{
    std::array<char, 1520> buffer;

    tap_device.async_read_some(boost::asio::buffer(buffer),
                                  [&](const boost::system::error_code& error, std::size_t bytes_transferred) {
                                      start(tap_device);
                                      handle_packet(error, bytes_transferred, buffer);
                                  });
}

int main() {
    try {
        boost::asio::io_context io;
        const ::size_t mtu = 1500;
        std::clog << "Create TUN device." << std::endl;
        tun_tap dev = tun_tap("test", tun_tap_mode::tap);
        std::clog << "Set MTU to " << mtu << "." << std::endl;
        dev.set_mtu(1500);
        std::clog << "Set the TUN device up." << std::endl;
        dev.up();

        boost::asio::posix::stream_descriptor tap_device(io, ::dup(dev.native_handler()));
        start(tap_device);


        io.run();
    } catch (const std::exception &e) {
        std::cerr << "Error: " << e.what() << std::endl << "Exit program.";
        ::exit(EXIT_FAILURE);
    }
    return EXIT_SUCCESS;
}

(Full source)

My question now is, how can I read from the TAP device with Boost.Asio without losing packets?

Answer 1

Your problems are not primarily related to the device specifics. They are about C++ and object lifetime in relation to asynchronous operations.

std::array<char, 1520> buffer;

tap_device.async_read_some( //
    asio::buffer(buffer), [&](boost::system::error_code const& error, std::size_t bytes_transferred) {
        start(tap_device);
        handle_packet(error, bytes_transferred, buffer);
    });

Firstly, that's not recursion. The completion handler is a continuation. It's by definition called when the async operation completed. So the new start never overlaps. Also, it doesn't execute in the stack frame of the initiating function. Instead it executes on a service thread.

However, that's also the problem here. buffer is a local variable. It's lifetime ends immediately after async_read_some is initiated, so by definition before it completes. So, when it is used inside the completion handler it has become invalid.

So, since the code is broken, let's ignore the flawed observations about speed. First, let's fix it. There are several ways, but this one seems to me to be most instructive/extensible:

#include "tun_tap.hpp"
#include <boost/asio.hpp>
#include <cstdlib>
#include <iostream>
#include <unistd.h>
namespace asio = boost::asio;
using boost::asio::posix::stream_descriptor;
using boost::system::error_code;

struct Client {
    using Handler = std::function<void(error_code, std::string)>;
    Client(asio::any_io_executor ex, tun_tap& dev, Handler handler = default_handler)
        : tap_device_(ex, ::dup(dev.native_handle()))
        , handler_(std::move(handler)) {
        start();
    }

  private:
    stream_descriptor      tap_device_;
    Handler                handler_;
    std::array<char, 1520> buffer_{};

    void start() {
        tap_device_.async_read_some( //
            asio::buffer(buffer_),   //
            [this](error_code ec, size_t bytes_transferred) {
                if (ec) {
                    std::cerr << "Error: " << ec.message() << std::endl;
                } else {
                    std::string packet(buffer_.data(), bytes_transferred);
                    start();

                    if (handler_)
                        handler_(ec, std::move(packet));
                };
            });
    }

    static void default_handler(error_code ec, std::string const& packet) {
        if (ec) {
            std::cerr << "Error: " << ec.message() << std::endl;
        } else {
            std::cout << "Received packet: " << packet.size() << " bytes" << std::endl;
        }
    }
};

int main() try {
    constexpr uint16_t mtu = 1500;
    asio::io_context io;

    tun_tap dev{"test", tun_tap_mode::tap};
    dev.set_mtu(mtu);
    dev.up();

    Client c(io.get_executor(), dev);

    io.run();
} catch (std::exception const& e) {
    std::cerr << "Error: " << e.what() << std::endl;
    return 1;
}

I also reviewed tun_tap.hpp/cpp fixing some issues and the leak:

• File tun_tap.hpp

   #ifndef TUNTAP_HPP
   #define TUNTAP_HPP
  
   #include <cstdint>
   #include <string>
  
   #include <tuntap.h>
   #include <memory>
  
   enum class tun_tap_mode { tun = 0, tap };
  
   class tun_tap {
     public:
       tun_tap(std::string const& ifname, tun_tap_mode const& mode);
       void set_ip(std::string const& ip, uint8_t netmask);
       void set_mtu(uint16_t mtu);
       void up();
       void down();
       int  native_handle();
  
     private:
       struct Destroy final {
           constexpr void operator()(device* dev) const noexcept {
               if (dev)
                   ::tuntap_destroy(dev);
           }
       };
       std::unique_ptr<device, Destroy> _device;
   };
  
   #endif
  

• File tun_tap.cpp

   #include "tun_tap.hpp"
   #include <linux/if.h>
   #include <stdexcept>
  
   tun_tap::tun_tap(std::string const& ifname, tun_tap_mode const& mode) {
       if (ifname.empty())
           throw std::invalid_argument("ifname");
       if (ifname.size() > IFNAMSIZ)
           throw std::invalid_argument("ifname");
       if (mode != tun_tap_mode::tun && mode != tun_tap_mode::tap)
           throw std::invalid_argument("tun_tap_mode");
       _device.reset(tuntap_init());
  
       int m = mode == tun_tap_mode::tun ? TUNTAP_MODE_TUNNEL : TUNTAP_MODE_ETHERNET;
       if (::tuntap_start(_device.get(), m, TUNTAP_ID_ANY))
           throw std::runtime_error("Failed to start tuntap device.");
  
       if (::tuntap_set_ifname(_device.get(), ifname.c_str()))
           throw std::runtime_error("Failed to set ifname for tuntap device.");
   }
  
   void tun_tap::up() {
       if (::tuntap_up(_device.get()))
           throw std::runtime_error("Failed to bring tuntap device up.");
   }
  
   void tun_tap::down() {
       if (::tuntap_down(_device.get()))
           throw std::runtime_error("Failed to bring tuntap device down.");
   }
  
   void tun_tap::set_mtu(uint16_t mtu) {
       if (::tuntap_set_mtu(_device.get(), mtu))
           throw std::runtime_error("Failed to set mtu for tuntap device.");
   }
  
   void tun_tap::set_ip(std::string const& ip, uint8_t netmask) {
       if (netmask > 128) // TODO FIXME? seems 32 should be the max due to ipv4
           throw std::invalid_argument("netmask");
  
       if (::tuntap_set_ip(_device.get(), ip.c_str(), netmask))
           throw std::runtime_error("Failed to set ip address for tuntap device.");
   }
  
   int tun_tap::native_handle() { //
       return ::tuntap_get_fd(_device.get());
   }
  

It works quite nicely here: