How to read arbitrary len bytes using helper bpf_skb_load_bytes()?

Question

Background

I am currently trying to build an eBPF program to read the content of application data (of arbitrary length) transmitted through two processes.

I tried intercepting packets with Linux TC (like it is done in example tc.bpf.c of this repo: https://github.com/libbpf/libbpf-bootstrap) Also tried intercepting packets with Socket Filters (like it is done in example sockfilter.bpf.c of this repo: https://github.com/libbpf/libbpf-bootstrap)

Problem

When reading from bpf_skb_load_bytes(), I need to provide an argument len (the last argument). Given variable length messages, that can be smaller than the MAX_BUF_SIZE, for bpf_skb_load_bytes() not to return an error I need to specify the exact size of the payload that should be read.

Question

Can anyone help me understand how to read the amount of bytes that are in the packet's payload? Passing to bpf_skb_load_bytes an arbitrary len argument, computed as the exact number of those bytes?

Attempt 1 to solve the problem

In case it is MAX_BUF_SIZE and the payload size happens to be smaller than MAX_BUF_SIZE, error -14 is returned by bpf_skb_load_bytes; and no information is read. (I also tried this using sockfilter example on this repo: https://github.com/libbpf/libbpf-bootstrap and it is exactly what happens if the payload simply having an integer is intercepted).

This takes us to the necessity of computing the size of the payload to read the correct number of bytes from the payload.

Attempt 2 to solve the problem

In case I try to compute the payload_size from skb->len - total_hlen, I have problems with the verifier, which states that payload_len has a negative min value. total_hlen is the total size in bytes used by the eth header (eth_hlen), IP header (ip_hlen) and transport header (tp_hlen). Example in a program below:

    __u32 eth_hlen, ip_hlen, tp_hlen, data_len, total_hlen, payload_len;
    char payload[MAX_PAYLOAD_SIZE];
    ...
    total_hlen = eth_hlen + ip_hlen + tp_hlen;

    if (skb->len > total_hlen) {
        payload_len = skb->len - total_hlen;
        if (payload_len > sizeof(payload)) payload_len = sizeof(payload);

        long err = bpf_skb_load_bytes(skb, total_hlen, payload, payload_len);
        if (!err) {
            bpf_printk("New packet -> skb_len: %d, data_len: %d | Total_Hlen: %d ETH_Hlen: %d, IP_Hlen: %d, TP_Hlen: %d | payload_len: %d, payload: %s",
                skb->len, data_len, total_hlen, eth_hlen, ip_hlen, tp_hlen, payload_len, payload);
        } else bpf_printk("%d\n", err);
    }

Answer 1

TL;DR. The verifier is unable to infer that payload_len is never negative (in its signed representation). You can probably work around that with one more check to tell it payload_len is a positive value.

---

Explanation

The code that the verifier is unable to track is:

if (skb->len > total_hlen) {
    payload_len = skb->len - total_hlen;

The verifier cannot hold constraints between variables such as a > b, except for the skb->data_end special case. Instead, the verifier only hold simpler constraints such as a > 0.

That verifier limitation means that when it reaches the second line, it doesn't remember that skb->len > total_hlen. Hence, it cannot compute the proper minimum bound for payload_len and ends up thinking negative values are possible.

---

Proposed Solution

__u32 eth_hlen, ip_hlen, tp_hlen, data_len, total_hlen;
__s32 payload_len;
char payload[MAX_PAYLOAD_SIZE];
...
total_hlen = eth_hlen + ip_hlen + tp_hlen;

if (skb->len > total_hlen) {
    payload_len = skb->len - total_hlen;
    if (payload_len > sizeof(payload)) payload_len = sizeof(payload);
    
    /* NEW CHECK */
    if (payload_len <= 0) return;

    long err = bpf_skb_load_bytes(skb, total_hlen, payload, payload_len);
    if (!err) {
        bpf_printk("New packet -> skb_len: %d, data_len: %d | Total_Hlen: %d ETH_Hlen: %d, IP_Hlen: %d, TP_Hlen: %d | payload_len: %d, payload: %s",
            skb->len, data_len, total_hlen, eth_hlen, ip_hlen, tp_hlen, payload_len, payload);
    } else bpf_printk("%d\n", err);
}

Note I made payload_len signed so you may need to cast it when calling bpf_skb_load_bytes.

Answer 2

This is a follow-up from the discussion with @pchaigno, so I can share the verifier output:

I tried this, and even other checks, but nothing seemed to work. As a workaround for this, I managed to access the packet's content by:

1. First using bpf_skb_pull_data
2. Direct packet access and iterating over the data, copying its content to a char[] and null terminating it.

I wouldn't mind though trying to understand why this smin=-134 in the verifier persists.