Epoll wait modifies filedescriptor set

Question

I am currently building a server in c++, so I used epoll_wait to block when I don't have any request. It worked perfectly until I decided to add a pipe to the epollfd_set. I don't know if the epoll_wait can manage pipe, but since it's a file descriptor I would assume that it can. So I did it, and a weird thing happened. When epoll stop blocking I checked the epollfd_set and instead of having an array of fd like that : [] = {1,2}, I get [] = {2,2}, the first socket vanished and has been replaced by 2. I know epoll is supposed to only change the event not the fd. Could you please, tell me what is happening?

I reproduced the error in a simple main file.

#include <iostream>
#include <sys/epoll.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <sys/types.h>

bool loop;
struct epoll_event fds[255];


int main()
{
    int pipefd[2];

    //create a pipe, pipefd[0] is the read fd and pipefd[1] is the write one.

    if(pipe(pipefd) == -1)
    {
        std::cout<<"erreur"<<std::endl;
    }

    int epollfd = epoll_create1(0);
    if (epollfd == -1) {
        std::cout<<"erreur"<<std::endl;
    }

    if (fcntl(pipefd[0], F_SETFL, O_NONBLOCK) == -1) {
        std::cout<<"erreur"<<std::endl;
    }

    //adding it to the fds set

    fds[0].events  = EPOLLIN| EPOLLET;
    fds[0].data.fd  = pipefd[0];

    if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pipefd[0], &fds[0]) == -1) {
        std::cout <<"erreur"<<std::endl;
    }

    int on = 1;
    //Bind socket to adress and add some parameters

    sockaddr_in bindParams;
    bindParams.sin_family =AF_INET;
    bindParams.sin_port = htons(2000);
    int sock = socket(AF_INET,SOCK_STREAM,0);
    inet_aton("127.0.0.1",&bindParams.sin_addr);
    bind(sock,(struct sockaddr *)&bindParams,(socklen_t)sizeof(bindParams));
   
    setsockopt(sock, SOL_SOCKET,  SO_REUSEADDR,(char *)&on,(socklen_t)sizeof(on)); 
    fcntl(sock, F_SETFL, O_NONBLOCK);


    //add it to the fds set
    fds[1].events = EPOLLIN | EPOLLET;
    fds[1].data.fd = sock;
    if(epoll_ctl(epollfd, EPOLL_CTL_ADD, sock, &fds[1]) == -1)
    {
        std::cout<<"erreur"<<std::endl;
    }

   

    listen(fds[1].data.fd, 5);

    

    std::cout << "this is the first fd : " <<fds[0].data.fd<<std::endl;
    std::cout << "this is the second fd : " <<fds[1].data.fd<<std::endl;
    std::cout <<"before epollwait"<<std::endl;
    do {
        int fdr = epoll_wait(epollfd, fds, 255, -1);
        std::cout <<"after epollwait"<<std::endl;
        std::cout << "this is the first fd : " <<fds[0].data.fd<<std::endl;
        std::cout << "this is the second fd : " <<fds[1].data.fd<<std::endl;

        if(fds[0].events == EPOLLIN)
        {
            std::cout << "pipe called\n";
        }
        if(fds[1].events == EPOLLIN)
        {
            std::cout << "listener called\n";
        }
    }while(true);
    

    return 0;
};

Also to exit the epoll_wait you have to use a client like netcat sending something on port 2000. I hope this is just my understanding which is bad rather than a real bug of epoll_wait.

I hope you will be able to solve this annoying issue. Thanks in advance.

Answer 1

The shown code's usage of epoll events appears to be based on a misconception that epoll events function similar to how poll() manages its file descriptor array. This is untrue.

The shown code carefully initializes fds[0], and feeds it to EPOLL_CTL_ADD, initializes fds[1], and feeds it to EPOLL_CTL_ADD. Afterwards, the shown code calls epoll_wait, and assumes that fds[0] will have the first file descriptor's events, and fds[1] will have any second file descriptor's events.

This is not how epoll works. For starters, fds[0] can be used to set up both file descriptors that get consumed by EPOLL_CTL_ADD. EPOLL_CTL_ADD gobbles up each morsel that you feed it, and it gets stored, for safe-keeping in the kernel.

Afterwards, epoll_wait throws you events from all file descriptors in the epoll set, one event at a time.

The return value from epoll_wait tells you how many events there are in the epoll array parameter that gets passed to epoll_wait. The shown code expects there to always be two events, fds[0] will be pipefd[0]'s events, and fds[1] will be sock's events:

        if(fds[0].events == EPOLLIN)
        {
            std::cout << "pipe called\n";
        }
        if(fds[1].events == EPOLLIN)
        {
            std::cout << "listener called\n";
        }

This is wrong. epoll_wait might return 1. This means that fds[0] contains an event. That's it. fds[0].data.fd tells you which file descriptor generated an event, it can be either pipefd[0] or sock.

If epoll_wait returns 2 means that both fds[0] and fds[1] have events, and any file descriptor can appear in fds[0].data.fd and fds[1].data.fd.

At this point, you are directed to examine the "Example for suggested usage" in epoll(7)'s manual page. Feast your eyes on the example that explains to you, in code, how to do it right:

        nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);

You must capture the return value from epoll_wait. Don't ignore it. Put it in front of you, and hold it in reverence, and respect. Because you need to use it:

               for (n = 0; n < nfds; ++n) {

The return value from epoll_wait tells you how many events there are.

                   if (events[n].data.fd == listen_sock) {

And then you check each event's data.fd, which tells you which file descriptor the event is for, and then proceed accordingly.