How do I make HTTPS server pool processes work with secure SSL connections?

Question

I have successfully made an HTTP server in which the main process accepts connections and hands them off to a pool of previously spawned processes to handle the requests. It uses sendmsg() with msghdr and cmsghdr to send the open socket.

I have also successfully made a single process HTTPS server using <openssl/ssl.h>'s SSL_CTX_new, SSL_new, SSL_accept, SSL_read, etc in what I believe to be the usual way.

But I can not find out how to put the two together, and I have searched for a long time. I specifically do not want to fork a new process for each and every request, but to use a pool of pre-existing processes instead. Clearly the main server process has to run with privileges (root) to access the private key, but I really really don't want the pool processes to run as root.

Obviously the main process must create the SSL* with SSL_new and perform the SSL_accept() operation. But I can not find any way to send the SSL* to the pool processes along with the socket. I know that something like this is possible because I can see Apache doing it.

I am programming in C or C++ under freeBSD, and changing that is not an option.

I am including three outlines of my existing code that show everything of significance that I have got working. The first two are for the main process and pool processes of the non-secure server,and the third is for the single-process secure server.

**Non-secure main process**

union cmsghdr_with_int
{ struct cmsghdr hdr;
  unsigned char buffer[CMSG_SPACE(sizeof(int))]; };

main()
----
   listensock = socket(AF_INET, SOCK_STREAM, 0);
   setsockopt(listensock, SOL_SOCKET, SO_REUSEADDR, (char *) & r, sizeof(r));
   struct sockaddr_in addr; ...... fill it up ......
   bind(listensock, (struct sockaddr *) & addr, sizeof(addr));
   listen(listensock, 5);
   socketpair(PF_LOCAL, SOCK_STREAM, 0, sockpair);
   for each child
     pid_t p = fork();
     if (p == 0)
       close(sockpair[0]);
       child_run();
   close(sockpair[1]);
   loop:
      ... select(listensock + 1, & fds, NULL, NULL, & tv);
      acceptsock = accept(listensock, NULL, NULL);
      ... set up cmsghdr_with_int msgctrlbuffer
      ... and msghdr msghd with iov length 1
      ... and char socketbuffer[...]
      ... and struct cmsghdr * msgctrl with SOL_SOCKET, SCM_RIGHTS, CMSG_LEN
      * (int *) CMSG_DATA(msgctrl) = acceptsock;
      sendmsg(sockpair[0], & msghd, 0);
      r = close(acceptsock);
   close(listensock);

**Non-secure pool processes**

main()
----
   loop:
      ... set up msghdr msghd with iovlen 1, CMSG_SPACE
      recvmsg(fromserver, & msghd, 0);
      msgctrl = CMSG_FIRSTHDR(& msghd);
      while (msgctrl != NULL)
         if ... CMSG_LEN, SOL_SOCKET, SCM_RIGHTS ...
            acceptsock = * (int *) CMSG_DATA(msgctrl);
            read headers
            send response
            close acceptsock

**Secure single-process server**

main()
----

  const SSL_METHOD * meth = TLS_server_method();
  sslctxt = SSL_CTX_new(meth);
  SSL_CTX_use_certificate_file(sslctxt, "...apache...xxxxx.crt", SSL_FILETYPE_PEM);
  SSL_CTX_use_PrivateKey_file(sslctxt, "...apache...xxxxx.key", SSL_FILETYPE_PEM);
  ... the usual socket-bind-listen

  while (... still serving ...)
    int sock = accept(mainsock, ... usual
    SSL * sslio = SSL_new(sslctxt);
    SSL_set_fd(sslio, sock);
    SSL_accept(sslio);
    read headers:
      SSL_read(sslio, buffer, sizeof(buffer) - 1);
    send response:
      SSL_write(sslio, s.c_str(), s.length());
    SSL_shutdown(sslio);
    SSL_free(sslio);
    close(sock);
  close(mainsock);
  SSL_CTX_free(sslctxt);

Answer 1

The data structures associated with a TCP socket are in the OS kernel and are shared when sending the socket between processes. The data structures associated with SSL are inside the user space of the process though and not shared when sending the socket between processes. Given that these data structures include pointers into the running processes there is also no supported way to somehow transfer these to another process with a different memory layout.

There are two ways to deal with the issue: handle all SSL inside the main process or handle all SSL inside the worker process - but never handle some SSL in the main process and some in the worker.

In the first case (all SSL in the main process) the main process would act like some reverse proxy which terminates the SSL connections and sends the decrypted traffic to the worker. Since these are all operations which can block, your current design of the main process does not work well with such approach.

In the second case (all SSL in the worker) the main process can create the SSL context to load the certificate and key before forking the workers, but the main process will never do the SSL handshake. Instead it will only accept the TCP socket and transfer it to the worker, which then will proceed with the SSL handshake (SSL_accept) and all the reading and writing (SSL_read, SSL_write). This way the state of the current SSL session is contained only in the worker process.