Why call tcsetattr() with termios->c_cflag |= PARENB, will return -1 with errno is EINVAL?

Question

I'm studying Linux TTY. And there is a phenomenon that I cannot understand:
Set PARENB into c_cflag , tcsetattr() will return -1 with errno is EINVAL.
Why PARENB can cause EINVAL? Where return -1.

I check this with GDB on Linux kernel source code (linux-5.15.30), and I noticed that tty_ioctl return 0, not -1.
So will C lib return -1? I also try to check it in glibc source code, but I did not find anywhere could return -1, in my opinion.

The PARENB means: "Enable parity generation on output and parity checking for input."
I can understand some TTY would not support it. But how does the system know it's unsupported, even if the linux kernel tty_ioctl() return 0?

My test code is:

#include <stdio.h>
#include <pty.h>
#include <termios.h>
#include <unistd.h>
#include <errno.h>

int main(int argc, char *argv[])
{
        int ret = 0;
        int m = 0;
        int s = 0;

        struct termios t;

        ret = openpty(&m, &s, NULL, NULL, NULL);
        if (0 != ret) {
                printf("Error: openpty failed\n");
                return -1;
        }

        tcgetattr(s, &t);
        printf("> termios->c_cflag = %#x\n", t.c_cflag);

        t.c_cflag |= PARENB;
        printf(">>> will tcsetattr(), c_cflag = %#x, TCSETS = %d, TCSANOW = %d\n",
                        t.c_cflag, TCSETS, TCSANOW);
        ret = tcsetattr(s, TCSANOW, &t);
        if (0 != ret) {
                printf("Error: tcsetattr failed, ret = %d, errno = %d\n", ret, errno);
        }

        close(m);
        close(s);
        return 0;
}

The result is:

gcc test_attr.c -lutil
./a.out
> termios->c_cflag = 0xbf
>>> will tcsetattr(), c_cflag = 0x1bf, TCSETS = 21506, TCSANOW = 0
Error: tcsetattr failed, ret = -1, errno = 22

I wanna know where returns -1, and how does it can know PARENB is unsupported.

Answer 1

The observed behavior is due to a Debian-specific patch (also used by Debian-derived distros such as Ubuntu) applied to the glibc sources for the libc6 packages. The patch for 2.37-7 is shown below, but this patch has been used since 2003, related to Debian bug #218181:

https://sources.debian.org/patches/glibc/2.37-7/any/local-tcsetaddr.diff/

# All lines beginning with `# DP:' are a description of the patch.
# DP: Description: tcsetattr sanity check on PARENB/CREAD/CSIZE for ptys
# DP: Related bugs: 218131
# DP: Author: Jeff Licquia <[email protected]>
# DP: Upstream status: [In CVS | Debian-Specific | Pending | Not submitted ]
# DP: Status Details: 
# DP: Date: 2003-10-29

---
 sysdeps/unix/sysv/linux/tcsetattr.c |   55 +++++++++++++++++++++++++++++++++++-
 1 file changed, 54 insertions(+), 1 deletion(-)

--- a/sysdeps/unix/sysv/linux/tcsetattr.c
+++ b/sysdeps/unix/sysv/linux/tcsetattr.c
@@ -44,7 +44,12 @@
 __tcsetattr (int fd, int optional_actions, const struct termios *termios_p)
 {
   struct __kernel_termios k_termios;
+  struct __kernel_termios k_termios_old;
   unsigned long int cmd;
+  int retval, old_retval;
+
+  /* Preserve the previous termios state if we can. */
+  old_retval = INLINE_SYSCALL (ioctl, 3, fd, TCGETS, &k_termios_old);
 
   switch (optional_actions)
     {
@@ -75,7 +80,55 @@
   memcpy (&k_termios.c_cc[0], &termios_p->c_cc[0],
      __KERNEL_NCCS * sizeof (cc_t));
 
-  return INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
+  retval = INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
+
+  /* The Linux kernel silently ignores the invalid c_cflag on pty.
+     We have to check it here, and return an error.  But if some other
+     setting was successfully changed, POSIX requires us to report
+     success. */
+  if ((retval == 0) && (old_retval == 0))
+    {
+      int save = errno;
+      retval = INLINE_SYSCALL (ioctl, 3, fd, TCGETS, &k_termios);
+      if (retval)
+   {
+     /* We cannot verify if the setting is ok. We don't return
+        an error (?). */
+     __set_errno (save);
+     retval = 0;
+   }
+      else if ((k_termios_old.c_oflag != k_termios.c_oflag) ||
+          (k_termios_old.c_lflag != k_termios.c_lflag) ||
+          (k_termios_old.c_line != k_termios.c_line) ||
+          ((k_termios_old.c_iflag | IBAUD0) != (k_termios.c_iflag | IBAUD0)))
+   {
+     /* Some other setting was successfully changed, which
+        means we should not return an error. */
+     __set_errno (save);
+     retval = 0;
+   }
+      else if ((k_termios_old.c_cflag | (PARENB & CREAD & CSIZE)) !=
+          (k_termios.c_cflag | (PARENB & CREAD & CSIZE)))
+   {
+     /* Some other c_cflag setting was successfully changed, which
+        means we should not return an error. */
+     __set_errno (save);
+     retval = 0;
+   }
+      else if ((termios_p->c_cflag & (PARENB | CREAD))
+           != (k_termios.c_cflag & (PARENB | CREAD))
+          || ((termios_p->c_cflag & CSIZE)
+          && (termios_p->c_cflag & CSIZE)
+           != (k_termios.c_cflag & CSIZE)))
+   {
+     /* It looks like the Linux kernel silently changed the
+        PARENB/CREAD/CSIZE bits in c_cflag. Report it as an
+        error. */
+     __set_errno (EINVAL);
+     retval = -1;
+   }
+    }
+   return retval;
 }
 weak_alias (__tcsetattr, tcsetattr)
 libc_hidden_def (tcsetattr)

The patched code sandwiches the TCSETS ioctl call between a pair of TCGETS ioctl calls and if all the calls were successful it does a 3-way difference between the original termios settings got from the first TCGETS, the replacement termios settings sent to TCSETS, and the final termios settings got from the second TCGETS. The patch is specifically looking for the case when changes to the PARENB, CREAD, and CSIZE bits of c_cflag have been silently ignored and no other changes have been made. The function will return an error (with errno set to EINVAL) in that case. (I am a bit puzzled by the (PARENB & CREAD & CSIZE) subexpression which it seems would always evaluate to 0. I think that is actually a harmless bug and the subexpression should be (PARENB | CREAD | CSIZE). The bug is harmless if the TTY driver either handles the requested changes to the c_cflag bits masked by (PARENB | CREAD | CSIZE) in the TCSETS ioctl call or always sets them to the to the same values for the TCGETS ioctl call.)

As described by the author of the patch, Jeff Licquia in bug 218131 message 30:

POSIX tcsetattr needs to handle three conditions correctly:

• If all changes are successful, return success (0).
• If some changes are successful and some aren't, return success.
• If no changes are successful, return error (-1, errno=EINVAL).

The problem occurs when setting certain flags (PARENB, CREAD, or one of the CSIZE parameters) on a pty. The kernel silently ignores those settings, so libc is responsible for doing the right thing.

The relevant part from the POSIX description of tcsetattr is as follows:

The tcsetattr() function shall return successfully if it was able to perform any of the requested actions, even if some of the requested actions could not be performed. It shall set all the attributes that the implementation supports as requested and leave all the attributes not supported by the implementation unchanged. If no part of the request can be honored, it shall return -1 and set errno to [EINVAL].