On x86-64, when a Linux user-mode program crashes, but there's a signal handler for that crash, does AVX extended state always get saved to the stack in XSAVE format, or can it be in XSAVEC ("compacted") format as well?
I'm wondering this so I can access the saved AVX context in my signal handler.
Specifically, ucontext_t member fpregs points to a structure containing the FXSAVE floating-point + SSE state. _libc_fpstate contains a _fpx_sw_bytes overlaid at &__glibc_reserved1[12]. This contains a magic value FP_XSTATE_MAGIC1 marking that XSTATE (AVX, etc.) context is present.
However, to get to the portion of the context containing e.g. the high halves of the ymm registers, it seems like you have to use CPUID subfunction 0x0D to get the offset for that particular XSTATE feature. If the Linux kernel writes the user-mode signal frame using XSAVEC instead of XSAVE, it would mean that the algorithm to determine the offset is different.
Do I need to support both XSAVE and XSAVEC formats for finding the AVX context in a frozen context?