Quaternion-based camera with unintentional roll

Question

I have a problem with creating Camera class (C++, Eigen, Column-major, Right-handed) that uses a single quaternion for its rotation. Most of this class functionality works but not the rotations. I'm trying to make it behave like in FPS games but I still get the unintentional roll. Here's the rotation function I use:

void Camera::addLocalRotation(const Vec3f& rotation, bool rollEnabled) // rollEnabled has a default value of true
{
    Vec3f right = getRight();
    Vec3f up = getUp();
    Vec3f forward = getForward();

    if (rollEnabled)
    {
        m_rotation *= Quat(Eigen::AngleAxisf(rotation.z(), forward));
        m_rotation *= Quat(Eigen::AngleAxisf(rotation.x(), right));
        m_rotation *= Quat(Eigen::AngleAxisf(rotation.y(), up));
    }
    else
    {
        m_rotation *= Quat(Eigen::AngleAxisf(rotation.x(), right));
        m_rotation *= Quat(Eigen::AngleAxisf(rotation.y(), Vec3f(0.0f, 1.0f, 0.0f)));
    }

    m_rotation.normalize();

    m_updateBasis = true;
    m_updateMatrices = true;
}

Vec3f Camera::getRight() const
{
    return m_viewInv.col(0).head<3>();
}
Vec3f Camera::getUp() const
{
    return m_viewInv.col(1).head<3>();
}
Vec3f Camera::getForward() const
{
    return m_viewInv.col(2).head<3>();
}

In each frame I call update() on camera:

void Camera::update()
{
    m_rotation.normalize();

    updateBasis();
    updateMatrices();
}
void Camera::updateBasis()
{
    if (m_updateBasis)
    {
        m_updateBasis = false;

        Mat3f rotationMat = m_rotation.toRotationMatrix();

        m_viewInv.block(0, 0, 3, 3) = rotationMat;
    }
}
void Camera::updateMatrices()
{
    if (m_updateMatrices)
    {
        m_updateMatrices = false;

        updateBasis();

        m_view = m_viewInv.inverse();

        m_viewProj = m_proj * m_view;
        m_viewProjInv = m_viewInv * m_projInv;
    }
}

So basically, m_rotation (quaternion) is the one being modified when rotating the camera. Then on update, m_viewInv (inverse of view matrix or camera transformation matrix) 3x3 rotation part is set based on m_rotation. Then m_view is used and sent to the shader.

I've checked many solutions on this site for the same problem, yet none of them helped me or they were not what I'm looking for. I tried calculating m_rotation like:

m_rotation = pitchRotation * m_rotation * yawRotation // (or yawRotation * m_rotation * pitchRotation) - there's still roll.

Where is my mistake?

Or maybe there is something that I don't understand about quaternions? I don't get why there is a roll - I'm doing the pitch first - which simply cannot change the local X (Right) axis, then I'm rotating around global Y axis so the local X should stay on global XZ plane, right?

Additional information:
Camera class consists of those properties:

Mat4f m_view = Mat4f::Identity();
Mat4f m_viewInv = Mat4f::Identity();

Mat4f m_proj = Mat4f::Identity();
Mat4f m_projInv = Mat4f::Identity();

Mat4f m_viewProj = Mat4f::Identity();
Mat4f m_viewProjInv = Mat4f::Identity();

Quat m_rotation = Quat::Identity();

bool m_updateMatrices = false;
bool m_updateBasis = false;

I defined Math structs as:

using Vec3f = Eigen::Matrix<float, 3, 1, Eigen::ColMajor>;
using Vec4f = Eigen::Matrix<float, 4, 1, Eigen::ColMajor>;

using Mat3f = Eigen::Matrix<float, 3, 3, Eigen::ColMajor>;
using Mat4f = Eigen::Matrix<float, 4, 4, Eigen::ColMajor>;

using Quat = Eigen::Quaternion<float>;

I'm initializing camera with those functions:

void Camera::setPerspective(float fov, float aspect, float near, float far)
{
    m_proj = Math::createPerspectiveMatrix(fov, aspect, near, far);
    m_projInv = m_proj.inverse();

    m_updateMatrices = true;
}
void Camera::lookAt(const Vec3f& position, const Vec3f& target, const Vec3f& up)
{
    Vec3f f = (target - position).normalized();
    Vec3f r = up.cross(f).normalized();
    Vec3f u = f.cross(r);

    m_viewInv <<
        r.x(), u.x(), f.x(), position.x(),
        r.y(), u.y(), f.y(), position.y(),
        r.z(), u.z(), f.z(), position.z(),
        0.0f, 0.0f, 0.0f, 1.0f;

    m_view = m_viewInv.inverse();

    Mat3f mat = m_viewInv.block(0, 0, 3, 3);
    m_rotation = mat;
}

Mat4f createPerspectiveMatrix(float fov, float aspect, float near, float far)
{
    float ctg = 1.0f / (std::tanf(deg2rad(fov * 0.5f)));
    
    Mat4f mat;
    mat <<
           (ctg / aspect),     0.0f,     0.0f,                            0.0f,
           0.0f,               ctg,      0.0f,                            0.0f,
           0.0f,               0.0f,     (-far - near) / (near - far),    (2.0f * far * near) / (near - far),
           0.0f,               0.0f,     1.0f,                            0.0f;

    return mat;
}