Rotating a 3D direction vector upwards with `glm::rotate` and quaternions

Question

Given the following coordinate system¹, where positive z goes towards the ceiling:

I have a glm::vec3 called dir representing a (normalized) direction between two points A and B in 3D space²:

The two points A and B happen to be on the same plane, so the z coordinate for dir is zero. Now, given an angle α, I would like to rotate dir towards the ceiling by the specified amount. As an example, if α is 45 degrees, I would like dir to point in the same x/y direction, but 45 degrees towards the ceiling³:

My original idea was to calculate the "right" vector of dir, and use that as a rotation axis. I have attempted the following:

glm::vec3 rotateVectorUpwards(const glm::vec3& input, const float aRadians)
{
    const glm::vec3 up{0.0, 0.0, 1.0};
    const glm::vec3 right = glm::cross(input, glm::normalize(up));

    glm::mat4 rotationMatrix(1); // Identity matrix
    rotationMatrix = glm::rotate(rotationMatrix, aRadians, right);
    return glm::vec3(rotationMatrix * glm::vec4(input, 1.0));
}

I would expect that invoking rotateVectorUpwards(dir, glm::radians(45)) would return a vector representing my desired new direction, but it always returns a vector with a zero z component.

I have also attempted to represent the same rotation with quaternions:

glm::quat q;
q = glm::rotate(q, aRadians, right);
return q * input;

But, again, the resulting vector always seems to have a zero z component.

What am I doing wrong?

• Am I misunderstanding what the "axis of rotation" means?
• Is my right calculation incorrect?

How can I achieve my desired result?

Answer 1

You don't need to normalize your up vector because you defined it to be a unit vector, but you do need to normalize your right vector.

However, while I am unfamiliar with glm, I suspect the problem is you are rotating the matrix (or quaternion) around your axis rather than creating a matrix/quaternion that represents a rotation around your axis. taking a quick look at the docs, it looks like you might want to use:

glm::mat4 rotationMatrix = glm::rotate(radians, right);