How to make a concisely, elegantly, and human-friendly Quaternion camera?

Question

I have spent three weeks struggling with the quaternion camera!

Now I have two Implementations. One has some kind of gimbal lock issue or something like that. Another one is totally anti-human ( I feel like I am using a ZBrush camera to play a game).

the faulty one:

the zbrush and anti-human one:

key codes of that two cameras:

the faulty one:

void MouseInput(GLfloat xOffset, GLfloat yOffset, bool constrainPitch = 0) {
        xOffset *= turnSpeed;
        yOffset *= turnSpeed;
    if (glm::abs(glm::fmod(pitch, 360.0f)) < 90.0f || glm::abs(glm::fmod(pitch, 360.0f)) > 270.0f)
        yaw += xOffset;
    else
        yaw -= xOffset;

    pitch += yOffset;

    if (constrainPitch) 
    {
        ...
    }
}

void Update() {

        glm::quat qPitch = glm::angleAxis(glm::radians(-pitch), glm::vec3(1.0f, 0, 0));
        glm::quat qYaw = glm::angleAxis(glm::radians(-yaw), glm::vec3(0, 1.0f, 0));
        glm::quat qRoll = glm::angleAxis(glm::radians(roll), glm::vec3(0, 0, -1.0f));

        orientation = glm::normalize(qYaw * qPitch * qRoll);

        front = glm::normalize(orientation * glm::vec3(0, 0, -1.0f));
        up = glm::normalize(orientation * glm::vec3(0, 1.0f, 0));
        right = glm::normalize(orientation * glm::vec3(1.0f, 0, 0));

        fps = ViewMatrix();
        tps = ViewMatrixOrbital();
    }

the zbrush and anti-human one:

void MouseInput(GLfloat xOffset, GLfloat yOffset) {
        xOffset *= turnSpeed;
        yOffset *= turnSpeed;
    yaw = xOffset;
    pitch = yOffset;
}

void Update() {
        glm::quat qYaw = glm::angleAxis(glm::radians(-yaw), up);
        glm::quat qPitch = glm::angleAxis(glm::radians(-pitch), right);
        glm::quat qRoll = glm::angleAxis(glm::radians(roll), front);

        orientation = glm::normalize(qRoll * qPitch * qYaw * orientation);

        front = glm::normalize(orientation * glm::vec3(0, 0, -1.0f));
        up = glm::normalize(orientation * glm::vec3(0, 1.0f, 0));
        right = glm::normalize(orientation * glm::vec3(1.0f, 0, 0));

        fps = ViewMatrix();
        tps = ViewMatrixOrbital();

                //Clear Everything
        yaw = 0;
        pitch = 0;
        roll = 0;
    }

The view matrix functions of both of them are the same:

    glm::mat4 ViewMatrix() {
        glm::mat4 translate = glm::translate(glm::mat4(1.0f), position);
        glm::mat4 rotate = glm::mat4_cast(orientation);
        return glm::inverse(translate * rotate);
    }
glm::mat4 ViewMatrixOrbital() {
    glm::mat4 translate = glm::translate(glm::mat4(1.0f), localPosition);
    glm::mat4 aroundOnePoint = glm::translate(glm::mat4(1.0f), aroundPoint);
    glm::mat4 rotate = glm::mat4_cast(orientation);
    return glm::inverse(aroundOnePoint * rotate * translate);
}

The first one is messed up when I look directly from up to down or down to up.

The second one, I can never go back to the same position when I just drag/draw a circle with my mouse...

I really need some help to make a human-friendly, "robust", and elegantly quaternion camera(some things like the camera in the Ace Combat or EVERSPACE)...

The resources I have used:

https://stackoverflow.com/questions/49609654/quaternion-based-first-person-view-camera

http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-17-quaternions/

https://www.3dgep.com/understanding-the-view-matrix/

and of course...

https://www.youtube.com/watch?v=d4EgbgTm0Bg&ab_channel=3Blue1Brown

Sorry for my English...

Answer 1

OK, thanks for the links from DMGregory and the ideal form HolyBlackCat.

I think that I find a temporary solution.

Of course, it is not very elegant and not perfect... Any suggestions or comments are very welcome！

new Update() function

void Update(float delta) {
    glm::quat qYaw = glm::angleAxis(glm::radians(-yaw), up);
    glm::quat qPitch = glm::angleAxis(glm::radians(-pitch), right);
    glm::quat qRoll = glm::angleAxis(glm::radians(roll), front);

        orientation = glm::normalize(tempQuat * qRoll * qPitch * qYaw * orientation);

Here I add a tempQuat which is used for correcting the "roll".

        front = glm::normalize(orientation * glm::vec3(0, 0, -1.0f));
        up = glm::normalize(orientation * glm::vec3(0, 1.0f, 0));
        right = glm::normalize(orientation * glm::vec3(1.0f, 0, 0));

        auto ClosestAxis = [](glm::vec3 v) {...};

ClosestAxis is a helper function that is used to find the closest axis(like (0,1,0), actually I add all the axes and all the vectors between those axes).

        glm::vec3 UpNormal = glm::normalize(glm::cross(front, ClosestAxis(up)));
        glm::vec3 targetUp = glm::normalize(glm::cross(UpNormal, glm::cross(up, UpNormal)));
        if (!roll) 
        {
            glm::quat q = glm::rotation(up, targetUp);
            tempQuat = glm::slerp(glm::identity<glm::quat>(), q, delta * 2.0f);
        }

The first cross-product gets a normal fo a plane, the "human-friendly intuitive vertical plane".

The Next double cross-product comes from the ideal: How can you make a projection of a vector onto a plane?. It projects the up to the "human-friendly intuitive vertical plane".

Now we can get a rotation quaternion between up and targetUp(the projected one), and we slerp them. I arbitrarily give a speed (delta*2.0f) of "correcting roll".

Because Update() will be executed every frame(actually, this function does something like infinitely close to targetUp), it can be anything smaller than 1.0(it may feel uncomfortable without a delta-time, like a plain number 0.01f ).

Other codes remain the same.

        fps = ViewMatrix();
        tps = ViewMatrixOrbital();
            //Clear Everything
    yaw = 0;
    pitch = 0;
    roll = 0;
}

the ClosestAxis lambda helping function

auto ClosestAxis = [](glm::vec3 v) {
            auto axis = std::max
            (
                { 
                  glm::normalize(glm::vec3(1.0f,1.0f,0)),glm::normalize(glm::vec3(1.0f,-1.0f,0)),
                  glm::normalize(glm::vec3(-1.0f,1.0f,0)),glm::normalize(glm::vec3(-1.0f,-1.0f,0)),
                  glm::normalize(glm::vec3(1.0f,0,1.0f)),glm::normalize(glm::vec3(1.0f,0,-1.0f)),
                  glm::normalize(glm::vec3(-1.0f,0,1.0f)),glm::normalize(glm::vec3(-1.0f,0,-1.0f)),
                  glm::normalize(glm::vec3(0,1.0f,1.0f)),glm::normalize(glm::vec3(0,1.0f,-1.0f)),
                  glm::normalize(glm::vec3(0,-1.0f,1.0f)),glm::normalize(glm::vec3(0,-1.0f,-1.0f)),
                  glm::vec3{0, 0, 1.0f},glm::vec3{0, 0, -1.0f},
                  glm::vec3{1.0f, 0, 0},glm::vec3{-1.0f, 0, 0},
                  glm::vec3{0,1.0f, 0},glm::vec3{0, -1.0f, 0},
                  glm::normalize(glm::vec3(1.0f,1.0f,1.0f)),glm::normalize(glm::vec3(-1.0f,1.0f,1.0f)),
                  glm::normalize(glm::vec3(1.0f,1.0f,-1.0f)),glm::normalize(glm::vec3(-1.0f,1.0f,-1.0f)),
                  glm::normalize(glm::vec3(1.0f,-1.0f,1.0f)),glm::normalize(glm::vec3(-1.0f,-1.0f,1.0f)),
                  glm::normalize(glm::vec3(1.0f,-1.0f,-1.0f)),glm::normalize(glm::vec3(-1.0f,-1.0f,-1.0f)),
                },
                [&](glm::vec3 v1, glm::vec3 v2) {return glm::dot(v, v1) < glm::dot(v, v2); }
            );
            return axis;
        };