I'm sure this is going to be a pretty simple answer but I can't quite figure it out on my own. Each frame, I'm getting the mouse's distance from the center of the screen (left/up is positive) and using it to change the camera's pitch and yaw.
Then, I added camera roll. Now, if the camera is rolled 45° left, moving the mouse left needs to change both the pitch AND yaw. I know I need to use math.sin and math.cos but I can't get it to work quite right.
Pitch and yaw are relative to the world, not the camera's view.
AddYaw and AddPitch are hypothetical functions that apply world yaw/pitch to the camera. So if the camera is rolled 90 degrees left (so right is world-up), applying yaw will appear to make the camera pitch "up", which is still left according to the world.
Due to the functions available to me through the game's API, I can't just apply camera-relative pitch/yaw, I'm restricted to world-relative pitch/yaw only. Basically, it boils down to applying some trigonometry to the mouse x/y input to "rotate" the pitch and yaw values added to the camera.
-- change.x = distance mouse moved left/right in this frame
-- change.y = distance mouse moved up/down in this frame
-- Fix me!
local yawChange = math.sin(rollInRads) * change.x + math.cos(rollInRads) * change.y
local pitchChange = math.cos(rollInRads) * change.y + math.sin(rollInRads) * change.x
-- Apply pitch/yaw changes to the camera
Camera:AddYaw(yawChange)
Camera:AddPitch(pitchChange)
A few examples of what it needs to return. For clarity, yaw = left positive, right negative, pitch = up positive, down negative.
Situation 1: No roll. Moving the mouse left applies positive yaw to the camera, moving the mouse up applies positive pitch. No conversion needed since camera yaw = world yaw, camera pitch = world pitch.
Situation 2: Camera is rolled 90 degrees left. Now, positive camera yaw (left) is actually negative world pitch (down). Therefore, moving the mouse left needs to apply PITCH, not yaw. And moving the mouse up needs to apply YAW, not pitch.
Situation 3: Camera is rolled 45 degrees left. Now, the camera/world axes are a bit mixed. Moving the mouse left should apply equal amounts of negative pitch AND positive yaw so the camera's view moves left relative to its own view.
