Unity clamping local rotation on LookAt issues

Question

I'm new to game development, and I've been trying to code a "smooth camera" rotation that's limited within the camera's local rotation, using a smoothed cube and transform.lookAt. The code works fine when parent is rotated straight on, but begins to flicker when the parent is rotated -90 degrees, and becomes completely reversed when rotated -180 and +180.

Here's a video of the issue in play: https://streamable.com/sqzyb4

and here's the code:

[Header("References")]
public Camera cam;
public Transform cursorDelay;
public Transform cameraHandler;
[Header("Options")]
public float smoothSpeed;
public float zOffset;
[Header("Camera Restrictions")]
public float maxCameraRotationX;
public float maxCameraRotationY;
[Header("Sensitivity/Reach")]
public float cameraXSensitivity;
public float cameraYSensitivity;
private float camX;
private float camY;
private void Start()
{
    Cursor.lockState = CursorLockMode.Confined;
}
private void Update()
{
    //Takes mouse position and lerps cursorDelay with it, smoothing the camera movement.
Vector3 cursorPos = Input.mousePosition;
cursorPos.z = zOffset;
Vector3 smoothedPos = Vector3.Lerp(cursorDelay.transform.position, cam.ScreenToWorldPoint(cursorPos), smoothSpeed * Time.deltaTime);
cursorDelay.transform.position = smoothedPos;

cam.transform.LookAt(smoothedPos);

CameraRestrictions();


}
//Restricts Camera's local rotation
private void CameraRestrictions()
{
    camX = cam.transform.localRotation.x;
    camY = cam.transform.localRotation.y;
camX += Mathf.Clamp(camX, -maxCameraRotationX, maxCameraRotationX);
camY += Mathf.Clamp(camY, -maxCameraRotationY, maxCameraRotationY);

cam.transform.localRotation = Quaternion.Euler(camX * cameraXSensitivity, camY * cameraYSensitivity, 0f);


}

If anyone knows a fix or something I did wrong, please let me know.