Object Keeps Rotating with Mouse Even When No Code for Rotation is Active unity

I'm working on a Unity project where I need to rotate an object using either the mouse or the controller. However, I've run into an issue: the object keeps rotating with the mouse even when the left mouse button is not pressed, but only when both mouse and controller input methods are enabled at the same time.

Here’s a basic overview of the code:

Mouse Rotation: The object should rotate when the mouse moves, but only when the left mouse button is pressed.

Controller Rotation: The object should also rotate when the right analog stick of the controller is moved.

The Issue:

• The object rotates unintentionally with the mouse when both the mouse and controller input methods are active.
• The object should only rotate with the mouse when the left mouse button is pressed.

private void Update()
{
    if (!isInspecting || objectInspect == null) return;

    RotateWithController();
    RotateObject();
}

private void RotateObject()
{
    if (Input.GetMouseButton(0))
    {
        Vector2 mouseDelta = Mouse.current.delta.ReadValue();
        if (mouseDelta != Vector2.zero)
        {
            objectInspect.transform.Rotate(new Vector3(-mouseDelta.y, mouseDelta.x, 0) * Time.deltaTime * rotationSpeed, Space.World);
        }
    }
}

private void RotateWithController()
{
    Vector2 rotationInput = rotateItem.ReadValue<Vector2>();

    if (rotationInput != Vector2.zero)
    {
        objectInspect.transform.Rotate(new Vector3(-rotationInput.y, rotationInput.x, 0) * Time.deltaTime * rotationSpeed * 8f, Space.World);
    }
}