Two obvious problems here:

• You've labelled a variable degree that's actually measured in radians, not in degrees.

• You're telling the object how much to rotate by, not giving it a target angle it should rotate to.

These can both be solved by reading the documentation for the methods you're using. It tends to show when you don't. 😉

float radians = (float)Math.atan2(mousePos.y - playerPos.y, mousePos.x - playerPos.x);

this.instance.setFromEulerAnglesRad(0, 0, radians).setTranslation(playerPos);

Two obvious problems here:

• You've labelled a variable degree that's actually measured in radians, not in degrees.

• You're telling the object how much to rotate by, not giving it a target angle it should rotate to.

These can both be solved by reading the documentation for the methods you're using. It tends to show when you don't. 😉

float radians = (float)Math.atan2(mousePos.y - playerPos.y, mousePos.x - playerPos.x);

this.instance.transform.setFromEulerAnglesRad(0, 0, radians).setTranslation(playerPos);