I study how Unity editor handles rotation with mouse movements. You can rotate by directly clicking on the object, but I didn't look too deeply to that rotation since it doesn't answer your question.

You can hold the circular handles to rotate on only one local axis.

It proceeds like this:

1. It creates a line from the projection of the point clicked on the handle (your mouse), and the projection of the center of the object (the pink dot)
   
   When moving perpendicularly to that line, it rotates the object. Here, cyan is positive and red negative because...
2. The positive direction of the axis (here, we are rotating around Z axis) is pointing left depending of the projection of the center of the object (the pink dot).
   
   When pointing right, the positive direction is up/right depending of the line angle, else positive is at down/left of the line.

You can easily get the direction of the axis by measuring the angle of it, which you can get by removing its Euler angles from the Euler angles of the camera.

Another approach would be to use rays and planes (They are both your best friends !) and apply the delta rotation each time the mouse moves. This way, there's no ambiguity.

I study how Unity editor handles rotation with mouse movements. You can rotate by directly clicking on the object, but I didn't look too deeply to that rotation since it doesn't answer your question.

You can hold the circular handles to rotate on only one local axis.

It proceeds like this:

1. It creates a line from the projection of the point clicked on the handle (your mouse), and the projection of the center of the object (the pink dot)
   
   When moving perpendicularly to that line, it rotates the object. Here, cyan is positive and red negative because...
2. The positive direction of the axis (here, we are rotating around Z axis) is pointing left depending of the projection of the center of the object (the pink dot).
   
   When pointing right, the positive direction is up/right depending of the line angle, else positive is at down/left of the line.

You can easily get the direction of the axis by measuring the angle of it, which you can get by removing its Euler angles from the Euler angles of the camera.
This only works when you have the Y axis of the camera pointing up.

Another approach would be to use rays and planes (They are both your best friends !) and apply the delta rotation each time the mouse moves. This way, there's no ambiguity.