I am trying to let the user of my app rotate a 3D object drawn in the center of the screen by dragging their finger on screen. A horizontal movement on screen means rotation around a fixed Y axis, and a vertical movement means rotation around the X axis. The problem I am having is that if I just allow rotation around one axis the object rotates fine, but as soon as I introduce a second rotation the object doesn't rotate as expected.

Here is a picture of what is happening:

The blue axis represents my fixed axis. Picture the screen having this fixed blue axis. This is what I want the object to rotate in relation to. What is happening is in red.

Here's what I know:

1. The first rotation around Y (0, 1, 0) causes the model to move from the blue space (call this space A) into another space (call this space B)
2. Trying to rotate again using the vector (1, 0, 0) rotates around the x axis in space B NOT in space A which is not what I mean to do.

Here's what I tried, given what I (think) I know (leaving out the W coord for brevity):

1. First rotate around Y (0, 1, 0) using a Quaternion.
2. Convert the rotation Y Quaternion to a Matrix.
3. Multiply the Y rotation matrix by my fixed axis x Vector (1, 0, 0) to get the X axis in relation to the new space.
4. Rotate around this new X Vector using a Quaternion.

Here's the code:

private float[] rotationMatrix() {

    final float[] xAxis = {1f, 0f, 0f, 1f};
    final float[] yAxis = {0f, 1f, 0f, 1f};
    float[] rotationY = Quaternion.fromAxisAngle(yAxis, -angleX).toMatrix();

    // multiply x axis by rotationY to put it in object space
    float[] xAxisObjectSpace = new float[4];
    multiplyMV(xAxisObjectSpace, 0, rotationY, 0, xAxis, 0);

    float[] rotationX = Quaternion.fromAxisAngle(xAxisObjectSpace, -angleY).toMatrix();

    float[] rotationMatrix = new float[16];
    multiplyMM(rotationMatrix, 0, rotationY, 0, rotationX, 0);
    return rotationMatrix;
  }

This isn't working how I expect. I'm not sure if my understanding is wrong, or if something else is causing a problem. I have some other transformations I'm doing to the object besides rotation. I move the object to the center before applying rotation. I rotate it using the matrix returned from my function above, then I translate it -2 in the Z direction so I can see the object. I don't think this is messing up my rotations, but here's the code for that anyways:

private float[] getMvpMatrix() {
    // translates the object to where we can see it
    final float[] translationMatrix = new float[16];
    setIdentityM(translationMatrix, 0);
    translateM(translationMatrix, 0, translationMatrix, 0, 0f, 0f, -2);

    float[] rotationMatrix = rotationMatrix();

    // centers the object
    final float[] centeringMatrix = new float[16];
    setIdentityM(centeringMatrix, 0);
    float moveX = (extents.max[0] + extents.min[0]) / 2f;
    float moveY = (extents.max[1] + extents.min[1]) / 2f;
    float moveZ = (extents.max[2] + extents.min[2]) / 2f;
    translateM(centeringMatrix, 0, //
      -moveX, //
      -moveY, //
      -moveZ //
    );

    // apply the translations/rotations
    final float[] modelMatrix = new float[16];
    multiplyMM(modelMatrix, 0, translationMatrix, 0, rotationMatrix, 0);
    multiplyMM(modelMatrix, 0, modelMatrix, 0, centeringMatrix, 0);

    final float[] mvpMatrix = new float[16];
    multiplyMM(mvpMatrix, 0, projectionMatrix, 0, modelMatrix, 0);
    return mvpMatrix;
  }

I've been stuck on this for a few days. Help is much appreciated.