Edit:
When doing this in a libGDX application, under a OrthographicCamera, you need to unproject screen coordinates to camera space, then call my screenToMap function as that works on unprojected coordinates.
So at the beginning of the map's render you need to do something like this;
public Vector2 unproject(OrthographicCamera camera, Vector2 position) {
Vector3 unprojected = camera.unproject(new Vector3(position.x, position.y, 0));
return new Vector2(unprojected.x, unprojected.y);
}
private Vector2 screenToMap(Vector2 screen) {
float mapx = (screen.x / tileSize + screen.y / (tileSize * 0.5f)) * 0.5f;
float mapy = (screen.y / (tileSize * 0.5f) - (screen.x / tileSize)) * 0.5f;
return new Vector2(mapx - 0.5f, mapy + 0.5f); // -.5/+.5 because the drawing isn't aligned to the tile, it's aligned to the image
}
public void render(OrthographicCamera camera, Vector2 mousePosition) {
Vector2 mp = screenToMap(unproject(camera, mousePosition));
spriteBatch.setProjectionMatrix(camera.combined);
Fact that libGDX has a y-axis that increases as you go up on the screen doesn't matter, the unproject will fix that for you as you'll also render in the different direction.
For sake of completion, here's a link to a libGDX application of what I am explaining here: libGDX source. It will render a map and highlight the cell you're over with the mouse.
I assume you're doing this because you want to learn, because if you just want a isometric map in your game, libGDX comes with Tmx-classes that can load and render a isometric map created in for example Tiled.