I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

The problem is that I am always seeing the 'full overlap' no matter the position of the quads. I can never get that partial masking that I'm looking for.

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) { //if we have valid mask texture coordinates, we are masking
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord); //get the rgba values of the mask texture
          if(maskColor.a == 0) { //if mask texture has no alpha, discard
               discard;
          }
     }
}

Do I need to pass location data of the masking quad into the shader? If so, how do I do that?

I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

The problem is that I am always seeing the 'full overlap' no matter the position of the quads. I can never get that partial masking that I'm looking for.

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) { //if we have valid mask texture coordinates, we are masking
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord); //get the rgba values of the mask texture
          if(maskColor.a == 0) { //if mask texture has no alpha, discard
               discard;
          }
     }
}

Do I need to pass location data of the masking quad into the shader? If so, how do I do that?

EDIT: ilmale's method using the stencil buffer worked for me! All I needed to add was a uniform boolean value to tell the shader if I was drawing a mask or not.

Here's the relevant shader code:

if(isMask) { //if this is a mask, discard where texture alpha is 0
    if(texColor.a == 0) {
        discard;
    } else {
        out_Color = vec4(0, 0, 0, 0); //make the rest transparent, so we don't see the mask texture
    }
}

I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

Note that the textures and the quads involved may be of varying sizes and positions.

I think I need to compare the color of both the textures inside the fragment shader and discard the fragment if the mask texture has an alpha of 0, but I am unsure of exactly what I need to pass to the fragment shader.

  I am passing both the drawn texture and the mask texture as uniforms. For normal non-mask rendering, I'm passing a vec2 with the s,t texture coordinates. I assume I need to pass another vec2 with s,t coordinates for the mask texture too, right?

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) {
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord);
          if(maskColor.a == 0) {
               discard;
          }
     }
}

I'm passing -1 for the mask texture coordinates whenever I do not want an object to be masked. I am sure that some of the objects should have -1 for that value when I pass the vertex data into my VBO with a FloatBuffer object. However, the result I get is that all pixels are discarded, which makes it seem like incorrect data is being passed to my fragment shader.

Can anyone see what I'm doing wrong here? Or could someone show me the minimum that should be necessary to achieve this effect?

I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

The problem is that I am always seeing the 'full overlap' no matter the position of the quads. I can never get that partial masking that I'm looking for.

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) { //if we have valid mask texture coordinates, we are masking
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord); //get the rgba values of the mask texture
          if(maskColor.a == 0) { //if mask texture has no alpha, discard
               discard;
          }
     }
}

Do I need to pass location data of the masking quad into the shader? If so, how do I do that?