(skip to edit for the actual problem)

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

Edit:

As DMGregory pointed out, this is a job for stencil buffer.

What I thought of now is to check if the pixel is inside one of the triangles (not sure how fast that is), using for example this code from this question, then write that result to stencil buffer and filter out all points that aren't in that triangle. Then merge the result with the map 'background'.

I know how to merge them (at least I think so) and how to filter out pixels based on their stencil buffer value (direct example in docs).

What I need now is how to set screen shader, pass arguments to it and code the shader itself (transpile the triangle check code into shaderlab, write to the stencil buffer based on that).

(skip to edit for the actual problem)

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

Edit:

As DMGregory pointed out, this is a job for stencil buffer.

What I thought of now is to check if the pixel is inside one of the triangles (not sure how fast that is), using for example this code from this question, then write that result to stencil buffer and filter out all points that aren't in that triangle. Then merge the result with the map 'background'.

I know how to merge them (at least I think so) and how to filter out pixels based on their stencil buffer value (direct example in docs).

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

(skip to edit for the actual problem)

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

Edit:

As DMGregory pointed out, this is a job for stencil buffer.

What I thought of now is to check if the pixel is inside one of the triangles (not sure how fast that is), using for example this code from this question, then write that result to stencil buffer and filter out all points that aren't in that triangle. Then merge the result with the map 'background'.

I know how to merge them (at least I think so) and how to filter out pixels based on their stencil buffer value (direct example in docs).

What I need now is how to set screen shader, pass arguments to it and code the shader itself (transpile the triangle check code into shaderlab, write to the stencil buffer based on that).

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page to creating a simple screen shader in unity (I couldn't find how to apply shader to camera from unity docs at all): https://www.alanzucconi.com/2015/07/08/screen-shaders-and-postprocessing-effects-in-unity3d/, but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).

I have a simple plane mesh generated via script (has arbitrary shape).

I want the shader to discard (set to black) all pixels that aren't passing through that mesh. Currently, the mesh is in MeshFilter but isn't rendered through Mesh Renderer, as I don't want the mesh itself to be visible - it's only purpose is to filter the camera output.

I could also send the mesh data to shader, but since it has arbitrary shape (and arbitrary number of vertices), I don't know how I can check if the pixel is running through that mesh or not. The mesh is made of few vertices (201) generated each fixedUpdate (currently, all the mesh vertices are changed in fixedUpdate (vertices set to empty array with fixed length, then filled). I didn't see low performance yet, so I didn't want to dive into optimizations just yet). All the vertices have same y axis (I don't care about y axis at all, so a vertical check can be omitted) and they don't overlap (I'm using simple raycasting to get visible area, something similar to this, but in 3D and the result is a plane-like mesh).

What I want is to show movable objects (like enemies) only when they're under that mesh (camera is from top, aiming down), else just show the map itself (which doesn't change), slightly darkened.

I found this page for creating a simple screen shader in Unity (I couldn't find how to apply shader to camera from Unity docs at all), but it didn't work (no errors, it just didn't show the screen with the black and white effect with intensity set to 1 (which should be fully black and white if I understand it correctly) or 0).