Collision checking in pixel shader

Question

I'm implementing kind of 3D Model tools using DirectX 11 and I need to paint the collision area between 2 meshes with Pixel-level accuracy.

So I think I should do something in pixel shader but I don't have enough experience with shader programming.

here is my approximate solution and I'm not sure if it will work or not and whether it's effective or not.

1. first model is passed to the shader through inputlayout.

2. second model is passed to the shader through structuredbuffer.

3. when pixel shader draw first model, calculate collision algorithm in pixel shader

   if it is true paint it red.

4. UAV is not required because there is no output from shader. it is used in pixel shader only.

is there any other good way? this solution have to check with all triangles in second model at every pixel.

I know there is the compute shader, but I've never use it before. if it is necessary, I will take this opportunity to study.

Answer 1

I think you should implement a screen-space approach. Here’s an approximate workflow.

0. When creating your render target, create extra depth buffer of the same size as your render target, with the same or similar format (e.g. if your depth is DXGI_FORMAT_D32_FLOAT_S8X24_UINT, you don’t need stencil, you only need DXGI_FORMAT_D32_FLOAT). Create two views of that buffer, ID3D11DepthStencilView to write, and ID3D11ShaderResourceView to read.

1. Before drawing model A, render a depth-only pass of your model B into that separate depth buffer. Don’t forget to clear the view first. Use the same vertex/geometry shaders you’d normally use. You don’t need any pixel shaders (pass nullptr into PSSetShader), and be sure to unbind any render targets, you only need the depth-stencil bound on output. Once done that, revert to your main render target view and the main depth-stencil view.

2. Now, when rendering model A, modify your pixel shader, should be something like this:

    Texture2D<float> modelB_depth: register( t0 );
   
    float4 main( float4 pos: SV_Position ) : SV_Target0
    {
        const int3 depthLoadPosition = int3( (int2)pos.xy, 0 );
        const float b_depth = modelB_depth.Load( depthLoadPosition );
        if( b_depth < pos.z )
        {
            // Depth of the model B in this pixel is less than the current position of model A.
            // This means this pixel of model A is occluded by model B.
            // Paint it in red.
        }
        else
        {
            // Paint normally
        }
    }
   

If you use MSAA, you'll need to replace Texture2D with Texture2DMS there and adjust the shader accordingly.

3. Once you’ll get it working, you’ll notice your model A is painted red even without intersections, when model B is in front of model A. If you want to fix that, you gonna need 2 of these extra depth buffers. Render your model B into both buffers. Clear first of them with 1.0 and render with D3D11_COMPARISON_LESS depth comparison, clear the second of them with 0 and render with D3D11_COMPARISON_GREATER depth comparison. Also don’t forget D3D11_CULL_FRONT when rendering into the second one. Once done, you’ll get 2 depth textures, together they tell you the range of depths of model B at each pixel. Then, when rendering model A’s triangles, read both depths and only paint red if the current pixel’s depth is within that interval. That’s not perfect still, gonna fails in some edge cases when your model B is a concave polyhedron, but performance-wise it’s very efficient. GPUs are asynchronous, the two depth-only passes of model B don’t have dependencies and will run in parallel. In general, GPUs are very fast at rasterizing triangles, as long as the shaders aren’t too complicated.

Update: Also, it’s possible to compute both min and max depth textures with a single render pass. Setup two render targets, backed by textures of type DXGI_FORMAT_R32_FLOAT. Initialize with ClearRenderTargetView() to 1.0 and 0.0. Setup a blend state for that pass that uses D3D11_BLEND_OP_MIN for the first RT and D3D11_BLEND_OP_MAX for the second one. Write a pixel shader that outputs Z component of SV_Position into both render targets. Don’t use depth buffer, render your model B into these two RTs, both front faces and back faces i.e. D3D11_CULL_NONE. Might be slightly faster because this way vertices are only transformed once.