How to find edge pixels in shader?

Question

Is it possible to understand in the shader that pixel is on the edge of the geometry?

Or at least understand that pixel on the edge of the triangle?

Maybe ddx()/ddy() can help?

Answer 1

Possible but relatively tricky, and you gonna need a geometry shader. They are almost always available in Direct3D: the last GPU with feature level < 11.0 was Intel Sandy Bridge from 2011, computers don’t usually live that long. However, availability in GL and especially GLES varies.

Edges of Triangles

In the geometry shader, generate float2 for each vertex of the triangle, with the values [ 1, 0 ], [ 0, 1 ] and [ 0, 0 ] for the 3 vertices of the triangle.

Then in the pixel shader, use the following HLSL function for edge detection, untested:

inline bool detectEdge( in float2 bary2 )
{
    // Compute minimum of these barycentric coordinates
    // The third one is redundant because they add up to 1.0
    float minBary = min( bary2.x, bary2.y );
    minBary = min( minBary, 1.0 - ( bary2.x + bary2.y ) );
    // Compute screen-space derivative of that variable
    float w = fwidth( minBary );
    // We want to return true for 1 pixels-wide line on the edge
    // Edges are shared by 2 triangles, each triangle needs 0.5 pixel wide line
    return ( minBary * 2 ) < w;
}

Edges of Geometry

Your first question is more complicated. One possible approach, add one more vertex attribute of type DXGI_FORMAT_R8_UINT. Set these bytes to 1 for vertices on the edge of the mesh, 0 for the rest of them.

Pass these integers from vertex to geometry shader. There, you can find out which edges of the triangle are on the edge of the geometry:

inline uint outerEdgesBitmap( uint v0, uint v1, uint v2 )
{
    uint res = 0;
    if( v0 != 0 && v1 != 0 )
        res = 1;    // v0-v1 edge is external
    if( v1 != 0 && v2 != 0 )
        res |= 2;   // v1-v2 edge is external
    if( v2 != 0 && v0 != 0 )
        res |= 4;   // v2-v0 edge is external
    return res;
}

With that bitmap passed down the graphic pipeline, you can detect pixels on the outer edges of the geometry in the pixel shader:

inline bool detectEdge( in float2 bary2, uint outerBitmap )
{
    float minBary = 1.0;
    if( 0 != ( outerBitmap & 1 ) )
        minBary = bary2.x;
    if( 0 != ( outerBitmap & 2 ) )
        minBary = min( minBary, bary2.y );
    if( 0 != ( outerBitmap & 4 ) )
        minBary = min( minBary, 1.0 - ( bary2.x + bary2.y ) );

    // For open edges, we want the entire 1-pixel wide line inside the triangle
    return minBary < fwidth( minBary );
}

P.S. I haven’t tested these HLSL functions, but I have used a similar approach in the past, it works.