One of the defining characteristics of traditional pixel art is crisp, single-pixel thick outlines. When trying to achieve this effect with 3D geometry, simple outline methods like the inverse-hull method lead to outlines with varying pixel widths, which doesn’t lend well to capturing the pixel art aesthetic.

Instead, we can leverage Sobel filtering on various buffers using screen coordinates to get the nice single-pixel crisp highlights we’re after. This has quickly become the standard for 3d pixel art outlines, popularized by creators like t3ssel8r and others.

Others have already covered using the depth and normals buffers for edge detection, so I intentionally will keep that area brief in this writeup. Instead, I’ll be detailing some related techniques I use in combination with these to push the effect a bit further.

Depth-normals edge detection

This part might be a bit of recap if you’ve already looked into 3d pixel art outlines before. The idea here is to sample a neighborhood of pixels in the depth and normals textures, compare depth/normal differences, and draw outlines when these differences surpass some threshold. The idea is quite simple, but I’ve found tuning these thresholds to be the finnicky part.

Depth edge detection would look something like:

// center depth
float d_c = texture(DEPTH_TEXTURE, screenUV);

// neighbor depths
float d_t = texture(DEPTH_TEXTURE, screenUV + vec2(0,  1) * texelSize);
float d_b = texture(DEPTH_TEXTURE, screenUV + vec2(0, -1) * texelSize);
float d_l = texture(DEPTH_TEXTURE, screenUV + vec2(-1, 0) * texelSize);
float d_r = texture(DEPTH_TEXTURE, screenUV + vec2( 1, 0) * texelSize);

// compare neighbors against center
float delta_t = d_t - d_c;
float delta_b = d_b - d_c;
float delta_l = d_l - d_c;
float delta_r = d_r - d_c;

// only let the nearer side of the discontinuity draw the edge
float edge_t = step(DEPTH_THRESHOLD, abs(delta_t)) * step(delta_t, 0.0);
float edge_b = step(DEPTH_THRESHOLD, abs(delta_b)) * step(delta_b, 0.0);
float edge_l = step(DEPTH_THRESHOLD, abs(delta_l)) * step(delta_l, 0.0);
float edge_r = step(DEPTH_THRESHOLD, abs(delta_r)) * step(delta_r, 0.0);

float depth_edge = max(max(edge_t, edge_b), max(edge_l, edge_r));
float3 preview = float3(depth_edge, 0.0, 0.0);

We can visualize our depth edges in red:

Similarly, we can detect differences in the normals buffer to get interior edges. Below shows depth differences in red, and now normal differences in green:

Now, we can blend this mask into our scene. You can get creative here - for example, set different colors per mask channel so depth outlines get darkened, and interior normal edges get brightened, or use the mask to brighten/darken the original pixel’s scene color using light attenuation.

If you are doing your edge detection per-renderer instead of in a fullscreen pass, you can get really detailed with things like outline colors, intensities, etc., but for now, let’s stick with a simple light attenuation blend. We will darken our mask in shade, and brighten in light.

That blend looks like this:

This approach works well and gives pretty convincing results. However, there are some strange artifacts visible where geometry intersects; taking a look at the cubes intersecting the plane, there’s a visible bulge. That happens because the depth difference across those pixels is still too small to pass our depth threshold, so the intersection is not classified as a depth edge. Instead, it’s treated as an internal normal change, and since depth-edge ownership is ambiguous around those nearly equal depth samples, the normal edge ends up looking a bit swollen.

We could try to keep fighting this with more robust ownership checks and thresholding, but what we really want for silhouettes is a buffer that tells us when one rendered object ends and another begins, regardless of how similar their depth or surface orientation is. This is where an object ID buffer is useful.

Object ID silhouettes

An object ID buffer stores a flat identifier per renderer. When two neighboring pixels have different IDs, we know we crossed an object boundary.

The exact implementation of this buffer varies depending on the engine being used. In Unity, a custom render pass is a clean apporach. I would recommend an R16F target for scenes containing large counts of renderers in order to avoid ID collisions.

A simple way to assign IDs is to derive them from object data like world-space object position. This can work well, but has some tradeoffs:

1. dynamic objects will not have a stable ID as their world-space object position hashes differently during translation, and
2. separate renderers occupying the same position can collapse to the same ID.

In some cases, this is acceptable, but isn’t particularly robust.

A CPU-side assignemnt scheme allows for much more control. Instead of deriving object IDs implicitly, you can decide exactly how objects should be grouped. This makes it possible to treat a cluster of renderers as one silhouette object, or to split everything into distinct IDs when you want finer control.

In Godot, I handled this by forking the engine and adding explicit outline-group support on MeshInstance3D, along with a dedicated backbuffer for the object ID texture. If you are building your own renderer, this becomes much simpler conceptually. You can decide exactly when this object ID prepass runs, how the IDs are authored, how objects are grouped, and how the result is fed into later passes.

Let’s take a look at what an object ID buffer would look like:

Now, we can detect discontinuities in the object ID buffer for silhouettes. The ID buffer tells us that a boundary exists, and the depth buffer tells us which side should own it. In practice, that means we get cleaner one-pixel silhouettes, especially at contact points where depth and normal edges tend to get messy.

Things look good so far. However, if we only use object IDs, we lose internal depth differences within individual objects. To resolve this, we can augment our object ID silhouettes with our previous depth silhouettes.

We can fold these in with our existing normal edges, resulting in clean, crisp outlines. The difference is subtle, but in my opinion noticeable enough to warrant a little bit of extra render target memory.

Segment highlights

Another bit of detail we can add in is “segment” highlights. I read about this from starbi’s amazing writeup on a 3D recreation of a pixel art piece. They explained a clever approach to achieving “pixel-perfect lines on mesh surfaces”, calling it segment edge detection.

The idea is similar to the object ID buffer described above. With segment edge detection, instead of assigning a unique ID per renderer, we can author textures (I will refer to them as segment maps) containing identifiers for distinct surface segments within an area.

We can have our shaders write to another prepass, this time using the mesh’s UVs to sample the segment map, and writing the raw segment cell IDs to a segment buffer.

Just like all of our other edge detection approaches above, we can determine discontinuities in neighboring pixels’ segment cell IDs, resulting in pixel-perfect lines within our objects. In order to avoid false segment edge classification where cell IDs are reused across different objects, we also constrain segment comparisons to pixels that belong to the same object ID.

Below is our combined mask at this point:

Segment edge indents

Starbi also describes optionally “generating indented normals along the line”. This is a nice-looking effect where segment edges appear to get a sort of “bevel” along the edge. Since our bevel needs to go “along the line”, we can first inset our segment edges:

// raw segment edge values
float segment_c = SegmentMask(screenUV);
float segment_l = SegmentMask(screenUV + float2(-1,  0) * texelSize);
float segment_r = SegmentMask(screenUV + float2( 1,  0) * texelSize);
float segment_u = SegmentMask(screenUV + float2( 0,  1) * texelSize);
float segment_d = SegmentMask(screenUV + float2( 0, -1) * texelSize);

// any non-line pixel directly adjacent to a segment line becomes the inset
float inset = (1.0 - segment_c) * max(
    max(segment_l, segment_r),
    max(segment_u, segment_d)
);

// blue = segment line, magenta = inset
float3 debug = float3(inset, 0.0, max(segment_c, inset));

This will be our mask for the areas in which our indented normals will be applied. To build those normals, we can generate a helper texture for our segment map that stores the direction from each texel in a segment cell toward that cell’s center. We will be turning these directions into world-space normals later on in-shader. I chose to create two cell direction field textures; one with smooth continuous values, and one quantized into discrete directions derived from cell boundaries. I found that using a blend of these two leads to a nice effect once we apply lighting later. You can cut down on texture memory by pre-baking your blend into one texture, if you have a blend value in mind that you like and don’t plan on changing.

We now have a view-stable basis to derive our normals from, and a mask for where to apply them.

We have to convert these from local UV-space directions to world-space normals. Our texture gives us x as direction along the tangent axis, and y as our direction along the bitangent axis.

// 2d direction field from texture space
float2 indent = texture(SEGMENT_FIELD_TEXTURE, screenUV).xy * 2.0 - 1.0;

// convert normal from object to world-space
float3 normalWS = normalize((MODEL_MATRIX * float4(NORMAL, 0.0)).xyz);

// convert tangent from object to world-space
float3 tangentWS = normalize((MODEL_MATRIX * float4(TANGENT.xyz, 0.0)).xyz);

// B = cross(N, T) * handedness
float3 bitangentWS = normalize(cross(normalWS, tangentWS) * TANGENT.w);

// indent normal in world-space
float3 indentNormal = normalize(
    tangentWS   * indent.x * indentStrength +
    bitangentWS * indent.y * indentStrength +
    baseNormal * baseNormalWeight
);

Finally, we have our indented world space normals, and can use this to modulate attenuation along our segment edges. The benefit of using a blend between smooth and quantized direction field maps is that we have more control over how the light seems to “catch” along straight edges. I find that blending quantization in helps for grid-like segment maps, whereas keep continuous direction fields looks better for circular/rounded segment maps.

I like to use a primary threshold with some subtle falloff to keep the indent focused but still soft as it tails out; the falloff helps avoid popping when the light direction/surface orientation changes.

Again, we can blend this into our scene. Below shows the original segment edges darkened, with indented normal edges brightened.

During camera rotation, the indented lighting stays stable.

We can also experiment with how it looks using some point lights:

Closing remarks

Here is one more scene showing everything together: object ID silhouettes, internal depth and normals edges, segment edges, and segment indented normals.

Resources

• “Creating a Pixel Art Scene in Realtime 3D” by t3ssel8r: https://imgur.com/gallery/creating-pixel-art-scene-realtime-3d-qwhbHQq
• “Empty Spaces” by starbi: https://starbi.dev/#empty-spaces