Metal #11 Deferred Rendering

This note covers deferred rendering, and the version of it that Metal can do on Apple Silicon and other APIs cannot.

Why defer

Forward rendering costs O(objects × lights). Worse, occluded fragments still run the full lighting computation before the depth test discards them — the more overdraw, the more waste.

Deferred rendering splits shading into two steps:

1. Geometry pass — write each pixel's surface attributes (albedo, normal, roughness, depth) into a set of textures, collectively the G-Buffer. Attributes only; no lighting.
2. Lighting pass — for every pixel on screen, read the attributes back and light it once.

Every pixel is shaded exactly once, independent of scene complexity. The cost becomes O(objects + lights × covered pixels), and hundreds of dynamic lights suddenly become feasible.

Laying out the G-Buffer

G-Buffer design is the central trade-off in deferred rendering: more stored data means more flexibility, and linearly more bandwidth.

A pragmatic layout:

Several decisions here are worth explaining.

Do not store world position. Position can be reconstructed from depth and screen coordinates; storing it wastes 12 bytes per pixel for nothing.

MSLfloat3 reconstructWorldPos(float2 uv, float depth, float4x4 invViewProj) {
    float4 ndc = float4(uv * 2.0 - 1.0, depth, 1.0);
    ndc.y = -ndc.y;
    float4 world = invViewProj * ndc;
    return world.xyz / world.w;
}

Octahedral-encode the normal. A unit vector has two degrees of freedom, so three components are redundant. Octahedral mapping projects the unit sphere onto a square, and two 16-bit components give far better precision than three 8-bit ones:

MSLfloat2 octEncode(float3 n) {
    n /= (abs(n.x) + abs(n.y) + abs(n.z));
    float2 e = n.xy;
    if (n.z < 0) e = (1.0 - abs(e.yx)) * sign(e);
    return e * 0.5 + 0.5;
}

The G-Buffer's store action. If the lighting pass consumes them immediately, in the same frame and the same encoder, then on an Apple GPU they can be .dontCare — which is exactly the point of the next section.

The lighting pass

The lighting pass draws a fullscreen triangle, reads the G-Buffer, and accumulates over lights.

One triangle rather than a two-triangle quad, because the quads along a quad's diagonal get shaded twice, once per triangle, producing a wasted seam. A single oversized triangle covering the screen has no such seam and needs no vertex buffer at all:

MSLvertex VertexOut fullscreen_vertex(uint vid [[vertex_id]]) {
    float2 uv = float2((vid << 1) & 2, vid & 2);
    VertexOut out;
    out.position = float4(uv * float2(2, -2) + float2(-1, 1), 0, 1);
    out.uv = uv;
    return out;
}

For point and spot lights with a finite range there is a further optimization: instead of a fullscreen draw, draw the light's bounding sphere or cone, so only pixels inside the volume are shaded. Combined with a stencil test this avoids work on pixels outside and behind the volume.

What deferred costs you

Deferred rendering is not free:

• Transparency does not work. The G-Buffer holds one surface per pixel. Transparent objects have to be drawn afterwards with forward rendering, so the engine maintains two shading code paths.
• MSAA becomes very expensive. Hardware MSAA operates during rasterization; deferred shading happens later, so doing MSAA correctly means shading per sample in the lighting pass. This is why deferred engines almost all moved to FXAA or TAA.
• Bandwidth. Writing the G-Buffer and reading it back is real memory traffic. On mobile hardware this is frequently the deciding disadvantage.
• The material model is constrained. The G-Buffer layout is fixed, so materials needing extra parameters — skin, hair, cloth — require a material ID and branching.

Apple Silicon: single-pass deferred

That last disadvantage, the bandwidth, can be almost entirely eliminated on an Apple GPU.

In the TBDR architecture, each tile's attachment data lives in on-chip memory during rendering. Metal exposes that: later draws within the same render pass can read attachment values straight out of tile memory, with no round trip through device memory.

Three things have to line up:

1. Declare the G-Buffer textures memoryless.

Swiftd.storageMode = .memoryless
d.usage = .renderTarget

They then occupy no device memory at all.

2. Put the geometry pass and the lighting pass in the same render pass descriptor, as two groups of draws, with no endEncoding() in between.

3. Have the lighting fragment function read the attachments back through [[color(n)]] inputs rather than sampling textures:

MSLstruct GBufferData {
    float4 albedo   [[color(0)]];
    float4 normal   [[color(1)]];
    float  depth    [[color(2)]];
};

fragment float4 deferred_lighting(VertexOut in [[stage_in]],
                                  GBufferData gbuffer,          // read straight from tile memory
                                  constant Light *lights [[buffer(13)]])
{
    float3 n = octDecode(gbuffer.normal.xy);
    // ... lighting ...
}

The result is that the G-Buffer never leaves the chip. The bandwidth cost drops to near zero, and the 32 MB of device memory is saved as well. This is a genuine advantage of Metal over other APIs — Vulkan's subpasses express a similar intent, but on desktop GPUs they do not actually keep the data on-chip.

Going further, tile shading lets you insert compute dispatches inside the same pass, operating directly on tile memory. Tiled light culling can be implemented this way: a tile's light list is computed and immediately consumed by that tile's lighting draw, without touching device memory at any point.

When not to defer

The trend has actually swung back. Forward+ (tiled forward) keeps forward rendering's support for transparency and MSAA while solving the many-lights problem with a compute-based culling pass. On Apple Silicon, where the hardware already works tile by tile, Forward+ falls out naturally and is often the better default.

Deferred still earns its place when there are very many lights, most of them small and dynamic (a city at night), or when you need a lot of screen-space effects — SSAO, SSR, screen-space shadows — since those want a depth and normal buffer anyway, and the G-Buffer hands you one.

The next note covers particle systems, and uses compute shaders for simulation for the first time.