Metal #15 Reflection and Refraction

This note covers the routes to specular reflection and refraction, and what each of them can and cannot do.

Cube maps and image-based lighting

The basic environment reflection is a cube map: six images facing six directions, sampled with a direction vector.

MSLfloat3 r = reflect(-viewDir, normal);
float3 env = environmentMap.sample(cubeSampler, r).rgb;

Creating one in Metal:

Swiftlet d = MTLTextureDescriptor.textureCubeDescriptor(
    pixelFormat: .rgba16Float, size: 512, mipmapped: true)
d.usage = [.shaderRead, .renderTarget]

For PBR, sampling the reflection direction directly is only correct for a perfect mirror. A rough surface reflects a cone of the environment, which means the environment map has to be pre-convolved. This is the split-sum approximation of IBL, which breaks the integral into two parts:

1. A prefiltered environment map. Convolve each mip level of the cube map with a GGX distribution at a different roughness. Mip 0 is the original (roughness 0); the last level is almost entirely blurred (roughness 1). At runtime, select the mip by roughness:

MSLfloat  lod = roughness * float(prefilteredMap.get_num_mip_levels() - 1);
float3 prefiltered = prefilteredMap.sample(s, r, level(lod)).rgb;

2. A BRDF integration lookup table. A 2D table independent of any environment, indexed by NdotV and roughness, holding a scale and bias for the Fresnel term. Compute it offline once and share it across every scene:

MSLfloat2 ab = brdfLUT.sample(s, float2(NdotV, roughness)).rg;
float3 specular = prefiltered * (F0 * ab.x + ab.y);

The diffuse half uses a separately pre-integrated irradiance map — the environment integrated over the cosine-weighted hemisphere, which needs to be tiny (32×32 is plenty). It can also be stored as spherical harmonic coefficients, nine float3s, which is cheaper still.

The whole construction is a real-time approximation of what an offline renderer integrates directly. It looks good, its cost is fixed, and it is standard equipment in every PBR engine today.

Planar reflections

For a flat surface — water, a mirror — you can simply mirror the camera through the plane and render the scene again.

Swiftfunc mirrorMatrix(plane: SIMD4<Float>) -> float4x4 {
    let n = plane.xyz, d = plane.w
    return float4x4(
        SIMD4(1 - 2*n.x*n.x,    -2*n.y*n.x,    -2*n.z*n.x, 0),
        SIMD4(   -2*n.x*n.y, 1 - 2*n.y*n.y,    -2*n.z*n.y, 0),
        SIMD4(   -2*n.x*n.z,    -2*n.y*n.z, 1 - 2*n.z*n.z, 0),
        SIMD4(   -2*n.x*d,      -2*n.y*d,      -2*n.z*d,   1)
    )
}

The result is exactly correct with no approximation, at the cost of rendering the whole scene twice. Planar reflections therefore only suit a small number of planes that genuinely matter — one mirror, one lake.

Mirroring flips triangle winding, so flip setFrontFacingWinding as well or back-face culling removes everything. You also need an oblique near plane to clip geometry below the reflector, or it shows up inside the reflection.

Screen-space reflection

SSR marches rays through the already-rendered depth and colour buffers in screen space:

MSLfloat3 rayStep(float3 origin, float3 dir, texture2d<float> depthTex, constant Params &p) {
    float3 pos = origin;
    for (uint i = 0; i < p.maxSteps; i++) {
        pos += dir * p.stepSize;
        float2 uv = projectToUV(pos, p.projection);
        if (any(uv < 0.0) || any(uv > 1.0)) break;      // left the screen: miss

        float sceneDepth = linearDepth(depthTex.sample(s, uv).r, p);
        float delta = pos.z - sceneDepth;
        if (delta > 0.0 && delta < p.thickness) {
            return binarySearchRefine(pos, dir, p);      // hit: refine by bisection
        }
    }
    return float3(-1);                                   // miss
}

SSR's advantage is that it reflects dynamic objects in the scene with no extra scene rendering. Its disadvantages are structural failures that no amount of parameter tuning fixes:

1. Anything off screen cannot be reflected. The floor at your feet will not reflect the wall behind you. The standard treatment is to fade SSR out towards the screen edges and fall back to a cube map.
2. Occluded surfaces cannot be reflected. The depth buffer only records the frontmost surface; the back of an object simply is not there.
3. Thickness is a guess. That thickness parameter is an assumption about how thick the surface in the depth buffer is. Guess too small and rays pass through objects; too large and you get smeared artefacts.

So SSR is never used alone. It is always a layer on top of a cube map: use the SSR result where a ray hits, and the environment map where it misses.

SSR on rough surfaces needs many rays and gets expensive fast. The practical approach is one jittered ray per pixel plus temporal (TAA) and spatial filtering to clean up the noise.

Refraction

The refracted direction follows Snell's law, and MSL provides it directly:

MSLfloat3 refracted = refract(-viewDir, normal, eta);   // eta = n1 / n2

refract returns a zero vector under total internal reflection — going from a denser to a thinner medium past the critical angle — and that case must fall back to reflection:

MSLfloat3 t = refract(-v, n, eta);
float3 dir = (length_squared(t) < 1e-6) ? reflect(-v, n) : t;

Common indices: water 1.33, glass 1.5, diamond 2.42.

The usual real-time approximation is a screen-space grab: project the refracted direction into a screen-space offset and sample the background colour with it.

MSLfloat2 offset = refracted.xy * strength / in.position.w;   // divide by w: less offset at distance
float3 behind = sceneColor.sample(s, screenUV + offset).rgb;

This is physically wrong — it assumes the background lies on a plane — but it is entirely convincing for glass, water and heat haze, and it costs one texture sample.

Chromatic dispersion is simulated by refracting the three colour channels with slightly different indices:

MSLfloat3 refr;
refr.r = sampleRefracted(eta - dispersion).r;
refr.g = sampleRefracted(eta).g;
refr.b = sampleRefracted(eta + dispersion).b;

Three samples buy you rainbow fringing, which reads strongly on gems and prisms.

Hardware ray tracing

Everything above is an approximation, each with a specific failure mode. Metal's hardware-accelerated ray tracing addresses them directly: a reflection is a ray cast from the surface, with no screen-space limit and no baked environment map.

The cost is performance. Even with hardware BVH traversal, one reflection ray per pixel plus shading is far more expensive than a cube map fetch. The current practice is hybrid: trace the primary reflections, use IBL for rough surfaces and distant contributions, and run a denoiser to reconstruct from a sparse set of traced samples.

Ray tracing gets three full notes of its own after #16 Animation: #17, #18 and #19.

The next note is animation.