Metal #5 Texture

This note covers textures in Metal: how to load them, how to sample them, and the settings that look like details but will wreck an image outright.

Loading a texture

MetalKit provides MTKTextureLoader, which reads an MTLTexture out of an asset catalog or a file in a few lines:

Swiftlet loader = MTKTextureLoader(device: device)

let texture = try loader.newTexture(
    name: "brick_albedo",
    scaleFactor: 1.0,
    bundle: nil,
    options: [
        .textureUsage      : MTLTextureUsage.shaderRead.rawValue,
        .textureStorageMode: MTLStorageMode.private.rawValue,
        .generateMipmaps   : true,
        .SRGB              : true
    ]
)

All four options are worth explaining:

• textureUsage declares how the texture will be used. For a read-only texture say only .shaderRead, and the driver can optimize more aggressively. Add .renderTarget if it will be rendered into.
• textureStorageMode: .private keeps the texture in GPU memory only, inaccessible to the CPU. For assets that are never modified after loading, this is the fastest mode.
• generateMipmaps has the loader build the mip chain for you.
• SRGB is the one that deserves a section of its own.

sRGB: leave it to the hardware

Most colour textures — albedo, UI icons — are stored sRGB encoded: a non-linear curve close to gamma 2.2, which spends more of the limited 8-bit code range on the darks where the eye is sensitive.

Lighting maths, however, must happen in linear space. Multiplying an sRGB value by a light intensity is simply wrong, and produces an image that is too dark overall with muddy midtones.

The fix is not pow(color, 2.2) in the shader — it is an sRGB pixel format (.bgra8Unorm_srgb):

Swiftoptions: [.SRGB: true]

Now the hardware's texture unit converts during sampling, and crucially it does so before bilinear interpolation. That ordering matters: interpolating two colours in a non-linear space gives the wrong answer, and the hardware guarantees the right order. A hand-written pow cannot, and it pays for an exponentiation on every sample.

The output side has to agree: when the drawable's pixel format is .bgra8Unorm_srgb, the linear colour written by the fragment function is encoded back to sRGB by the hardware. The whole chain is sRGB texture → hardware decode → linear maths → hardware encode → sRGB display.

One important exception: normal maps, roughness, metallic and AO must never use an sRGB format. They store numbers, not perceived brightness, and running them through a gamma curve corrupts every value. This is one of the most common rendering bugs there is, and it shows up as skewed normal directions and materials that look too smooth or too rough.

Samplers

An MTLSamplerState describes how values are fetched from a texture:

Swiftlet descriptor = MTLSamplerDescriptor()
descriptor.minFilter = .linear
descriptor.magFilter = .linear
descriptor.mipFilter = .linear          // trilinear
descriptor.sAddressMode = .repeat
descriptor.tAddressMode = .repeat
descriptor.maxAnisotropy = 8
samplerState = device.makeSamplerState(descriptor: descriptor)

Filter mode: .nearest preserves hard edges (pixel art, palette textures, ID maps); .linear does bilinear interpolation. mipFilter decides whether mip levels are blended — .nearest leaves a visible band where the level changes, .linear is trilinear filtering.

Address mode decides what happens outside [0,1]: .repeat tiles, .clampToEdge stretches the edge texel, .mirrorRepeat tiles mirrored, .clampToZero returns transparent. For any non-tiling texture — character maps, atlases — use .clampToEdge. With .repeat, bilinear interpolation pulls texels from the opposite edge and produces a one-pixel-wide wrong border, which in an atlas shows up as neighbouring tiles bleeding into each other.

Anisotropic filtering fixes blurring at grazing angles. When a surface is nearly edge-on, the UV changes at very different rates along screen x and y; ordinary mipmapping can only pick the larger of the two, so one direction ends up over-blurred. Anisotropic filtering takes several samples along the longer axis instead. maxAnisotropy = 8 is close to mandatory on floors and walls, and it is not expensive.

Binding and use in the shader:

MSLfragment float4 fragment_main(VertexOut in [[stage_in]],
                              texture2d<float> albedoTex [[texture(0)]],
                              sampler          s         [[sampler(0)]])
{
    float4 albedo = albedoTex.sample(s, in.uv);
    return albedo;
}

A sampler can also be declared as a constant inside the shader, saving a binding:

MSLconstexpr sampler s(filter::linear, mip_filter::linear, address::repeat);

Why mipmaps are a performance feature

Mipmaps are usually presented as antialiasing, but their more important role is performance.

A distant triangle sampling a full-resolution texture reads texels that are far apart in memory for adjacent pixels, so nearly every sample is a cache miss. Texture bandwidth becomes the bottleneck immediately. Mipmaps let distant objects sample a small image, where neighbouring pixels land on the same cache line.

The cost is a third more memory (1 + 1/4 + 1/16 + … = 4/3), which is almost always worth it. Unless you have a specific reason, every texture should have mipmaps.

Besides the loader option above, they can be generated at runtime with a blit encoder:

Swiftlet blit = commandBuffer.makeBlitCommandEncoder()!
blit.generateMipmaps(for: texture)
blit.endEncoding()

Texture arrays and going bindless

Every texture switch is an encoder state change. With many objects, those switches become a CPU bottleneck.

A texture array (type2DArray) packs several textures of the same size and format into one object, selected by an integer slice index:

MSLfloat4 c = texArray.sample(s, in.uv, in.materialIndex);

An argument buffer goes further, packing a whole set of resources — textures, samplers, buffers — into one GPU-addressable structure. A shader can index arbitrary resources out of it, which is Metal's bindless story:

MSLstruct Material {
    texture2d<float> albedo;
    texture2d<float> normal;
    float            roughness;
};

fragment float4 f(constant Material *materials [[buffer(0)]],
                  uint id [[flat]] /* ... */)
{
    float4 c = materials[id].albedo.sample(s, uv);
}

With this, hundreds of differently-textured objects collapse into a handful of draw calls — and combined with indirect command buffers, into work the GPU drives by itself.

The next note covers cameras and interaction.