70 %

Mipmaps

What mipmaps are and how to make them

note

This post comes with a series of Bevy examples you can run.

Here is an image where two planes are rendered side by side. The plane on the left does not use mipmaps, while the plane on the right does.

no mipmaps on left, mipmaps on right
Figure 1: no mipmaps on left, mipmaps on right

You should be able to notice a large amount of artifacting on the left, while on the right the plane seems to fade into a more expected color.

This is one of the applications of mipmaps: reducing artifacts in rendered output.

Mipmaps

note

We’re talking about using mipmaps in a specific way, but don’t forget that like with many graphics tools, you can actually put any data that can fit in these images.

Mipmaps are typically when you take an image and you halve its size over and over until you have a 1 pixel image. This means a 2048x2048 pixel image will result in having 12 mips, starting at 2048x2048 and going down to 1x1 pixel.

2048, 1024, 512, 256, 128, 64, 32, 16, 8, 4, 2, 1,

They key aspect of this process is that as you downscale an image, there is a choice being made. That choice is which pixels from the higher resolution image merge into a specific pixel in the lower resolution image. As a consequence of this choice, this means you can get a pre-baked answer to “if I sampled a bunch of pixels in this area, what color would that be?”.

This becomes useful when one screen pixel covers multiple texture pixels on a far away or heavily rotated surface because you can sample one of the smaller mips instead of many samples of the full resolution texture.

There are many algorithms for performing this downscaling. At the time this sentence was written, the ktx tools support the list shown here, and defaults to lanczos4. The actual options here are less relevant than the point that you can decide how to create mipmaps.

box, tent, bell, b-spline, mitchell, blackman, lanczos3, lanczos4, lanczos6, lanczos12, kaiser, gaussian, catmullrom, quadratic_interp, quadratic_approx, quadratic_mi

Manual mipmaps

Let’s dive in to how mips show up when used. We can create a series of images that have the proper dimensions and use them as mips individually. The color of each successive image will be a totally unique color, which will clearly show which mip has been chosen for a specific area of our mesh.

5 mips visualized side-by-side
Figure 2: 5 mips visualized side-by-side

In Bevy, we can load these individual images using the AssetServer.

let manual_mips = ManualMips {
    x128: asset_server.load("128.png"),
    x256: asset_server.load("256.png"),
    x512: asset_server.load("512.png"),
    x1024: asset_server.load("1024.png"),
    x2048: asset_server.load("2048.png"),
};

Once loaded, we can take the data for each image and append them to each other. In this case I’m appending the image data to the end of the 2048x2048 px sized image, then setting the mip_level_count to 5 to account for the 5 images that are now contained in this image.

let mut new_image = mip_2048.as_bytes().to_vec();
new_image.append(&mut mip_1024.as_bytes().to_vec());
new_image.append(&mut mip_512.as_bytes().to_vec());
new_image.append(&mut mip_256.as_bytes().to_vec());
new_image.append(&mut mip_128.as_bytes().to_vec());

x2048.texture_descriptor.mip_level_count = 5;
x2048.data = Some(new_image);

Using this newly created image that has mips as a texture results in a visualization of how the mips are chosen for a plane. Notice how the red 2048x2048 sized image is closest to the camera, and as the plane is further from the camera, the lower mips are chosen.

This is not directly related to “distance from camera” alone, but also the angle of the plane. Multiple factors contribute to how many texture pixels are at a specific screen pixel location.

manually implemented mips
Figure 3: manually implemented mips

The mipmaps here are filtered linearly, which results in smooth gradients between mips. This doesn’t have to be true and we can use a nearest filtering mode instead.

let sampler = x2048.sampler.get_or_init_descriptor();
sampler.mipmap_filter = bevy::image::ImageFilterMode::Nearest;
nearest filtering
Figure 4: nearest filtering

This results in hard edges between the mip choices, clearly indicating where each starts and ends.

There are also more extensive treatments of how mipmap selection happens [1] for those that want more details.

Mips

In real world usage, textures represent bricks and other surfaces. Here are three mips pulled from a mip chain for a 2048x2048 pixel image of a set of tiles. Each mip has been resized to be the same size here, which shows off how mips further down the chain get “blurrier”.

note

The extracted mips shown here were extracted from a .ktx2 file that used the default ktx create --generate-mipmaps settings. These settings include a default --mipmap-wrap of clamp, when for repeating use cases a better option is wrap.

mip level 0: 2048 px
Figure 5: mip level 0: 2048 px
mip level 5: 64 px
Figure 6: mip level 5: 64 px
mip level 8: 8 px
Figure 7: mip level 8: 8 px

Derivatives with dpdy and dpdx

Often when talking about fragment shaders, they are described as operating on pixels but in actuality they operate on 2x2 pixel groups. For these quads, some of the pixels are active lanes and some are helper lanes [2] (see the “Hardware Partial Derivatives” section of this reference).

The dpdy and dpdx WGSL functions take advantage of these lanes to do what feels a bit like a “time travel” effect. Each invocation of dpdy has access to the other values in the quad that it needs to calculate the rate of change of the value passed in across the quad.

Partial derivative of e with respect to window y coordinates.

— WGSL Spec

Mipmaps, as you might guess, are closely related to the dpdy/dpdx wgsl functions. Here is an example that uses the uv y coordinate as an argument to dpdy and displays that value as a red channel color.

@fragment
fn fragment(
    mesh: VertexOutput,
) -> @location(0) vec4<f32> {
    let dy = dpdy(mesh.uv.y);
    // * 20. is arbitrary and only serves to bump the number up
    // into a range we can visualize
    return vec4(abs(dy) * 20., 0., 0., 1.);
}
dpdy applied to uv.y
Figure 8: dpdy applied to uv.y

The rate of change becomes important when we talk about Anisotropy.

Anisotropy

Mipmaps are great, but they can be blurry in certain cases because by default they treat all directions the same. This can show up as an issue when dealing with rates of change that are much higher in one direction than another.

To address this, we can use a lopsided sampling methodology: Anisotropic filtering. To do this in Bevy, we need to both set the anisotropy_clamp and ensure all filtering is set to linear.

Common anisotropy_clamp values include 1, 2, 4, 8, or 16, where 1 is basically “off” compared to 16. The rest of the values can be thought of as ratios, like “2-to-1” or “16-to-1”.

This can be set on the sampler when loading an image.

asset_server.load_with_settings(
    "floor_graph_base_color_uncompressed.ktx2",
    |settings: &mut ImageLoaderSettings| {
        let descriptor = settings.sampler.get_or_init_descriptor();
        descriptor.address_mode_u = bevy::image::ImageAddressMode::Repeat;
        descriptor.address_mode_v = bevy::image::ImageAddressMode::Repeat;
        descriptor.address_mode_w = bevy::image::ImageAddressMode::Repeat;
 
        let sampler = settings.sampler.get_or_init_descriptor();
        sampler.anisotropy_clamp = 16;
        sampler.min_filter = ImageFilterMode::Linear;
        sampler.mag_filter = ImageFilterMode::Linear;
        sampler.mipmap_filter = ImageFilterMode::Linear;
    }
)
two mipmaps; anisotropic filtering 16x on right
Figure 9: two mipmaps; anisotropic filtering 16x on right

Here are two closeups on the left and right planes. Notice how the anisotropic filtering retains more detail.

linear filtering
Figure 10: linear filtering
anisotropic filtering 16x
Figure 11: anisotropic filtering 16x

Anisotropic filtering is more expensive, and 16x is the most expensive, so rather than throw 16x on everything, use it when it matters.

Generating Mipmaps

Mipmaps are mostly generated ahead of time. This allows the most control and the highest quality processing when compared to runtime processing, which is flexible but has downsides like processing time.

Mipmaps in general increase texture sizes by 33%.

I’ve discussed kram before, alongside texture compression, in Substance Designer and KTX2. Kram will generate mipmaps for you if you’re ready to use texture compression.

kram encode -f bc7 -type 2d -srgb -zstd 0 -i input_base_color.png -o base_color.ktx2

When it comes to not using texture compression, we still need a format that can hold mipmaps. For this, ktx2 is still a reasonable choice, and ktx tools can generate these mipmaps.

ktx create --format R8G8B8A8_SRGB --generate-mipmap floor_base_color.png floor_with_mips.ktx2

Since ktx2 is a container format, it can hold compressed or uncompressed textures inside it.

If you used ktx create then the ktx2 file will be viewable in VSCode with the HDR Preview extension by mate-h, who has contributed to Bevy’s Atmosphere support.

If you used kram you will need to use kramv to view the textures since they will be compressed textures.

bevy_mod_mipmap_generator takes an approach that is very similar to the approach we used at the beginning of this post, and packs image data together after generating mips at runtime.

This plugin is intended for situations where the use of those formats is impractical (mostly prototyping/testing). With this plugin, mipmap generation happens slowly on the cpu.

— bevy_mod_mipmap_generator

It also has a list of ahead-of-time texture generation options, such as kram, ktx tools, and compressonator.

References