Building Images with BuildKit

This guide focuses on best practices and techniques for writing Dockerfiles and building container images using Panfactum's BuildKit system. 1

The practices covered here assume basic familiarity with using Dockerfiles and building images. If you would like a tutorial from first-principles, see this guide.

Building Images

Basics

The BuildKit server (buildkitd) comes with a companion CLI called buildctl (docs) that is used to submit builds.

Panfactum wraps that CLI in a command called pf-buildkit-build that provides extra functionality:

• Turns on BuildKit if it is scaled-to-zero
• Establishes a network tunnel to the remote BuildKit servers
• Load balances across available BuildKit instances
• Authenticates with ECR in the region where BuildKit is running
• Multiplexes the build across both amd64 and arm64 builds; combines the results into a multi-platform image
• Adds a shared S3 layer cache

pf-buildkit-build passes all arguments to buildctl except:

• --repo: The ECR repo to push the image to.
• --tag: The tag to use for the generated image.
• --context: A file system path to the build context
• --file: A file system path to the Dockerfile

As an example, consider the command pf-buildkit-build --repo=example --tag=foo --context=. --file=./Dockerfile --opt target=bar.

This will generate the image <aws-account-id>.dkr.ecr.<aws-region>.amazonaws.com/example:foo as the bar target in the Dockerfile found in the current working directory with the current working directory as the build context.

Note that arguments that would normally be passed directly to a CLI like docker such docker build --target=bar must be specified with --opt such as pf-buildkit-build --opt target=bar. This is because Dockerfiles are just one of the many frontends that BuildKit can support.

Minimizing Build Context

The build context is the set of files that are sent to BuildKit before it begins executing any build commands. By default, this includes all files in the directory tree specified by --context, even if they are never referenced in a COPY or ADD stanza.

A .dockerignore (.dockerignore) file in the root of the build context can prevent certain files from being included. The .dockeringore file follows .gitignore syntax.

Setting this up correctly is critically important. Removing unnecessary files from the build context prevents local secrets from accidentally being included in your builds and also improves overall build performance, especially since the build context must be transferred over the network to BuildKit.

For these reasons, we recommend an ignore-by-default approach:

# Ignore-by-default
*
# Example of including a specific file or directory
!some/path

Multi-platform Builds

As we strongly encourage using both arm64 and amd64 instances to run workloads in your Kubernetes cluster to dramatically reduce runtime costs, pf-buildkit-build builds multi-platform images.

This requires some additional care:

• Your base images must also be multi-platform. To verify this, run podman manifest inspect <image> and examine the results:

  {
      "schemaVersion": 2,
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "manifests": [
          {
              "platform": {
                  "architecture": "amd64",
              }
          },
          {
              "platform": {
                  "architecture": "arm64",
              }
          }
      ]
  }
  

  You must see that the manifests array contains support for at minimum the amd64 and arm64 platforms in the platform.architecture field.

• Your build commands must be multi-platform. For example, you cannot simply download binaries for one cpu architecture inside your image; you must parameterize these commands based on the current platform. See these docs for information on how to do this.

Build-time Secrets

Occasionally, you may need to pass in a secret and make it available to a RUN stanza. For example, perhaps you need to authenticate to a private package registry.

Do NOT use build arguments. Build arguments are included in the image and thus will be available to anyone who can pull the image or access a container running the image. Never add secrets to your images.

Instead, use build secrets. builctl has top-level support for secrets, so you would pass one in as follows: pf-buildkit-build --secret id=some-secret,src=$HOME/.github/token. Note that you can also source secrets from environment variables.

Authenticating with Private Registries

BuildKit uses the client's credentials to authenticate with registries; it does not maintain any credentials itself.

By default, we automatically authenticate your local client with AWS ECR, but you may need to pull base images from other private registries.

buildctl will use the credentials at $DOCKER_CONFIG/config.json. We set up devenv to use a repo-local file at <buildkit_dir>/config.json, and it will look something like this:

{
	"auths": {
		"ghcr.io": {
			"auth": "some auth token"
		}
	},
	"credHelpers": {
		"<buildkit-account-id>.dkr.ecr.<buildkit-region>.amazonaws.com": "panfactum",
		"public.ecr.aws": "panfactum"
	}
}

You can add additional auths either manually or by running login on your favorite container tool (e.g., podman login ghcr.io --username test --password $GITHUB_TOKEN). Note that auth is the base64 encoding of <username>:<password>.

Working with buildctl Directly

If pf-buildkit-build does not meet your needs, you can always use buildctl directly. However, it is your responsibility to take care of managing and connecting to the BuildKit instances.

Here are some helpful starter tips:

• Use pf-buildkit-tunnel to establish a remote tunnel to a BuildKit instance of the specified CPU architecture.
• Set BUILDKIT_HOST to the tcp://127.0.0.1:<local-tunnel-port>.
• Use manifest-tool to create the multi-platform manifest if desired. This comes included in the devenv.

Clearing the Cache

Coming soon!

Dockerfile Recommendations

Below are recommendations for your Dockerfiles to optimize the performance of your image builds and utility of the resulting images.

Minimal Base Images

While minimizing the size of your base images will improve the performance of your system and reduce storage costs, the most important reason for choosing minimal images is security.

If an attacker gains access to a container, they will be able to exploit all the utilities included in the image. Ideally, there should be no extra utilities to aid in furthering an exploit. This includes utilities like Bash interpreters that can be used to create interactive shells.

We recommend using the Google's hardned "Distroless" images as base images whenever possible.

Layer Ordering

Every stanza (e.g., RUN, COPY, etc.) in a Dockerfile generates a new image layer. BuildKit will cache layers but will rebuild them if any previous layer has changed. As a result, you should organize your stanzas so that they are ordered by the frequency they will be updated. Note that the COPY and ADD layers are "changed" when the files they include are changed.

Consolidate Layers

Every time you define a new layer (via a new stanza), the final state of the layer will always be added to the final image. You should choose your layers intentionally and avoid creating layers that introduce inefficiencies.

For example:

FROM ubuntu
RUN apt-get update -y
RUN apt-get install -y
RUN apt-get clean -y

This Dockerfile attempts to remove the package manager cache that is not necessary in the final image. However, it is still included since each command generates its own unique layer. To optimize this, all three commands should be included in the same layer:

FROM ubuntu
RUN apt-get update -y &&  apt-get install -y && apt-get clean -y

Multi-stage Builds

A multi-stage build enables a single Dockerfile to define multiple images (stages). While every stage is executed serially, multiple staged may be built concurrently. As a result, if it is possible to parallelize build steps, you should split them into intermediate stages to maximize the use of available resources on the BuildKit server.

Cache Mounts

RUN stanzas in Dockerfiles can include mounts. Mounts are files or directories that are added to the image **only for the duration of executing that RUN command.

Cache mounts provide the ability to load save a persistent build cache between image builds without having the cache included in the final image. This can dramatically improve the performance of commands that support caching (many build processes).

The cache mount is stored directly on the BuildKit server so mounting it is nearly instant regardless of its size. You should endeavor to make use of it whenever possible.

Pin Base Images

Image tags can be updated over time and can make your build non-reproducible. For example, ubuntu/nginx:latest may point to one image today but another image tomorrow. This can be a vector for security issues or unexpected bugs.

To combat this, you can pin the images you reference in FROM stanzas using its sha256 digest. To get a digest, run podman image inspect <image> | jq -r .[0].Digest (you may need to pull it first).

You can pin your image references like this:

FROM ubuntu/nginx:latest@sha256:391f340899edc4ff0aebf5eae11910b81aa4eb5490d76de09f32877c9c8b7283

This ensures you will always get the exact same image.

Add Labels

Labels can be picked up by observability tooling to provide insights into your running workloads. Specifically, it is best practice to add these industry-standard labels that will automatically be detected by most tools.

Remember to add labels that frequently change at the end of your Dockerfiles to not invalidate the layer cache.

Setting the User

While build commands executed using RUN stanzas can be executed as the root user for ease of use, you should not allow root to be the default user of the resulting container image.

At the end of your Dockerfiles, ensure you explicitly set the user and don't forget to change ownership of files needed to runtime to the new user.

Environment Variables

While you can set environment variables via ENV stanzas, this is only appropriate for variables that will never change during the lifetime of the image (e.g., application version).

Other environment variables should be explicitly set by the system that runs the images in order to prevent confusion about the source of truth for configuration settings.

Extra Resources

For additional recommendation, you can use these resources:

• Docker's Best Practice Guide
• Red Hat's Dockerfile Security Tips
• Consider running a Dockerfile linter to catch potential problems