Rework container image rebuilds

[lbussell]

Current state

graph TD

subgraph docker-tools repo
check-base-images["check base images pipeline \n (multiple repos)"]
subscriptions[subscriptions.json]
end

subgraph consuming["consuming repo"]
build-pipeline[build-pipeline]
end

subgraph versions["versions repo"]
image-info[image-info.json]
end

subgraph acr
public["public images"]
end

check-base-images -->|reads| subscriptions
check-base-images -->|"reads (many)"| image-info
check-base-images -->|"queues (many)"| build-pipeline
subscriptions -->|"references (many)"| consuming
build-pipeline <-->|updates and reads| image-info
build-pipeline -->|pushes images| public

Loading

Problems with current setup

• Requires a separate repo to store image-info.json metadata file. Git is in the critical path for builds.
• Only the docker-tools repo can queue base image re-bulids, and the docker-tools repo must know about all other consuming repos in order to queue their builds.
• The docker-tools repo and the consuming repos can go out-of-sync. For example, if the manifest.json or image-info formats change, then the docker-tools check-base-images pipeline can fail until all consuming repos have adopted the updated image-builder version.

Desired state

Key ideas:

• Consuming repos should own their own base image update pipeline + schedule.
• The docker-tools repo should provide sane defaults/pipeline templates for base image re-builds.
• Base image re-builds should work both in public and internal scenarios without extra work.
• There should not be a dependency on external repos or storage locations for the critical base image re-build path. image-info.json, if it still needs to exist, should not be a source of truth. It should be computed separately, strictly downstream of image build pipelines.

graph TD

subgraph consuming[consuming repo]
check-base-images["check base images pipeline \n (one repo)"]
build-pipeline[build pipeline]
end

subgraph acr
public["public images \n with labels/annotations"]
end

public -->|reads| check-base-images
public -->|reads| build-pipeline
check-base-images -->|queues| build-pipeline
build-pipeline -->|pushes images| public

Loading

Future work

Here's the work that I see required to reach this desired state:

• Use image metadata (labels, annotations) to drive automated image re-builds #1602
  • Attach metadata labels to images during build - Add image labels for base image, base image digest, and source #2163
• Make rebuild decisions repo-local
  • Implement a new version of the getStaleImages command which reads these labels to determine which images are out of date.
  • Implement a new pipeline which runs the new command and queues a build for stale images.
• Backwards compatibility with existing image-info.json publishing
  • ImageBuilder should still synthesize image-info.json and write it to disk, at least for the time being.
  • For backwards compatibility, we need to provide some kind paved path for publishing image-info.json to a location like a git repo, OCI artifact, or storage account. This could be in the form of a pipeline extension point/parameter, a new imagebuilder command, a totally new pipeline/template, or some combination of those.

[lbussell]

[Triage] @mthalman and I decided not to go with labels because:

1. They are legacy. Annotations are the way forwards.
2. labels propagate to images, meaning this would be a breaking change for downstream images.

Instead, we decided to go forwards with using annotations on a referrer artifact. Using a referrer artifact has the following benefits:

• We are free to version/change its format at any time.
• We can choose to switch from annotations to arbitrary blob data in the future if we want.
• We aren't beholden to publishing image LABELs forever.

The exact artifact type is up in the air. It is probably best to create our own. Something like application/vnd.microsoft.imagebuilder.image-info.v1+json, with vnd.microsoft.imagebuilder.* namespace for annotations.