Skip to content

Make Renovate upgrade base images - #746

Open
erikgb wants to merge 1 commit into
cert-manager:mainfrom
erikgb:reno-upgrade-base-images
Open

erikgb wants to merge 1 commit into
cert-manager:mainfrom
erikgb:reno-upgrade-base-images

Conversation

@erikgb

@erikgb erikgb commented Sep 13, 2026

Copy link
Copy Markdown
Member

Instead of a custom script and GH workflow + Makefile plumbing.

This change was motivated by failing workflows in my fork of the project, but instead of limiting the workflow run, I think this is better.

@cert-manager-prow cert-manager-prow Bot added the dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. label Sep 13, 2026
@cert-manager-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wallrj for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cert-manager-prow cert-manager-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 13, 2026
@erikgb
erikgb force-pushed the reno-upgrade-base-images branch from 65e24a8 to 6f2c850 Compare September 13, 2026 09:29
@cert-manager-prow cert-manager-prow Bot added dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. and removed dco-signoff: no Indicates that at least one commit in this pull request is missing the DCO sign-off message. labels Sep 13, 2026
@erikgb
erikgb requested a lite review from Copilot September 13, 2026 09:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Renovate does not scan modules/oci-build, and the retained upgrade script does not support the new image reference format.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Moves base-image upgrades from a custom workflow and Make target to Renovate-managed dependencies.

Changes:

  • Adds Renovate annotations and digest-pinned image references.
  • Removes the scheduled workflow and obsolete Makefile upgrade plumbing.
File summaries
File Summary
modules/oci-build/00_mod.mk Adds Renovate metadata. Moderate findings: the path is not scanned by the current manager (3 votes), and the retained script does not support the new reference format (1 vote).
Makefile Removes the obsolete upgrade target and help entry.
.github/workflows/base-images-upgrade.yaml Removes the scheduled upgrade workflow.
Review details

Suppressed comments (1)

modules/oci-build/00_mod.mk:17

  • This new :latest@digest form no longer matches the still-tracked scripts/upgrade_base_images.sh, whose replacement pattern searches for quay.io/jetstack/base-static@... (and the CSI equivalent). Running that script now succeeds without changing these lines, so either remove the retired script as part of this migration or update it for the new reference format.
base_image_static := quay.io/jetstack/base-static:latest@sha256:dc719ea428e9b88843862bd8de896bbe974088bc838d000a33f091a835744c33
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread modules/oci-build/00_mod.mk Outdated
@erikgb
erikgb force-pushed the reno-upgrade-base-images branch from 6f2c850 to 9af9927 Compare September 13, 2026 09:39
@erikgb
erikgb requested a lite review from Copilot September 13, 2026 09:40
@erikgb
erikgb force-pushed the reno-upgrade-base-images branch from 9af9927 to 91a8545 Compare September 13, 2026 09:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Renovate matching and the obsolete upgrade script need correction.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

.github/renovate.json5:10

  • Widening this manager now makes it discover the two new Docker dependencies, but the existing match strings capture the entire latest@sha256:... as currentValue and never capture currentDigest. The Docker datasource therefore has no separate digest to compare, so Renovate may not create digest updates and the removed daily workflow is not actually replaced. Add a Docker-specific match (or split the generic one) that captures latest as currentValue and sha256:... as currentDigest.
        'modules/**/*.mk',

Makefile:91

  • Removing the target and workflow leaves scripts/upgrade_base_images.sh with no remaining callers. That script is now stale as well: its replacement pattern expects image@..., while these values use image:latest@..., so any direct invocation silently fails to update them. Since this PR replaces that custom mechanism, delete the unused script too (or keep the target and update the script if it is still intended to be supported).
# Upgrade the kind images to the latest available version from

modules/oci-build/00_mod.mk:21

  • The custom regex's second match string captures the entire right-hand side as currentValue, so Renovate will see quay.io/jetstack/base-static-csi:latest@sha256:... as the Docker version rather than latest with the digest in currentDigest. That is not a valid Docker tag/version for the datasource, so this dependency will not be matched to the latest release and its digest will not be upgraded. Update the matcher to capture the tag and digest separately while keeping the full reference in this assignment.
base_image_csi-static := quay.io/jetstack/base-static-csi:latest@sha256:83a2ffc4266da49036b08fef76e8f424a76b12ab870a702231bb4132ae3dbb88
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread modules/oci-build/00_mod.mk

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Renovate must capture Docker tags and digests separately for reliable image updates.

Review details

Suppressed comments (3)

.github/renovate.json5:10

  • The new file pattern makes the base-image annotations eligible, but the existing matcher still captures the entire right-hand side as currentValue. For these entries that value is quay.io/jetstack/base-static:latest@sha256:..., even though packageName is already supplied separately; the Docker datasource expects the tag in currentValue and the pin in a currentDigest capture. As written, Renovate will not reliably recognize or update these dependencies, so the removed scheduled workflow leaves the images unmanaged. Update the matcher to capture the tag and digest separately (and avoid also matching the same line with the generic assignment pattern).
        'modules/**/*.mk',

modules/oci-build/00_mod.mk:17

  • The Docker reference is captured by the generic currentValue group as the entire latest@sha256:... string; this does not provide Renovate's regex manager with a currentDigest. Digest-pinned Docker dependencies should capture latest as currentValue and sha256:... as currentDigest, otherwise Renovate may not recognize the pinned digest and will not create the intended digest updates. Update the matcher and keep this value in the tag@digest form it writes back.
base_image_static := quay.io/jetstack/base-static:latest@sha256:dc719ea428e9b88843862bd8de896bbe974088bc838d000a33f091a835744c33

modules/oci-build/00_mod.mk:21

  • The Docker reference is captured by the generic currentValue group as the entire latest@sha256:... string; this does not provide Renovate's regex manager with a currentDigest. Digest-pinned Docker dependencies should capture latest as currentValue and sha256:... as currentDigest, otherwise Renovate may not recognize the pinned digest and will not create the intended digest updates. Update the matcher and keep this value in the tag@digest form it writes back.
base_image_csi-static := quay.io/jetstack/base-static-csi:latest@sha256:83a2ffc4266da49036b08fef76e8f424a76b12ab870a702231bb4132ae3dbb88
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The overlapping Renovate expressions extract duplicate, malformed dependencies for both base images.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread .github/renovate.json5 Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The overlapping Renovate patterns extract malformed duplicate dependencies for both base images.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread .github/renovate.json5 Outdated
@erikgb
erikgb force-pushed the reno-upgrade-base-images branch from b7e5961 to e797e63 Compare September 13, 2026 11:07
@erikgb
erikgb requested a balanced review from Copilot September 13, 2026 11:08
@erikgb
erikgb force-pushed the reno-upgrade-base-images branch 2 times, most recently from e797e63 to 258fdf6 Compare September 13, 2026 11:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The Renovate configuration correctly detects existing dependencies and the new base-image digest references.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@erikgb

erikgb commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

/cc @wallrj @inteon

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Renovate correctly discovers the base-image dependencies while the obsolete automation is removed consistently.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Signed-off-by: Erik Godding Boye <egboye@gmail.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Renovate correctly extracts all existing annotated Make dependencies, including both base-image tags and digests.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Comment thread .github/renovate.json5
matchStrings: [
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+ \\+= \\S+=(?<currentValue>\\S+)',
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+ (:|\\?)= (?<currentValue>\\S+)'
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+\\s+(?:\\+=\\s+[^=]*=|(:|\\?)\\s*=\\s*)(?<currentValue>[^\\s:@]+)\\s*(?:\\r\\n|\\r|\\n|$)',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new end-of-line terminator on this pattern makes Renovate skip any # renovate: entry that directly follows another one.

\s*(?:\r\n|\r|\n|$) consumes the newline after the value (and any blank lines after it, via the greedy \s*). Renovate applies matchStrings with a global regex, so matches do not overlap. The next entry's leading (?:^|\r\n|\r|\n) anchor then has no newline left to match and that entry is silently dropped. It only works today because every tools += entry happens to have a # https://... comment line above it.

Reproduced with Python re (same non-overlapping semantics as RE2 + g):

# renovate: datasource=github-releases packageName=a/a
tools += a=v1
# renovate: datasource=github-releases packageName=b/b
tools += b=v2

new pattern: ['a/a']; the two patterns on main: ['a/a', 'b/b']. Same result with a blank line between the entries. A value with a trailing comment (x := v1.2.3 # pinned) also stops matching.

The rewrite is not needed for the exclusivity Copilot asked for. The old patterns already cannot match the two docker lines because they require \s+packageName= in the comment, and the docker comments have none. I checked: old patterns and new pattern give identical matches (47/47) on modules/tools and modules/cert-manager, and both give 0 matches on the two base_image_* lines.

Suggested change
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+\\s+(?:\\+=\\s+[^=]*=|(:|\\?)\\s*=\\s*)(?<currentValue>[^\\s:@]+)\\s*(?:\\r\\n|\\r|\\n|$)',
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+ \\+= \\S+=(?<currentValue>\\S+)',
'(?:^|\\r\\n|\\r|\\n)#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+packageName=(?<packageName>\\S+)(?:^|\\r\\n|\\r|\\n)\\S+ (:|\\?)= (?<currentValue>\\S+)',

[Claude Fable 5.1]

Comment thread .github/renovate.json5
'modules/cert-manager/**/*.mk',
'modules/tools/**/*.mk',
'modules/**/*.mk',
],

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new base image PRs will not auto-merge, and there will be two of them per rebuild.

The deleted workflow added the skip-review label (see #735: dco-signoff: yes, skip-review, merged by tide 30 seconds after creation). Tide only merges renovate[bot] PRs that carry skip-review plus dco-signoff: yes (testing config.yaml#L306-L318). The shared preset adds skip-review only for gcr.io/distroless/**, gcr.io/cloud-builders/**, docker.io/library/golang and github.com/cert-manager/** (default.json5), so quay.io/jetstack/base-static* digest PRs will wait for a human. The repo's own groupName: null rule for digest updates also splits the two images into separate PRs; 4 of the last 5 bumps changed both digests in one commit.

cert-manager has the rule to copy (renovate.json5#L47-L58). Place it after the ungroup rule so it wins:

    {
      groupName: 'Base images',
      matchManagers: ['custom.regex'],
      matchFileNames: ['modules/oci-build/**'],
      addLabels: ['skip-review'], // Adding label to allow PRs to automerge
    },

If the intent is that a human now reviews base image bumps, please say so in the PR description.

[Claude Fable 5.1]

# To get latest SHA run "crane digest quay.io/jetstack/base-static:latest"
base_image_static := quay.io/jetstack/base-static@sha256:dc719ea428e9b88843862bd8de896bbe974088bc838d000a33f091a835744c33
# renovate: datasource=docker
base_image_static := quay.io/jetstack/base-static:latest@sha256:dc719ea428e9b88843862bd8de896bbe974088bc838d000a33f091a835744c33

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding :latest to these values exists only to give the second regex a currentValue. Renovate's currentValueTemplate does that without changing the data, and cert-manager already uses it for the same job (renovate.json5#L14-L27, currentValueTemplate: 'nonroot'; it produced #9350 and #9329 this month).

A separate manager scoped to this file would replace the second matchStrings entry, the optional packageName group, the widening to modules/**/*.mk, and both # renovate: comments:

    {
      customType: 'regex',
      managerFilePatterns: ['modules/oci-build/00_mod.mk'],
      matchStrings: [
        '(?<depName>quay\\.io/jetstack/[^@\\s]+)@(?<currentDigest>sha256:[a-f0-9]{64})',
      ],
      datasourceTemplate: 'docker',
      currentValueTemplate: 'latest',
    },

Why it matters:

  • The value shape flows to the eight repos that klone oci-build and feed it to ko as .defaultBaseImage (01_mod.mk:32). ko handles repo:tag@sha256:... correctly (go-containerregistry NewDigest strips the tag), so nothing breaks, but every consumer gets a churn diff and a :latest tag that reads as floating when the digest pins it.
  • The two matchStrings are only disjoint because the docker values contain : and the tools values do not. The second regex also accepts only := (not ?=), stops depName at the first : (registry ports), and does not match the pre-PR repo@sha256:... form; all fail silently.
  • The deleted crane digest quay.io/jetstack/base-static:latest hint was the only documented manual bump path. With a scoped manager the lines and their comments can stay exactly as on main.

[Claude Fable 5.1]

Comment thread .github/renovate.json5
managerFilePatterns: [
'modules/cert-manager/**/*.mk',
'modules/tools/**/*.mk',
'modules/**/*.mk',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the manager widened to modules/**/*.mk, the first packageRules entry (matchManagers: ['custom.regex'] with postUpgradeTasks: make vendor-go learn-image-shas) now also fires for the base image digest PRs.

scripts/learn_image_shas.sh reads only modules/cert-manager/00_mod.mk and runs eight crane digest calls against the cert-manager release images. For a base image PR that is a Go toolchain download, a crane build and network round trips for a no-op. If quay.io is flaky the unrelated task fails the base image PR; if a release tag were ever re-pushed, cert-manager digest edits would land inside a base image PR. #748 shows the task does run and does rewrite that file.

Scope the rule the same way the Tools rule below it already is:

      matchFileNames: ['modules/cert-manager/**'],

[Claude Fable 5.1]

@@ -1,97 +0,0 @@
name: base-images-upgrade

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR body gives failing workflows in a fork as the motivation. kind-images-upgrade.yaml is a copy of this file with only the names and make target changed, and it still runs nightly in every fork (the erikgb fork ran it on 12, 13 and 15 September).

It cannot move to Renovate the same way: the source of truth for kind node images is the kind release notes, not the registry. The fix this repo already uses for that case is a job-level guard, as in govulncheck.yaml#L20:

    if: github.repository == 'cert-manager/makefile-modules'

Fine as a follow-up PR; raising it so the second half of the motivation is not forgotten.

[Claude Fable 5.1]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates that all commits in the pull request have the valid DCO sign-off message. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants