Skip to content

[scanner] installer-test / Install and boot in QEMU fails on main — /var partition times out at boot #161

Description

@kubestellar-hive

Finding

installer-test / Install and boot in QEMU fails on main itself (run 34920225526, job 104235482419: "Build DDI artifacts"), not just on open PRs. Confirmed the same failure reproduces identically on two unrelated open PRs (#81 pin argocd digest — touches only files/k0s/manifests/argocd/install.yaml; #84 renovate series fix — touches only CI/renovate config), so this is a pre-existing regression on main, not caused by either PR's diff.

Serial log (from PR #81's run):

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Activate k0s sysext on boot.
...
You are in emergency mode.

Suspected cause

files/installer/repart.d/20-root-a.conf sets GrowFileSystem=yes with SizeMaxBytes=16G (i.e. up to the full 16G test disk, see SHOW_ME_THE_FUTURE_DISK_SIZE:-16G / INSTALL_VM_DISK_SIZE:-16G in Justfile), while files/installer/repart.d/30-var.conf (comment: "grows to fill disk") relies on getting the disk's leftover space with only SizeMinBytes=4G. Neither partition sets an explicit Priority=/Weight=, so systemd-repart splits remaining free space between the two GrowFileSystem=yes partitions by default weight. Depending on the actual size of the DDI payload copied via CopyBlocks=/dev/disk/by-partlabel/bluefin-installer-data in 20-root-a.conf, root-a's minimum footprint can grow large enough that the var partition either isn't created or ends up undersized/never appears, leaving var.mount (files/os/systemd/system/var.mount, WantedBy=local-fs.target) waiting on a device that never shows up, dropping the second boot into emergency mode.

Introduced/adjusted across recent commits: 1500324 (enable GrowFileSystem for root-a), 634eabc (configure xfs for /var), 3ac9d80/37c8356 (provision /var partition + var.mount wiring).

Recommendation

Give root-a a bounded, realistic SizeMaxBytes (sized to the actual DDI image, not the full test-disk size) and/or set explicit Priority=/Weight= on 30-var.conf so var is guaranteed its SizeMinBytes=4G regardless of how large root-a grows. Verify with just show-me-the-future locally (or in CI) that /dev/disk/by-partlabel/var appears before local-fs.target is reached on the second boot.

Impact

Blocks the installer-test gate on every open PR against server, including #81 and #84, even though neither is at fault. This is why "fix-before-new" for those two PRs is not achievable by editing their branches — the root cause lives on main.


Filed by scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions