[PATCH 00/13] bootstd: bls: Scan every partition; Ubuntu autoinstall via BLS
From: Simon Glass <sjg@chromium.org> The Boot Loader Specification places loader/entries/*.conf on the ESP or XBOOTLDR partition, and bootmeth_bls relies on the iterator's BLS-target carve-out to reach those even when not marked bootable. In practice, however, distros that drive entries through kernel-install write to whichever partition holds /boot, which on common Ubuntu/Debian layouts is the ext4 rootfs. Two consequences follow today: efi-x86_app64 overrides bootcmd to "bootflow scan -lpb" so the scan reaches non-BLS-target data partitions on hybrid ISOs, and the Ubuntu autoinstall path writes BLS entries onto the ESP rather than the rootfs. Patch 1 reverts the bootcmd override. Patch 2 sets BOOTMETHF_ANY_PART on bootmeth_bls so the iterator hands it every partition, with a partition-size filter so ext4 probes do not run on raw single-LBA slots (e.g. ChromeOS kernel partitions). Patch 3 honours --kernel in the grub-cfg parser. Patch 4 wires BLS into future kernel updates: the autoinstall path writes BLS entries directly to the ESP (kernel-install never works inside curtin's chroot anyway, because /boot/efi is a plain directory there), while the interactive path drops a first-boot systemd unit that runs kernel-install from the real installed system. Entries land on the ESP for autoinstall so the bootflow iterator's per-partition advance picks BLS up before the EFI bootmeth chain-loads our own BOOTX64.EFI fallback into a loop. Patches 5-11 cover further Ubuntu autoinstall script work: unattended autoinstall mode (5), planting u-boot.efi on the target ESP (6), and a small cleanup pass (7-11). Patch 12 adds a new CI test, test_distro_ubuntu_iso_install, that runs an unattended Ubuntu install through U-Boot + BLS under QEMU + OVMF and verifies the installed system boots back into U-Boot. Patch 13 wires it into the sjg-lab-slow gitlab-ci stage. Docs in doc/usage/bls.rst, doc/develop/bootstd/bls.rst, doc/develop/bootstd/overview.rst and doc/usage/os/ubuntu-live.rst are updated to match. The end-to-end CI test passes (4m46 on sjg-lab-slow). Simon Glass (13): Revert "configs: efi-x86_app64: Scan all partitions at autoboot" bootmeth: bls: Scan every partition for entries scripts: ubuntu: Honour --kernel in grub-cfg parser scripts: ubuntu: Wire BLS into future kernel updates scripts: ubuntu: Add unattended autoinstall mode scripts: ubuntu: Install U-Boot on target ESP scripts: ubuntu: Drop __future__ annotations import scripts: ubuntu: Refer readers to the noble release dir scripts: ubuntu: Use command.run() in hash_password() scripts: ubuntu: Lift BLS-setup magic strings scripts: ubuntu: Use Args:/Returns: docstrings throughout test/py: distro: Add test_distro_ubuntu_iso_install gitlab-ci: Add sjg-lab-slow stage and Ubuntu install job .gitlab-ci.yml | 37 +++ boot/bootmeth_bls.c | 24 +- configs/efi-x86_app64_defconfig | 1 - doc/develop/bootstd/bls.rst | 34 ++- doc/develop/bootstd/overview.rst | 11 +- doc/usage/bls.rst | 46 +++- doc/usage/os/ubuntu-live.rst | 195 +++++++++++---- scripts/ubuntu-iso-to-uboot.py | 415 +++++++++++++++++++++++++++++-- test/py/tests/test_distro.py | 53 ++++ 9 files changed, 730 insertions(+), 86 deletions(-) --- base-commit: 6aab88b82cfb744689466b02c7a6a7bc663f2e90 branch: fsf -- 2.43.0
From: Simon Glass <sjg@chromium.org> This reverts commit e2fe05eb6f3e4a5d24a4eec6c330edf77b4c8f9a. That commit overrides bootcmd to "bootflow scan -lpb" so the scan reaches data partitions on hybrid ISOs that the strict BLS partition-type filter in earlier bootmeth_bls skipped. The following patch lifts the filter altogether by setting BOOTMETHF_ANY_PART on bootmeth_bls, so the bootmeth visits every partition itself and the board-specific override is redundant. Fall back to the default "bootflow scan -lb" with no loss of coverage. Signed-off-by: Simon Glass <sjg@chromium.org> --- configs/efi-x86_app64_defconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/configs/efi-x86_app64_defconfig b/configs/efi-x86_app64_defconfig index cf9114021e8..50c700e20c5 100644 --- a/configs/efi-x86_app64_defconfig +++ b/configs/efi-x86_app64_defconfig @@ -22,7 +22,6 @@ CONFIG_BOOTMETH_BLS=y CONFIG_SHOW_BOOT_PROGRESS=y CONFIG_USE_BOOTARGS=y CONFIG_BOOTARGS="root=/dev/sdb3 init=/sbin/init rootwait ro" -CONFIG_BOOTCOMMAND="bootflow scan -lpb" CONFIG_SYS_PBSIZE=532 CONFIG_SYS_CONSOLE_INFO_QUIET=y # CONFIG_CONSOLE_PAGER is not set -- 2.43.0
From: Simon Glass <sjg@chromium.org> The Boot Loader Specification places loader/entries/*.conf on the ESP or XBOOTLDR partition, and bootmeth_bls relies on the iterator's BLS-target carve-out to reach those even when they are not marked bootable. In practice, however, kernel-install (the systemd helper that distros such as Ubuntu use for kernel package hooks) writes its entries to /boot/loader/entries on whichever partition holds /boot, which is typically the ext4 rootfs. Set BOOTMETHF_ANY_PART on the BLS bootmeth so the iterator hands it every partition on each bootdev rather than just bootable and BLS-target ones, matching bootmeth_android, bootmeth_rauc and bootmeth_cros. The default filename prefixes ('/' and '/boot/') already cover both ESP-relative and rootfs-relative entry paths, so the scan picks up entries on the ESP and on the rootfs. Visiting every partition means BLS now probes filesystems on raw data slots too (e.g. ChromeOS kernel partitions, which are a single LBA), and the ext4 superblock probe at sector 2 trips fs_devread()'s "Read outside partition" error on those. Filter out partitions below a small threshold up front so the FS probes only run on partitions that could plausibly hold one. Signed-off-by: Simon Glass <sjg@chromium.org> --- boot/bootmeth_bls.c | 24 ++++++++++++++++- doc/develop/bootstd/bls.rst | 34 ++++++++++++++++------- doc/develop/bootstd/overview.rst | 11 +++++--- doc/usage/bls.rst | 46 ++++++++++++++++++++++++++------ 4 files changed, 93 insertions(+), 22 deletions(-) diff --git a/boot/bootmeth_bls.c b/boot/bootmeth_bls.c index 85d8dfa91df..6f6d03d286b 100644 --- a/boot/bootmeth_bls.c +++ b/boot/bootmeth_bls.c @@ -48,6 +48,13 @@ #define BLS_ENTRIES_DIR "loader/entries" #define BLS_ENTRY_FILE "loader/entry.conf" +/* + * Minimum partition size, in blocks, that is plausibly a filesystem. + * Partitions below this are skipped before FS probing to avoid + * out-of-bounds superblock reads on raw data slots + */ +#define BLS_MIN_PART_BLOCKS 64 + /** * struct bls_info - context information for BLS getfile callback * @@ -320,6 +327,21 @@ static int bls_read_bootflow(struct udevice *dev, struct bootflow *bflow) prefixes = bootstd_get_prefixes(bootstd); desc = bflow->blk ? dev_get_uclass_plat(bflow->blk) : NULL; + /* + * BOOTMETHF_ANY_PART makes the iterator visit every partition, + * including small non-filesystem ones (e.g. ChromeOS kernel slots). + * Skip partitions too small to hold any filesystem before probing, + * so probe-time reads do not trip the "Read outside partition" + * error path + */ + if (desc) { + struct disk_partition info; + + if (!part_get_info(desc, bflow->part, &info) && + info.size < BLS_MIN_PART_BLOCKS) + return -ENOENT; + } + /* Try each prefix: first scan entries/, then fall back to entry.conf */ i = 0; ret = -ENOENT; @@ -521,7 +543,7 @@ static int bls_bootmeth_bind(struct udevice *dev) plat->desc = IS_ENABLED(CONFIG_BOOTSTD_FULL) ? "Boot Loader Specification (BLS) Type #1" : "bls"; - plat->flags = BOOTMETHF_MULTI; + plat->flags = BOOTMETHF_MULTI | BOOTMETHF_ANY_PART; return 0; } diff --git a/doc/develop/bootstd/bls.rst b/doc/develop/bootstd/bls.rst index 327ab7c1c55..37eddd455f3 100644 --- a/doc/develop/bootstd/bls.rst +++ b/doc/develop/bootstd/bls.rst @@ -9,15 +9,31 @@ Fedora, RHEL and other distributions. .. _Boot Loader Specification: https://uapi-group.org/specifications/specs/boot_loader_specification/ -The entry file ``loader/entry.conf`` is searched for under each boot prefix -(``{"/", "/boot"}`` by default). These prefixes can be selected with the -`filename-prefixes` property in the bootstd device. - -The bootmeth sets ``BOOTMETHF_MULTI`` so that each ``.conf`` file in the -entries directory produces a separate bootflow. The ``bflow->entry`` field -selects which file to use during scanning. - -When invoked on a bootdev, the ``bls_read_bootflow()`` function searches for the +For each partition the bootmeth tries every boot prefix (``{"/", "/boot/"}`` +by default; selectable via the ``filename-prefixes`` property on the bootstd +device). Within each prefix it scans ``loader/entries/`` for ``.conf`` files +first, then falls back to the singular ``loader/entry.conf`` if the directory +is missing or empty. + +The bootmeth sets two flags at bind time: + +* ``BOOTMETHF_MULTI`` -- each ``.conf`` file in the entries directory produces + a separate bootflow, with ``bflow->entry`` indexing which one is returned. +* ``BOOTMETHF_ANY_PART`` -- the iterator visits every partition rather than + only bootable partitions and the ESP / XBOOTLDR / MBR-0xea targets carved out + by ``part_is_bls_target()``. Distros that drive their entries through + ``kernel-install`` typically place ``loader/entries/`` on whichever partition + holds ``/boot``, which on Debian/Ubuntu layouts is the ext4 rootfs rather + than the ESP, and ``BOOTMETHF_ANY_PART`` is what lets BLS reach those. + +Because ``BOOTMETHF_ANY_PART`` invites the bootmeth onto every partition, +``bls_read_bootflow()`` looks the partition up before mounting and returns +``-ENOENT`` for partitions below ``BLS_MIN_PART_BLOCKS`` blocks. This avoids +running the ext4 superblock probe (which reads sector 2) on raw single-LBA +slots like ChromeOS kernel partitions, where the probe would otherwise trip +``fs_devread()``'s "Read outside partition" error path. + +When invoked on a bootdev, the ``bls_read_bootflow()`` function searches for an entry file, reads it and passes it to ``bls_parse_entry()`` which processes the key-value pairs into a ``struct bls_entry``. The parser uses an enum-based token lookup to map field names, with most values pointing directly into the diff --git a/doc/develop/bootstd/overview.rst b/doc/develop/bootstd/overview.rst index b69156cb51d..89fd77cf695 100644 --- a/doc/develop/bootstd/overview.rst +++ b/doc/develop/bootstd/overview.rst @@ -815,10 +815,13 @@ calls the bootmeth device once more, this time to read the bootflow. Note: Normally a filesystem is needed for the bootmeth to be called on block devices, but bootmeths which don't need that can set the BOOTMETHF_ANY_PART -flag to indicate that they can scan any partition. An example is the ChromiumOS -bootmeth which can store a kernel in a raw partition. Note also that sandbox is -a special case, since in that case the host filesystem can be accessed even -though the block device is NULL. +flag to indicate that they can scan any partition. The ChromiumOS bootmeth +sets it because it stores a kernel in a raw partition; the BLS bootmeth sets +it so it can find ``loader/entries/`` on the rootfs partition that distros +mount at ``/boot`` (in addition to the ESP and XBOOTLDR partitions called out +by the BLS spec). Note also that sandbox is a special case, since in that +case the host filesystem can be accessed even though the block device is +NULL. If we take the example of the `bootmeth_extlinux` driver, this call ends up at `extlinux_read_bootflow()`. It has the filesystem ready, so tries various diff --git a/doc/usage/bls.rst b/doc/usage/bls.rst index 128092ae25b..8d8e7246243 100644 --- a/doc/usage/bls.rst +++ b/doc/usage/bls.rst @@ -13,11 +13,13 @@ Overview BLS provides a standardised way to describe boot entries. U-Boot's BLS support allows it to boot operating systems configured with BLS entries, which is used -by Fedora, RHEL, and other distributions. +by Fedora, RHEL, Ubuntu (via ``kernel-install``) and other distributions. -The current implementation supports a single BLS entry file at -``loader/entry.conf``. Future versions may support multiple entries in -``loader/entries/``. +U-Boot scans both the standard ``loader/entries/<machine-id>-<version>.conf`` +layout used by ``kernel-install`` and a single ``loader/entry.conf`` fallback +under each boot prefix. When an entries directory is present, each ``.conf`` +file becomes a separate bootflow; ``loader/entry.conf`` is consulted only when +the directory is absent or empty. Configuration ------------- @@ -28,6 +30,30 @@ Enable BLS support with:: This automatically selects ``CONFIG_PXE_UTILS`` for booting. +Discovery +--------- + +The BLS bootmeth sets ``BOOTMETHF_ANY_PART``, so the bootflow iterator hands +it every partition on each bootdev rather than just the bootable partition or +those typed as ESP / XBOOTLDR / MBR-0xea. This matches the way ``kernel-install`` +places entries: distros that mount the ESP at ``/boot/efi`` keep +``loader/entries/`` on the ext4 rootfs at ``/boot/loader/entries/``, while +distros that mount the ESP at ``/boot`` keep them on the ESP at +``loader/entries/``. Both cases are reached without any board configuration. + +For each partition the bootmeth tries each filename prefix configured on the +bootstd device (``/`` and ``/boot/`` by default; see ``filename-prefixes`` in +:doc:`/develop/bootstd/overview`) and within each prefix tries the entries +directory first, then the singular ``entry.conf`` fallback:: + + <prefix>loader/entries/*.conf + <prefix>loader/entry.conf + +Partitions too small to hold any filesystem are filtered out before the +filesystem probe runs, so raw data partitions (for example ChromeOS kernel +slots, which are a single LBA) do not produce spurious "Read outside +partition" probe errors. + BLS Entry Format ---------------- @@ -128,23 +154,27 @@ BLS boot entries are discovered automatically by standard boot:: => bootflow select 0 => bootflow boot -The BLS entry at ``loader/entry.conf`` on any available media is recognised as -a bootflow. +Each ``.conf`` file in a discovered ``loader/entries/`` directory, or the +single ``loader/entry.conf`` fallback, becomes its own bootflow. Implementation Notes -------------------- -* Single BLS entry file support (``loader/entry.conf``) * Boot execution reuses U-Boot's PXE infrastructure for kernel loading * Unknown fields are ignored for forward compatibility * The bootmethod is ordered as ``bootmeth_2bls`` (after extlinux) * Zero-copy parsing: most fields point into bootflow buffer (except ``options`` which is allocated for concatenation) +* ``BOOTMETHF_MULTI`` is set, so each ``.conf`` file in the entries directory + produces a separate bootflow +* ``BOOTMETHF_ANY_PART`` is set, so the iterator visits every partition rather + than only those tagged as bootable or as BLS-target Current Limitations ------------------- -* Only single entry file, not multiple entries directory scanning +* Only the first ``.conf`` file in ``loader/entries/`` is loaded per call; + the iterator advances ``bflow->entry`` to retrieve subsequent ones * No devicetree-overlay support * No architecture/machine-id filtering * No version-based or sort-key sorting -- 2.43.0
From: Simon Glass <sjg@chromium.org> parse_grub_cmdline() hard-codes 'casper/vmlinuz' in its regex, so a user passing -k with a non-default kernel path silently falls through to the 'could not find a casper linux entry' error even when grub.cfg does carry a matching entry under the requested path. Take the kernel path as a parameter, re.escape() it before splicing into the pattern, and quote it in the failure message so the user sees what was searched for. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index f6326b7eceb..22c6928091b 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -104,10 +104,10 @@ def check_tools() -> None: ) -def parse_grub_cmdline(iso: Path) -> str: +def parse_grub_cmdline(iso: Path, kernel: str) -> str: """Return the kernel cmdline from the ISO's default grub entry. - Parses the first `linux /casper/vmlinuz ...` line in /boot/grub/grub.cfg + Parses the first `linux <kernel> ...` line in /boot/grub/grub.cfg and strips the kernel path, so the caller can pass the remaining tokens to the kernel (e.g. '--- quiet splash' on Ubuntu 24.04.1). """ @@ -119,12 +119,17 @@ def parse_grub_cmdline(iso: Path) -> str: ) cfg = dst.read_text(errors='replace') + # The kernel path comes from the user (or its default), so escape it + # before splicing it into the regex. + kernel_re = re.escape(kernel.lstrip('/')) m = re.search( - r'^\s*linux\s+\S*casper/vmlinuz\S*\s*(.*)$', + rf'^\s*linux\s+\S*{kernel_re}\S*\s*(.*)$', cfg, re.MULTILINE, ) if not m: - tout.fatal('could not find a casper linux entry in /boot/grub/grub.cfg') + tout.fatal( + f'could not find a {kernel} linux entry in /boot/grub/grub.cfg' + ) # Collapse any run of whitespace to a single space return ' '.join(m.group(1).split()) @@ -249,7 +254,7 @@ def main() -> None: title = args.title or f'U-Boot BLS boot ({vol_id})' cmdline = args.cmdline if cmdline is None: - cmdline = parse_grub_cmdline(args.iso) + cmdline = parse_grub_cmdline(args.iso, args.kernel) tout.notice(f' Volume label: {vol_id}') tout.notice(f' ESP GUID: {esp_guid}') tout.notice(f' Cmdline: {cmdline}') -- 2.43.0
From: Simon Glass <sjg@chromium.org> A rewritten live ISO boots via U-Boot + BLS, but once the user selects 'Install Ubuntu', apt-managed kernel updates still go through update-grub and no longer refresh /boot/loader/entries/. The target system therefore keeps booting the kernel the installer unpacked, not whatever 'apt install linux-image-*' lays down later. Reach both install paths: - Autoinstall path. Plant /autoinstall.yaml at the ISO 9660 root alongside the existing /loader/entry.conf. Its late-commands write a BLS Type #1 entry directly to /boot/efi/loader/entries/<v>.conf for every kernel the installer unpacks and copy the kernel and initrd alongside on the ESP. The kernel-install round-trip is bypassed because curtin's chroot has /boot/efi as a plain directory rather than the ESP mountpoint, so systemd's 90-loaderentry.install plugin silently exits without writing entries. Interactive installs ignore /autoinstall.yaml entirely. - Interactive path. Unpack the default install squashfs (casper/minimal.squashfs), drop in a oneshot systemd unit plus its helper script, and repack. On first boot of the installed system the unit runs 'apt install systemd-boot-efi' and kernel-install for each installed kernel - this time inside the real installed system where /boot/efi is the ESP - then touches a done-file so it does not re-run. The unit is gated on ConditionPathExists=!/cdrom so it stays dormant in the live session. Runs under one fakeroot invocation so ownership makes it through the unpack/modify/repack cycle without root. Place autoinstall entries on the ESP rather than on /boot/loader/entries on the rootfs because the bootflow iterator advances per-partition: the ESP is the lower-numbered partition and the EFI bootmeth there matches our BOOTX64.EFI fallback before BLS ever reaches the rootfs further down, chain-loading U-Boot into itself. With BLS entries on the ESP, BLS wins on that partition before EFI gets a turn. For interactive systems, kernel-install's install.d hooks maintain the entries on every subsequent 'apt install linux-image-*'. Autoinstall systems ship with one entry per installer-unpacked kernel but do not auto-refresh; the autoinstall path is intended to get the freshly installed system to its first boot under U-Boot, not to replace kernel-install. Either way U-Boot's BLS bootmeth keeps picking up the current kernel/initrd. Add -N / --no-target-bls for callers that want a plain rewritten ISO, and -I / --install-squashfs for ISOs whose install source is not casper/minimal.squashfs. Generalise repack_iso() to take an arbitrary list of (src, dst) mappings instead of a single entry_conf so the modified squashfs and other files can join without reshuffling the call site. Signed-off-by: Simon Glass <sjg@chromium.org> --- doc/usage/os/ubuntu-live.rst | 175 ++++++++++++++++++++++------- scripts/ubuntu-iso-to-uboot.py | 199 +++++++++++++++++++++++++++++++-- 2 files changed, 323 insertions(+), 51 deletions(-) diff --git a/doc/usage/os/ubuntu-live.rst b/doc/usage/os/ubuntu-live.rst index 991d79a9454..cfcc5243ca2 100644 --- a/doc/usage/os/ubuntu-live.rst +++ b/doc/usage/os/ubuntu-live.rst @@ -4,38 +4,42 @@ Booting Ubuntu live ISOs via U-Boot =================================== U-Boot can replace GRUB as the bootloader on an Ubuntu live ISO. -``scripts/ubuntu-iso-to-uboot.py`` rewrites the ISO so the appended EFI -system partition holds a U-Boot EFI application and the ISO 9660 tree -carries a :doc:`Boot Loader Specification Type #1 </usage/bls>` entry -pointing at the casper kernel and initrd that Ubuntu already places on -the disc. +``scripts/ubuntu-iso-to-uboot.py`` rewrites the ISO so the appended EFI system +partition holds a U-Boot EFI application and the ISO 9660 tree carries +a :doc:`Boot Loader Specification Type #1 </usage/bls>` entry pointing at the +casper kernel and initrd that Ubuntu already places on the disc. -All other boot records (BIOS El Torito, grub2 MBR, GPT layout) are -preserved verbatim by xorriso's ``-boot_image any replay``, and casper -still finds its squashfs by disc label at runtime. +All other boot records (BIOS El Torito, grub2 MBR, GPT layout) are preserved +verbatim by xorriso's ``-boot_image any replay``, and casper still finds its +squashfs by disc label at runtime. Host prerequisites ------------------ :: - sudo apt install xorriso mtools dosfstools qemu-system-x86 ovmf + sudo apt install xorriso mtools dosfstools squashfs-tools fakeroot \\ + qemu-system-x86 ovmf + +``squashfs-tools`` and ``fakeroot`` are only needed when the script modifies the +install squashfs (see :ref:`target-bls-setup`); pass ``-N`` to skip that step +if you'd rather not install them. Building the U-Boot EFI application ----------------------------------- The ``efi-x86_app64`` target enables ``CONFIG_BOOTMETH_BLS=y``, -``CONFIG_FS_ISOFS=y`` and ``CONFIG_JOLIET=y`` by default, so no Kconfig -tweaks are required. Build with:: +``CONFIG_FS_ISOFS=y`` and ``CONFIG_JOLIET=y`` by default, so no Kconfig tweaks +are required. Build with:: make O=/tmp/b/efi-x86_app64 efi-x86_app64_defconfig make O=/tmp/b/efi-x86_app64 -j$(nproc) -The output is ``/tmp/b/efi-x86_app64/u-boot-app.efi``, a PE32+ x86_64 -EFI application. +The output is ``/tmp/b/efi-x86_app64/u-boot-app.efi``, a PE32+ x86_64 EFI +application. -If ``rustc`` is not installed, also disable the rust example build -before the main ``make``:: +If ``rustc`` is not installed, also disable the rust example build before the +main ``make``:: scripts/config --file /tmp/b/efi-x86_app64/.config \\ -d RUST_EXAMPLES -d EXAMPLES @@ -44,8 +48,7 @@ before the main ``make``:: Rewriting an Ubuntu ISO ----------------------- -Fetch the ISO (desktop and server images both work) and run the -helper:: +Fetch the ISO (desktop and server images both work) and run the helper:: curl -LO https://releases.ubuntu.com/24.04.1/ubuntu-24.04.1-desktop-amd64.iso @@ -55,39 +58,51 @@ helper:: The script: -1. Reads the input ISO's boot record with ``xorriso -report_el_torito`` - to pick up the volume label and the EFI system partition GUID. +1. Reads the input ISO's boot record with ``xorriso -report_el_torito`` to pick + up the volume label and the EFI system partition GUID, and parses + ``/boot/grub/grub.cfg`` to inherit the kernel command line from the source + ISO's own default entry (``--- quiet splash`` on Ubuntu 24.04.1). 2. Builds a small FAT ESP (4 MiB by default) containing just ``/EFI/BOOT/BOOTX64.EFI`` -- the U-Boot EFI application. -3. Writes a new ISO with - ``xorriso -indev ... -outdev ... -boot_image any replay - -append_partition 2 ... -map entry.conf /loader/entry.conf``, - which replaces the original ESP and adds ``/loader/entry.conf`` to - the ISO 9660 tree. The kernel and initrd stay in ``/casper/`` on the - ISO 9660 tree; U-Boot reads them directly via its isofs driver. +3. Writes a new ISO with ``xorriso -indev ... -outdev ... -boot_image any replay + -append_partition 2 ... -map entry.conf /loader/entry.conf``, which replaces + the original ESP and adds ``/loader/entry.conf`` to the ISO 9660 tree. The + kernel and initrd stay in ``/casper/`` on the ISO 9660 tree; U-Boot reads + them directly via its isofs driver. 4. Strips the shim, GRUB and MOK manager binaries - (``/EFI/boot/{bootx64,grubx64,mmx64}.efi``) from the ISO 9660 tree. - The UEFI firmware loads ``BOOTX64.EFI`` from the appended ESP, so - the ISO 9660 copies are unused dead weight. The BIOS El Torito - image under ``/boot/grub/`` is left in place, so legacy-BIOS boot - still chains into GRUB as before. + (``/EFI/boot/{bootx64,grubx64,mmx64}.efi``) from the ISO 9660 tree. The UEFI + firmware loads ``BOOTX64.EFI`` from the appended ESP, so the ISO 9660 copies + are unused dead weight. The BIOS El Torito image under ``/boot/grub/`` is + left in place, so legacy-BIOS boot still chains into GRUB as before. +5. Wires BLS maintenance into the target system that a later ``Install Ubuntu`` + produces; see :ref:`target-bls-setup`. Relevant options: * ``-u PATH`` -- the U-Boot EFI application (required). * ``-o PATH`` -- the output ISO (required). * ``-a ARGS`` -- override the kernel command line written to - ``loader/entry.conf``. The default is - ``console=ttyS0,115200 console=tty0 --- quiet``, which logs the - kernel to serial and video. Duplicate the ``console=`` arguments - after ``---`` as well if you want casper and the running system - logged to serial too. -* ``-k PATH`` / ``-i PATH`` -- override the kernel and initrd paths - inside the ISO if a distribution uses something other than - ``casper/vmlinuz`` and ``casper/initrd``. -* ``-s MiB`` -- force an ESP size; the default is 4 MiB, enough for the - U-Boot binary. + ``loader/entry.conf``. When not given, the arguments after the first + ``linux /casper/vmlinuz`` entry in the source ISO's ``/boot/grub/grub.cfg`` + are used verbatim, so the rewritten ISO boots exactly as the original. Pass + something like ``console=ttyS0,115200 console=tty0 --- quiet splash`` when + you want the kernel logged to serial too. +* ``-k PATH`` / ``-i PATH`` -- override the kernel and initrd paths inside the + ISO if a distribution uses something other than ``casper/vmlinuz`` and + ``casper/initrd``. +* ``-s MiB`` -- force an ESP size; the default is 4 MiB, enough for the U-Boot + binary. * ``-t TITLE`` -- override the BLS entry title. +* ``-N`` / ``--no-target-bls`` -- skip the :ref:`target-bls-setup` step and + produce a plain rewritten ISO. Useful when you do not want ``squashfs-tools`` + / ``fakeroot`` on the host or cannot afford the extra ~3 minutes the squashfs + repack adds. +* ``-I PATH`` / ``--install-squashfs PATH`` -- path inside the ISO of the + install squashfs to modify for interactive installs. Defaults to + ``casper/minimal.squashfs``; set to an empty string to skip the squashfs step + while still shipping the autoinstall snippet. +* ``-v`` -- show progress markers and subprocess output (xorriso, mksquashfs, + mkfs.vfat). Off by default. Testing under QEMU + OVMF ------------------------- @@ -113,8 +128,83 @@ The expected boot trace on the serial console is roughly:: Starting kernel ... The bootflow appears on ``part_1`` -- the ISO 9660 partition -- because -``entry.conf`` lives in the ISO 9660 tree and U-Boot reads kernel and -initrd from the same partition via isofs. +``entry.conf`` lives in the ISO 9660 tree and U-Boot reads kernel and initrd +from the same partition via isofs. + +.. _target-bls-setup: + +Keeping BLS entries fresh after install +--------------------------------------- + +A rewritten live ISO boots via U-Boot + BLS, but once the user clicks +*Install Ubuntu*, apt-managed kernel updates still go through ``update-grub`` +on the target. They do not refresh ``/boot/loader/entries/``, so the installed +system keeps booting the kernel the installer unpacked rather than whatever +``apt install linux-image-*`` lays down later. + +By default the script reaches both possible install paths to seed BLS entries +for the installer's kernel: + +1. **Autoinstall path.** ``/autoinstall.yaml`` at the ISO 9660 root carries + ``late-commands`` that, after subiquity has finished copying the system to + disk, write a BLS entry per kernel directly to + ``/boot/efi/loader/entries/<v>.conf`` on the install target, copying the + kernel and initrd alongside on the ESP so the entry can reference them by + ESP-relative paths. A ``/loader/entry.conf`` fallback is written too, + matching the format the live ISO already uses. Plain interactive installs + ignore ``/autoinstall.yaml`` entirely, so shipping the file is harmless when + autoinstall is not requested. + + The ``kernel-install`` round-trip used by other distros is bypassed: inside + curtin's chroot, ``/boot/efi`` is a plain directory rather than the ESP + mountpoint, and ``systemd``'s ``90-loaderentry.install`` plugin silently + exits when it cannot find a real ESP under ``$BOOT_ROOT``. + + Entries land on the ESP rather than on ``/boot/loader/entries/`` on the + rootfs because the bootflow iterator advances per-partition. On the install + target the ESP is the lower-numbered partition; without an entry there the + EFI bootmeth would match ``/EFI/BOOT/BOOTX64.EFI`` before the BLS bootmeth + ever reached the rootfs further down, chain-loading U-Boot into itself. + Putting the entry on the ESP lets the BLS bootmeth win on that partition + before the EFI bootmeth has a turn. A separately maintained + ``/boot/loader/entries/`` on the rootfs would still be picked up (the BLS + bootmeth scans every partition); this just guarantees a working entry on + the partition the iterator reaches first. + +2. **Interactive path.** The script unpacks the default install squashfs + (``casper/minimal.squashfs``), drops in a ``oneshot`` systemd unit plus its + helper script, and repacks. On first boot of the installed system the unit + runs:: + + apt-get update + apt-get install -y systemd-boot-efi + mkdir -p /boot/loader/entries + for k in /usr/lib/modules/*; do + kernel-install add "$(basename "$k")" "$k/vmlinuz" || true + done + touch /var/lib/ubuntu-iso-to-uboot-bls-setup.done + + The unit is gated on ``ConditionPathExists=!/cdrom`` so it stays dormant + inside the live session, and the done-file stops it re-running on subsequent + boots. It needs a network on first boot for the apt call; if that is not + available it fails gracefully and retries on the next boot. + +For interactive installs, every subsequent ``apt install linux-image-*`` on the +target updates ``/boot/loader/entries/`` automatically via the +``kernel-install`` hooks, and U-Boot's BLS bootmeth keeps picking up the current +kernel/initrd without regenerating the ISO. Autoinstall systems ship with one +entry per kernel the installer unpacked but do not auto-refresh on later kernel +updates -- the autoinstall path is intended to get the freshly installed system +to its first boot under U-Boot, not to replace ``kernel-install``. Run the +interactive path's first-boot unit (or ``kernel-install add`` manually) +afterwards if you want apt-driven kernel updates to maintain BLS entries on an +autoinstalled system. + +Pass ``-N`` / ``--no-target-bls`` to skip both steps and produce a plain +rewritten ISO. The squashfs repack is the slow part of the build (about three +extra minutes on the 24.04.1 desktop ISO); skipping it keeps the rewrite under +a minute at the cost of needing a manual ``kernel-install add`` on the +installed system. See also -------- @@ -122,3 +212,4 @@ See also * :doc:`/usage/bls` -- the U-Boot Boot Loader Specification bootmeth. * `Boot Loader Specification <https://uapi-group.org/specifications/specs/boot_loader_specification/>`_. +* ``kernel-install(8)`` -- systemd's kernel install hook. diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index 22c6928091b..1c09e226d16 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -76,9 +76,53 @@ sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'tools')) from u_boot_pylib import command from u_boot_pylib import tout -REQUIRED_TOOLS = ('xorriso', 'mcopy', 'mmd', 'mkfs.vfat') +REQUIRED_TOOLS = ( + 'xorriso', 'mcopy', 'mmd', 'mkfs.vfat', + 'unsquashfs', 'mksquashfs', 'fakeroot', +) MIB = 1024 * 1024 +# First-boot systemd unit + helper script written into the install +# squashfs so kernel-install manages BLS entries on the installed +# system. ConditionPathExists=!/cdrom keeps the unit dormant inside +# the live session, and the done-file stops it re-running on every +# subsequent boot. +FIRST_BOOT_SCRIPT = '''\ +#!/bin/sh +# Set up BLS entries for future kernel updates +set -e +apt-get update +apt-get install -y systemd-boot-efi +mkdir -p /boot/loader/entries +# Force kernel-install's BLS layout: with the default layout=auto +# the heuristic can pick 'other' on Ubuntu and skip the +# 90-loaderentry.install plugin +grep -q '^layout=' /etc/kernel/install.conf 2>/dev/null \\ + || echo layout=bls >> /etc/kernel/install.conf +for k in /usr/lib/modules/*; do + v=$(basename "$k") + kernel-install add "$v" "/boot/vmlinuz-$v" || true +done +touch /var/lib/ubuntu-iso-to-uboot-bls-setup.done +''' + +FIRST_BOOT_UNIT = '''\ +[Unit] +Description=Set up BLS entries for future kernel updates +ConditionPathExists=!/var/lib/ubuntu-iso-to-uboot-bls-setup.done +ConditionPathExists=!/cdrom +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/ubuntu-iso-to-uboot-bls-setup +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target +''' + def _quiet() -> bool: """True when tout is set below INFO (no progress chatter requested).""" @@ -180,16 +224,17 @@ def build_esp(esp_path: Path, size_mib: int, uboot_efi: Path) -> None: def repack_iso( in_iso: Path, out_iso: Path, esp_img: Path, esp_guid: str, - entry_conf: Path, + file_maps: list[tuple[Path, str]], ) -> None: """Stream the input ISO to a new ISO with the ESP and BLS entry replaced. -boot_image any replay preserves every other boot record (BIOS El Torito, grub2 MBR, GPT layout); only the bytes behind partition 2 are rewritten, - plus /loader/entry.conf is added to the ISO 9660 tree, and the shim, - GRUB and MokManager copies under /EFI/boot/ are removed since U-Boot - supplies the UEFI boot path via the appended ESP. The BIOS El Torito - path still uses /boot/grub/ so legacy boot continues to work. + plus each (src, dst) pair in @file_maps is added to the ISO 9660 tree, + and the shim, GRUB and MokManager copies under /EFI/boot/ are removed + since U-Boot supplies the UEFI boot path via the appended ESP. The + BIOS El Torito path still uses /boot/grub/ so legacy boot continues + to work. -find is tolerant of missing files: if a distribution does not ship one of these binaries, the call is a no-op. @@ -200,17 +245,131 @@ def repack_iso( """ if out_iso.exists(): out_iso.unlink() - _run( + cmd = [ 'xorriso', '-indev', str(in_iso), '-outdev', str(out_iso), '-boot_image', 'any', 'replay', '-append_partition', '2', esp_guid, str(esp_img), - '-map', str(entry_conf), '/loader/entry.conf', + ] + for src, dst in file_maps: + cmd += ['-map', str(src), dst] + cmd += [ '-find', '/EFI/boot', '-name', 'bootx64.efi', '-exec', 'rm', '--', '-find', '/EFI/boot', '-name', 'grubx64.efi', '-exec', 'rm', '--', '-find', '/EFI/boot', '-name', 'mmx64.efi', '-exec', 'rm', '--', '-commit', + ] + _run(*cmd) + + +def inject_first_boot_unit( + iso: Path, sqfs_in_iso: str, work: Path, +) -> Path: + """Unpack the install squashfs, drop in a first-boot BLS-setup unit + plus its helper script, and repack. + + The whole unpack/modify/repack cycle runs under one fakeroot + invocation so ownership survives round-tripping through a regular + user filesystem. Compression matches the original (xz) to keep the + resulting ISO close to the source in size; that does mean the + rewrite takes several minutes. + + Returns the path to the modified squashfs. + """ + extracted = work / 'orig.squashfs' + modified = work / 'modified.squashfs' + stage = work / 'sqfs-stage' + aux = work / 'aux' + aux.mkdir() + script_src = aux / 'ubuntu-iso-to-uboot-bls-setup' + script_src.write_text(FIRST_BOOT_SCRIPT) + script_src.chmod(0o755) + unit_src = aux / 'ubuntu-iso-to-uboot-bls-setup.service' + unit_src.write_text(FIRST_BOOT_UNIT) + + tout.notice(f'=> Extracting {sqfs_in_iso}') + _run( + 'xorriso', '-osirrox', 'on', '-indev', str(iso), + '-extract', '/' + sqfs_in_iso.lstrip('/'), str(extracted), + ) + + tout.notice('=> Unpacking, injecting unit, repacking (xz, slow)') + shell = f''' +set -e +unsquashfs -d '{stage}' '{extracted}' +install -D -m 755 -o root -g root '{script_src}' \\ + '{stage}/usr/local/sbin/ubuntu-iso-to-uboot-bls-setup' +install -D -m 644 -o root -g root '{unit_src}' \\ + '{stage}/etc/systemd/system/ubuntu-iso-to-uboot-bls-setup.service' +mkdir -p '{stage}/etc/systemd/system/multi-user.target.wants' +ln -sf ../ubuntu-iso-to-uboot-bls-setup.service \\ + '{stage}/etc/systemd/system/multi-user.target.wants/ubuntu-iso-to-uboot-bls-setup.service' +chown -h root:root \\ + '{stage}/etc/systemd/system/multi-user.target.wants/ubuntu-iso-to-uboot-bls-setup.service' +mksquashfs '{stage}' '{modified}' -noappend -comp xz -no-progress +''' + _run('fakeroot', 'sh', '-c', shell) + return modified + + +def autoinstall_yaml() -> str: + """Return an autoinstall snippet that seeds BLS entries on the + installed ESP. + + The Ubuntu installer (ubiquity/subiquity) reads /autoinstall.yaml from the + install media when autoinstall mode is invoked; the file is ignored in plain + interactive installs, so shipping it by default is harmless. When + autoinstall runs, the late-commands here write a BLS Type #1 entry to + /boot/efi/loader/entries/<v>.conf for every kernel the installer has + unpacked and copy the kernel and initrd alongside on the ESP, so U-Boot's + bootmeth_bls finds the entry on the ESP partition before the EFI bootmeth on + the same partition chain-loads U-Boot's BOOTX64.EFI fallback into a loop. + + The kernel-install path is bypassed because curtin's chroot has /boot/efi as + a plain directory rather than the ESP mountpoint, so systemd-boot-efi's + 90-loaderentry.install plugin silently exits without writing entries. + """ + return ( + '#cloud-config\n' + 'autoinstall:\n' + ' version: 1\n' + ' late-commands:\n' + # Write BLS entries to the install target's ESP. The kernel + # and initrd are copied alongside on the ESP so the entry can + # reference them by ESP-relative paths. We place the entries + # on the ESP rather than /boot/loader/entries on the rootfs + # because U-Boot's bootflow iterator advances per-partition: + # the ESP is the lower-numbered partition and EFI bootmeth + # finds our BOOTX64.EFI fallback there before BLS reaches the + # rootfs further down. Putting the entries on the ESP lets + # BLS win on that partition before EFI has a turn. We also + # write a /loader/entry.conf (singular) fallback matching the + # format the live ISO already uses, so U-Boot picks the entry + # whether its scan prefers the entries/ directory or the + # singular file. + ''' - | + curtin in-target -- sh -c 'set -e;\ + ESP=/boot/efi;\ + mkdir -p "$ESP/loader/entries";\ + uuid=$(findmnt -no UUID /);\ + last_v="";\ + for k in /usr/lib/modules/*; do\ + v=$(basename "$k");\ + cp "/boot/vmlinuz-$v" "$ESP/vmlinuz-$v";\ + cp "/boot/initrd.img-$v" "$ESP/initrd.img-$v";\ + printf "title Ubuntu (%s)\\n\ +linux /vmlinuz-%s\\n\ +initrd /initrd.img-%s\\n\ +options root=UUID=%s ro console=ttyS0,115200 console=tty0\\n"\ + "$v" "$v" "$v" "$uuid"\ + > "$ESP/loader/entries/$v.conf";\ + last_v="$v";\ + done;\ + if [ -n "$last_v" ]; then\ + cp "$ESP/loader/entries/$last_v.conf" "$ESP/loader/entry.conf";\ + fi' +''' ) @@ -234,6 +393,15 @@ def main() -> None: help='BLS entry title (default: derived from volume label)') p.add_argument('-s', '--esp-size', type=int, default=None, help='ESP size in MiB (default: 4 MiB)') + p.add_argument('-N', '--no-target-bls', action='store_true', + help='do not ship an autoinstall snippet or modify ' + 'the install squashfs to wire kernel-install + ' + 'BLS into the installed system') + p.add_argument('-I', '--install-squashfs', + default='casper/minimal.squashfs', + help='path inside the ISO of the install squashfs to ' + 'modify for interactive installs ' + '(default: %(default)s; set to empty to skip)') p.add_argument('-v', '--verbose', action='store_true', help='show progress markers and subprocess output') args = p.parse_args() @@ -275,8 +443,21 @@ def main() -> None: f'options {cmdline}\n' ) + file_maps = [(entry, '/loader/entry.conf')] + if not args.no_target_bls: + ai = work / 'autoinstall.yaml' + ai.write_text(autoinstall_yaml()) + file_maps.append((ai, '/autoinstall.yaml')) + if args.install_squashfs: + modified_sqfs = inject_first_boot_unit( + args.iso, args.install_squashfs, work, + ) + file_maps.append( + (modified_sqfs, '/' + args.install_squashfs.lstrip('/')), + ) + tout.notice(f'=> Repacking to {args.out}') - repack_iso(args.iso, args.out, esp, esp_guid, entry) + repack_iso(args.iso, args.out, esp, esp_guid, file_maps) size_mib = args.out.stat().st_size / MIB tout.notice(f'=> Done: {args.out} ({size_mib:.1f} MiB)') -- 2.43.0
From: Simon Glass <sjg@chromium.org> The rewritten ISO already ships /autoinstall.yaml, but the snippet only declares packages and late-commands. Anyone invoking subiquity with the 'autoinstall' kernel argument hits a halt on the missing identity and storage sections, so there is no way to drive an end-to-end install from the test harness. Add -A / --autoinstall. When set, the script: - appends 'autoinstall' to the BLS kernel cmdline so subiquity picks up /autoinstall.yaml on first boot - emits a complete autoinstall.yaml with interactive-sections: [], identity (hostname, username, hashed password), storage layout (direct), refresh-installer disabled, and openssh-server so a test can SSH in afterwards The password is supplied as plaintext via --password and hashed with 'openssl passwd -6' at build time, keeping the script's runtime dependencies unchanged - Python's crypt module is gone in 3.13. --hostname and --username round out the identity block. Guard against --autoinstall being combined with --no-target-bls (the yaml is mandatory in unattended mode) and fail early if openssl is missing. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 94 +++++++++++++++++++++++++++++++--- 1 file changed, 88 insertions(+), 6 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index 1c09e226d16..c22bd9423f2 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -65,6 +65,7 @@ import argparse import os import re import shutil +import subprocess import sys import tempfile from pathlib import Path @@ -313,7 +314,26 @@ mksquashfs '{stage}' '{modified}' -noappend -comp xz -no-progress return modified -def autoinstall_yaml() -> str: +def hash_password(password: str) -> str: + """Return a SHA-512 crypt hash for @password via openssl passwd -6. + + Subiquity's identity.password field wants a crypt(3) hash, not a plaintext + password. Shelling out to openssl keeps the script's dependency list + unchanged (Python's crypt module is gone in 3.13). + """ + out = subprocess.run( + ['openssl', 'passwd', '-6', '-stdin'], + input=password, capture_output=True, text=True, check=True, + ) + return out.stdout.strip() + + +def autoinstall_yaml( + unattended: bool = False, + hostname: str = 'ubuntu-uboot', + username: str = 'ubuntu', + password_hash: str = '', +) -> str: """Return an autoinstall snippet that seeds BLS entries on the installed ESP. @@ -329,11 +349,33 @@ def autoinstall_yaml() -> str: The kernel-install path is bypassed because curtin's chroot has /boot/efi as a plain directory rather than the ESP mountpoint, so systemd-boot-efi's 90-loaderentry.install plugin silently exits without writing entries. + + When @unattended is True, also emit identity and storage sections so + subiquity can complete without user input - required for the + test_distro_ubuntu_iso_install CI path. """ - return ( - '#cloud-config\n' - 'autoinstall:\n' - ' version: 1\n' + head = '''\ +#cloud-config +autoinstall: + version: 1 +''' + unattended_block = '' + if unattended: + unattended_block = f'''\ + interactive-sections: [] + refresh-installer: + update: no + identity: + hostname: {hostname} + username: {username} + password: '{password_hash}' + storage: + layout: + name: direct + ssh: + install-server: true +''' + body = ( ' late-commands:\n' # Write BLS entries to the install target's ESP. The kernel # and initrd are copied alongside on the ESP so the entry can @@ -371,6 +413,7 @@ options root=UUID=%s ro console=ttyS0,115200 console=tty0\\n"\ fi' ''' ) + return head + unattended_block + body def main() -> None: @@ -402,6 +445,20 @@ def main() -> None: help='path inside the ISO of the install squashfs to ' 'modify for interactive installs ' '(default: %(default)s; set to empty to skip)') + p.add_argument('-A', '--autoinstall', action='store_true', + help='build for unattended autoinstall: append ' + '"autoinstall" to the BLS cmdline and ship a complete ' + 'autoinstall.yaml so subiquity runs without user input') + p.add_argument('--hostname', default='ubuntu-uboot', + help='hostname for the autoinstalled system ' + '(default: %(default)s)') + p.add_argument('--username', default='ubuntu', + help='username created by autoinstall ' + '(default: %(default)s)') + p.add_argument('--password', default='ubuntu', + help='plaintext password for the autoinstall user; ' + 'hashed with `openssl passwd -6` before being ' + 'written to autoinstall.yaml (default: %(default)s)') p.add_argument('-v', '--verbose', action='store_true', help='show progress markers and subprocess output') args = p.parse_args() @@ -414,7 +471,11 @@ def main() -> None: tout.fatal(f'ISO not found: {args.iso}') if not args.uboot.is_file(): tout.fatal(f'EFI app not found: {args.uboot}') + if args.autoinstall and args.no_target_bls: + tout.fatal('--autoinstall requires target-BLS wiring; drop -N') check_tools() + if args.autoinstall and not shutil.which('openssl'): + tout.fatal('openssl is required when --autoinstall is set') tout.notice(f'=> Reading boot config from {args.iso}') # Extract the volume label and ESP partition GUID from xorriso's report @@ -423,6 +484,19 @@ def main() -> None: cmdline = args.cmdline if cmdline is None: cmdline = parse_grub_cmdline(args.iso, args.kernel) + if args.autoinstall: + # Force subiquity onto the serial console so the CI harness can watch + # progress: drop the live-ISO 'quiet splash' (kernel would otherwise + # silently swallow the reboot line), pin the console to ttyS0, and ask + # systemd-journald to mirror its stream onto the console so subiquity's + # own events land on serial too. + tokens = [t for t in cmdline.split() if t not in ('quiet', 'splash')] + for extra in ('console=ttyS0,115200', + 'systemd.journald.forward_to_console=1', + 'autoinstall'): + if extra not in tokens: + tokens.append(extra) + cmdline = ' '.join(tokens) tout.notice(f' Volume label: {vol_id}') tout.notice(f' ESP GUID: {esp_guid}') tout.notice(f' Cmdline: {cmdline}') @@ -446,7 +520,15 @@ def main() -> None: file_maps = [(entry, '/loader/entry.conf')] if not args.no_target_bls: ai = work / 'autoinstall.yaml' - ai.write_text(autoinstall_yaml()) + if args.autoinstall: + ai.write_text(autoinstall_yaml( + unattended=True, + hostname=args.hostname, + username=args.username, + password_hash=hash_password(args.password), + )) + else: + ai.write_text(autoinstall_yaml()) file_maps.append((ai, '/autoinstall.yaml')) if args.install_squashfs: modified_sqfs = inject_first_boot_unit( -- 2.43.0
From: Simon Glass <sjg@chromium.org> Once subiquity finishes the install it has written shim and GRUB to the target ESP and registered an 'ubuntu' NVRAM entry pointing at /EFI/ubuntu/shimx64.efi. Firmware prefers that entry over the removable-media fallback, so control leaves U-Boot as soon as the install reboots - and once the live ISO is ejected there is no U-Boot on the installed system at all. Plant u-boot.efi at /usr/lib/u-boot/u-boot.efi inside the modified install squashfs so it rides through the install, then copy it onto the target ESP at two locations: - /EFI/BOOT/BOOTX64.EFI - the firmware fallback path - /EFI/ubuntu/shimx64.efi - what the 'ubuntu' NVRAM entry points at Overwriting both means the installed disk boots U-Boot whichever path firmware takes. BLS then picks up /loader/entries/ on the same ESP and the installed kernel/initrd come up without the ISO. The autoinstall late-commands do the copy via 'curtin in-target' so the first reboot after install lands in U-Boot directly. The interactive first-boot unit does the same from the running installed system. Both paths read u-boot.efi from /usr/lib/u-boot/u-boot.efi inside the running install, so refuse --autoinstall when the squashfs is not being modified (--install-squashfs ''); otherwise the late-commands would silently fail to install U-Boot on the target. Signed-off-by: Simon Glass <sjg@chromium.org> --- doc/usage/os/ubuntu-live.rst | 40 +++++++++++++++++++++++-------- scripts/ubuntu-iso-to-uboot.py | 44 ++++++++++++++++++++++++++++++---- 2 files changed, 70 insertions(+), 14 deletions(-) diff --git a/doc/usage/os/ubuntu-live.rst b/doc/usage/os/ubuntu-live.rst index cfcc5243ca2..34d7b0eb71a 100644 --- a/doc/usage/os/ubuntu-live.rst +++ b/doc/usage/os/ubuntu-live.rst @@ -173,22 +173,42 @@ for the installer's kernel: 2. **Interactive path.** The script unpacks the default install squashfs (``casper/minimal.squashfs``), drops in a ``oneshot`` systemd unit plus its - helper script, and repacks. On first boot of the installed system the unit - runs:: - - apt-get update - apt-get install -y systemd-boot-efi - mkdir -p /boot/loader/entries - for k in /usr/lib/modules/*; do - kernel-install add "$(basename "$k")" "$k/vmlinuz" || true - done - touch /var/lib/ubuntu-iso-to-uboot-bls-setup.done + helper script and a copy of ``u-boot.efi`` at ``/usr/lib/u-boot/u-boot.efi``, + and repacks. On first boot of the installed system the unit runs + ``apt install systemd-boot-efi``, back-fills ``/boot/loader/entries/`` via + ``kernel-install``, and installs ``u-boot.efi`` onto the target ESP (see + :ref:`target-uboot-install` below). The unit is gated on ``ConditionPathExists=!/cdrom`` so it stays dormant inside the live session, and the done-file stops it re-running on subsequent boots. It needs a network on first boot for the apt call; if that is not available it fails gracefully and retries on the next boot. +.. _target-uboot-install: + +Installing U-Boot onto the target ESP +------------------------------------- + +Subiquity writes shim and GRUB to the installed ESP and registers an ``ubuntu`` +NVRAM entry pointing at ``/EFI/ubuntu/shimx64.efi``. Without further action, +firmware prefers that entry over the removable-media fallback, so control leaves +U-Boot as soon as the install reboots. + +Both install paths therefore plant ``u-boot.efi`` on the target ESP at two +locations: + +* ``/EFI/BOOT/BOOTX64.EFI`` -- the firmware fallback, picked up when no NVRAM + entry matches (for example after a CMOS reset). +* ``/EFI/ubuntu/shimx64.efi`` -- the file the ``ubuntu`` NVRAM entry points at, + overwritten in place so the entry boots U-Boot rather than chaining through + shim into GRUB. + +The autoinstall path does the copy in its ``late-commands`` via +``curtin in-target``; the interactive first-boot unit does it from the running +installed system. Either way ``u-boot.efi`` travels through the install inside +the modified squashfs at ``/usr/lib/u-boot/u-boot.efi``, so the installed disk +boots U-Boot + BLS on its own once the live ISO has been ejected. + For interactive installs, every subsequent ``apt install linux-image-*`` on the target updates ``/boot/loader/entries/`` automatically via the ``kernel-install`` hooks, and U-Boot's BLS bootmeth keeps picking up the current diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index c22bd9423f2..c431e15b9ae 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -90,7 +90,8 @@ MIB = 1024 * 1024 # subsequent boot. FIRST_BOOT_SCRIPT = '''\ #!/bin/sh -# Set up BLS entries for future kernel updates +# Set up BLS entries and install U-Boot on the target ESP so the +# installed system boots via U-Boot + BLS without the live ISO set -e apt-get update apt-get install -y systemd-boot-efi @@ -104,6 +105,15 @@ for k in /usr/lib/modules/*; do v=$(basename "$k") kernel-install add "$v" "/boot/vmlinuz-$v" || true done +# Plant u-boot.efi on the installed ESP. BOOTX64.EFI is the firmware +# fallback; shimx64.efi is what the 'ubuntu' NVRAM entry Subiquity +# registers points at, so overwriting both means the disk boots +# U-Boot whichever path the firmware takes. +UBOOT=/usr/lib/u-boot/u-boot.efi +install -D -m 644 "$UBOOT" /boot/efi/EFI/BOOT/BOOTX64.EFI +if [ -f /boot/efi/EFI/ubuntu/shimx64.efi ]; then + install -m 644 "$UBOOT" /boot/efi/EFI/ubuntu/shimx64.efi +fi touch /var/lib/ubuntu-iso-to-uboot-bls-setup.done ''' @@ -265,10 +275,15 @@ def repack_iso( def inject_first_boot_unit( - iso: Path, sqfs_in_iso: str, work: Path, + iso: Path, sqfs_in_iso: str, uboot_efi: Path, work: Path, ) -> Path: """Unpack the install squashfs, drop in a first-boot BLS-setup unit - plus its helper script, and repack. + plus its helper script and a copy of u-boot.efi, and repack. + + u-boot.efi travels through the install at /usr/lib/u-boot/u-boot.efi + so the first-boot unit (and the autoinstall late-commands, which + also run in-target after the squashfs has been unpacked) can copy + it onto the installed ESP. The whole unpack/modify/repack cycle runs under one fakeroot invocation so ownership survives round-tripping through a regular @@ -303,6 +318,8 @@ install -D -m 755 -o root -g root '{script_src}' \\ '{stage}/usr/local/sbin/ubuntu-iso-to-uboot-bls-setup' install -D -m 644 -o root -g root '{unit_src}' \\ '{stage}/etc/systemd/system/ubuntu-iso-to-uboot-bls-setup.service' +install -D -m 644 -o root -g root '{uboot_efi}' \\ + '{stage}/usr/lib/u-boot/u-boot.efi' mkdir -p '{stage}/etc/systemd/system/multi-user.target.wants' ln -sf ../ubuntu-iso-to-uboot-bls-setup.service \\ '{stage}/etc/systemd/system/multi-user.target.wants/ubuntu-iso-to-uboot-bls-setup.service' @@ -411,6 +428,19 @@ options root=UUID=%s ro console=ttyS0,115200 console=tty0\\n"\ if [ -n "$last_v" ]; then\ cp "$ESP/loader/entries/$last_v.conf" "$ESP/loader/entry.conf";\ fi' +''' + # Plant u-boot.efi on the installed ESP in-target, overwriting + # both the fallback BOOTX64.EFI and the shimx64.efi that the + # 'ubuntu' NVRAM entry points at, so the disk boots U-Boot + # whichever path firmware takes - no need to keep the ISO + # attached after the install reboot. + ''' - curtin in-target -- install -D -m 644\ + /usr/lib/u-boot/u-boot.efi /boot/efi/EFI/BOOT/BOOTX64.EFI + - | + curtin in-target -- sh -c '[ -f\ + /boot/efi/EFI/ubuntu/shimx64.efi ] && install -m 644\ + /usr/lib/u-boot/u-boot.efi\ + /boot/efi/EFI/ubuntu/shimx64.efi; true' ''' ) return head + unattended_block + body @@ -473,6 +503,12 @@ def main() -> None: tout.fatal(f'EFI app not found: {args.uboot}') if args.autoinstall and args.no_target_bls: tout.fatal('--autoinstall requires target-BLS wiring; drop -N') + if args.autoinstall and not args.install_squashfs: + tout.fatal( + '--autoinstall needs --install-squashfs; the autoinstall ' + "late-commands copy u-boot.efi from /usr/lib/u-boot/u-boot.efi, " + 'which is only planted when the install squashfs is modified' + ) check_tools() if args.autoinstall and not shutil.which('openssl'): tout.fatal('openssl is required when --autoinstall is set') @@ -532,7 +568,7 @@ def main() -> None: file_maps.append((ai, '/autoinstall.yaml')) if args.install_squashfs: modified_sqfs = inject_first_boot_unit( - args.iso, args.install_squashfs, work, + args.iso, args.install_squashfs, args.uboot, work, ) file_maps.append( (modified_sqfs, '/' + args.install_squashfs.lstrip('/')), -- 2.43.0
From: Simon Glass <sjg@chromium.org> The script targets Python 3.13 (its hash_password() relies on the crypt module being gone), so the PEP 563 future-import is a no-op: every annotation in the file already works under 3.13's runtime evaluator. Drop the line so it is not mistaken for a portability requirement. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index c431e15b9ae..3855f4a25ae 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -59,8 +59,6 @@ Assumptions: - xorriso, mtools and dosfstools are installed on the host """ -from __future__ import annotations - import argparse import os import re -- 2.43.0
From: Simon Glass <sjg@chromium.org> The quickstart hard-codes a ubuntu-24.04.1 ISO URL that 404s once Canonical promotes a new point release. Point at the stable https://releases.ubuntu.com/noble/ directory listing instead and hoist the filename into a $ISO shell variable so the curl line and the rewrite invocation track each other when the reader updates it. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index 3855f4a25ae..68ba4cd3d97 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -26,8 +26,11 @@ Quick start (run from the root of the U-Boot tree):: # 1. Install host tools sudo apt install xorriso mtools dosfstools qemu-system-x86 ovmf - # 2. Download an Ubuntu live ISO (desktop or server both work) - curl -LO https://releases.ubuntu.com/24.04.1/ubuntu-24.04.1-desktop-amd64.iso + # 2. Download an Ubuntu live ISO (desktop or server both work). + # Pick the current point release listed at + # https://releases.ubuntu.com/noble/ + ISO=ubuntu-24.04.1-desktop-amd64.iso + curl -LO https://releases.ubuntu.com/noble/$ISO # 3. Build U-Boot as an x86_64 EFI application. The defconfig enables # BOOTMETH_BLS, FS_ISOFS and JOLIET. If rustc is not installed, @@ -38,7 +41,7 @@ Quick start (run from the root of the U-Boot tree):: # produces /tmp/b/efi-x86_app64/u-boot-app.efi # 4. Rewrite the ISO to boot via U-Boot - scripts/ubuntu-iso-to-uboot.py ubuntu-24.04.1-desktop-amd64.iso \\ + scripts/ubuntu-iso-to-uboot.py $ISO \\ -u /tmp/b/efi-x86_app64/u-boot-app.efi \\ -o ubuntu-uboot.iso -- 2.43.0
From: Simon Glass <sjg@chromium.org> The rest of the script funnels every external invocation through u_boot_pylib's command.run(), so its CommandExc carries the captured output on failure and the -v/-quiet wiring keeps subprocess chatter out of the default run. hash_password() is the lone holdout, going straight to subprocess.run() to feed the plaintext password through stdin. command.run() takes the same data via stdin_data= and returns the captured stdout as a string when capture=True, so the conversion keeps the no-arg-on-the-command-line property of the password input and lets us drop the subprocess import altogether. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index 68ba4cd3d97..d43308676e3 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -66,7 +66,6 @@ import argparse import os import re import shutil -import subprocess import sys import tempfile from pathlib import Path @@ -339,11 +338,10 @@ def hash_password(password: str) -> str: password. Shelling out to openssl keeps the script's dependency list unchanged (Python's crypt module is gone in 3.13). """ - out = subprocess.run( - ['openssl', 'passwd', '-6', '-stdin'], - input=password, capture_output=True, text=True, check=True, - ) - return out.stdout.strip() + return command.run( + 'openssl', 'passwd', '-6', '-stdin', + capture=True, stdin_data=password, + ).strip() def autoinstall_yaml( -- 2.43.0
From: Simon Glass <sjg@chromium.org> The first-boot helper script's name, its done-file path and the in-target u-boot.efi location each appear in three places: the two shell heredocs (FIRST_BOOT_SCRIPT, FIRST_BOOT_UNIT) and the shell template inside inject_first_boot_unit() and autoinstall_yaml(). Renaming any of them today means hunting through six call sites and getting them all to agree, which is exactly the kind of edit that silently desyncs. Hoist BLS_SETUP_NAME, BLS_SETUP_DONE and TARGET_UBOOT_EFI to module scope and switch the heredocs to f-strings (no '{' in the original shell, so no escaping needed). Output is byte-identical: the rewrite just makes future renames a one-line change. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 49 +++++++++++++++++++++------------- 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index d43308676e3..ba11aedc89c 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -83,12 +83,23 @@ REQUIRED_TOOLS = ( ) MIB = 1024 * 1024 +# Name shared by the first-boot helper script (under /usr/local/sbin) +# and its systemd unit (with .service appended); also used as the +# stem of the done-file under /var/lib that gates the unit. +BLS_SETUP_NAME = 'ubuntu-iso-to-uboot-bls-setup' +BLS_SETUP_DONE = f'/var/lib/{BLS_SETUP_NAME}.done' + +# Where u-boot.efi lands inside the install squashfs. The first-boot +# unit, the autoinstall late-commands and inject_first_boot_unit() +# all read or write this path, so keep them in sync via this constant. +TARGET_UBOOT_EFI = '/usr/lib/u-boot/u-boot.efi' + # First-boot systemd unit + helper script written into the install # squashfs so kernel-install manages BLS entries on the installed # system. ConditionPathExists=!/cdrom keeps the unit dormant inside # the live session, and the done-file stops it re-running on every # subsequent boot. -FIRST_BOOT_SCRIPT = '''\ +FIRST_BOOT_SCRIPT = f'''\ #!/bin/sh # Set up BLS entries and install U-Boot on the target ESP so the # installed system boots via U-Boot + BLS without the live ISO @@ -109,25 +120,25 @@ done # fallback; shimx64.efi is what the 'ubuntu' NVRAM entry Subiquity # registers points at, so overwriting both means the disk boots # U-Boot whichever path the firmware takes. -UBOOT=/usr/lib/u-boot/u-boot.efi +UBOOT={TARGET_UBOOT_EFI} install -D -m 644 "$UBOOT" /boot/efi/EFI/BOOT/BOOTX64.EFI if [ -f /boot/efi/EFI/ubuntu/shimx64.efi ]; then install -m 644 "$UBOOT" /boot/efi/EFI/ubuntu/shimx64.efi fi -touch /var/lib/ubuntu-iso-to-uboot-bls-setup.done +touch {BLS_SETUP_DONE} ''' -FIRST_BOOT_UNIT = '''\ +FIRST_BOOT_UNIT = f'''\ [Unit] Description=Set up BLS entries for future kernel updates -ConditionPathExists=!/var/lib/ubuntu-iso-to-uboot-bls-setup.done +ConditionPathExists=!{BLS_SETUP_DONE} ConditionPathExists=!/cdrom After=network-online.target Wants=network-online.target [Service] Type=oneshot -ExecStart=/usr/local/sbin/ubuntu-iso-to-uboot-bls-setup +ExecStart=/usr/local/sbin/{BLS_SETUP_NAME} RemainAfterExit=yes [Install] @@ -298,10 +309,10 @@ def inject_first_boot_unit( stage = work / 'sqfs-stage' aux = work / 'aux' aux.mkdir() - script_src = aux / 'ubuntu-iso-to-uboot-bls-setup' + script_src = aux / BLS_SETUP_NAME script_src.write_text(FIRST_BOOT_SCRIPT) script_src.chmod(0o755) - unit_src = aux / 'ubuntu-iso-to-uboot-bls-setup.service' + unit_src = aux / f'{BLS_SETUP_NAME}.service' unit_src.write_text(FIRST_BOOT_UNIT) tout.notice(f'=> Extracting {sqfs_in_iso}') @@ -311,20 +322,22 @@ def inject_first_boot_unit( ) tout.notice('=> Unpacking, injecting unit, repacking (xz, slow)') + unit_in_etc = f'/etc/systemd/system/{BLS_SETUP_NAME}.service' + wants_link = ( + f'/etc/systemd/system/multi-user.target.wants/{BLS_SETUP_NAME}.service' + ) shell = f''' set -e unsquashfs -d '{stage}' '{extracted}' install -D -m 755 -o root -g root '{script_src}' \\ - '{stage}/usr/local/sbin/ubuntu-iso-to-uboot-bls-setup' + '{stage}/usr/local/sbin/{BLS_SETUP_NAME}' install -D -m 644 -o root -g root '{unit_src}' \\ - '{stage}/etc/systemd/system/ubuntu-iso-to-uboot-bls-setup.service' + '{stage}{unit_in_etc}' install -D -m 644 -o root -g root '{uboot_efi}' \\ - '{stage}/usr/lib/u-boot/u-boot.efi' + '{stage}{TARGET_UBOOT_EFI}' mkdir -p '{stage}/etc/systemd/system/multi-user.target.wants' -ln -sf ../ubuntu-iso-to-uboot-bls-setup.service \\ - '{stage}/etc/systemd/system/multi-user.target.wants/ubuntu-iso-to-uboot-bls-setup.service' -chown -h root:root \\ - '{stage}/etc/systemd/system/multi-user.target.wants/ubuntu-iso-to-uboot-bls-setup.service' +ln -sf ../{BLS_SETUP_NAME}.service '{stage}{wants_link}' +chown -h root:root '{stage}{wants_link}' mksquashfs '{stage}' '{modified}' -noappend -comp xz -no-progress ''' _run('fakeroot', 'sh', '-c', shell) @@ -433,12 +446,12 @@ options root=UUID=%s ro console=ttyS0,115200 console=tty0\\n"\ # 'ubuntu' NVRAM entry points at, so the disk boots U-Boot # whichever path firmware takes - no need to keep the ISO # attached after the install reboot. - ''' - curtin in-target -- install -D -m 644\ - /usr/lib/u-boot/u-boot.efi /boot/efi/EFI/BOOT/BOOTX64.EFI + f''' - curtin in-target -- install -D -m 644\ + {TARGET_UBOOT_EFI} /boot/efi/EFI/BOOT/BOOTX64.EFI - | curtin in-target -- sh -c '[ -f\ /boot/efi/EFI/ubuntu/shimx64.efi ] && install -m 644\ - /usr/lib/u-boot/u-boot.efi\ + {TARGET_UBOOT_EFI}\ /boot/efi/EFI/ubuntu/shimx64.efi; true' ''' ) -- 2.43.0
From: Simon Glass <sjg@chromium.org> build_esp() already documents its parameters with the Google-style Args: block u_boot_pylib uses, but the rest of the script only hand-waves about its arguments in prose (when it documents them at all). That makes generated help and IDE tooltips uneven and forces a reader to grep the body to find out what each parameter means. Add Args:/Returns:/Raises: sections to every public helper and to _run(); also give check_tools() a one-line summary it had been missing. No behaviour change. Signed-off-by: Simon Glass <sjg@chromium.org> --- scripts/ubuntu-iso-to-uboot.py | 71 ++++++++++++++++++++++++++++++---- 1 file changed, 64 insertions(+), 7 deletions(-) diff --git a/scripts/ubuntu-iso-to-uboot.py b/scripts/ubuntu-iso-to-uboot.py index ba11aedc89c..8dcd37fda82 100755 --- a/scripts/ubuntu-iso-to-uboot.py +++ b/scripts/ubuntu-iso-to-uboot.py @@ -156,12 +156,16 @@ def _run(*cmd) -> None: On failure u_boot_pylib's CommandExc still carries the captured output, so the user sees what went wrong. + + Args: + *cmd: Command and its arguments, as separate strings """ quiet = _quiet() command.run(*cmd, capture=quiet, capture_stderr=quiet) def check_tools() -> None: + """Abort with a helpful message if any REQUIRED_TOOLS binary is missing.""" missing = [t for t in REQUIRED_TOOLS if not shutil.which(t)] if missing: tout.fatal( @@ -176,6 +180,13 @@ def parse_grub_cmdline(iso: Path, kernel: str) -> str: Parses the first `linux <kernel> ...` line in /boot/grub/grub.cfg and strips the kernel path, so the caller can pass the remaining tokens to the kernel (e.g. '--- quiet splash' on Ubuntu 24.04.1). + + Args: + iso (Path): Input ISO to inspect + kernel (str): Kernel path inside the ISO to look for in grub.cfg + + Returns: + str: Kernel cmdline tokens, with whitespace collapsed """ with tempfile.TemporaryDirectory(prefix='iso2uboot.grub.') as td: dst = Path(td) / 'grub.cfg' @@ -203,8 +214,16 @@ def parse_grub_cmdline(iso: Path, kernel: str) -> str: def parse_boot_report(iso: Path) -> tuple[str, str]: """Return (volume_label, esp_partition_guid) from xorriso's mkisofs report. - Raises SystemExit if the ISO does not have an appended EFI System - Partition on slot 2 + Args: + iso (Path): Input ISO to inspect + + Returns: + tuple[str, str]: (volume label, partition-2 GUID) parsed from + xorriso's '-report_el_torito as_mkisofs' output + + Raises: + SystemExit: if the ISO does not have an appended EFI System + Partition on slot 2 """ # xorriso prints the report on stdout (which we need to parse) and # progress/status on stderr (which we swallow unless -v was passed). @@ -264,6 +283,15 @@ def repack_iso( xorriso refuses to write to an existing non-empty file when -indev and -outdev differ (it would treat the outdev as a session to extend), so unlink any stale output first. + + Args: + in_iso (Path): Source ISO to read + out_iso (Path): Destination ISO; unlinked first if it exists + esp_img (Path): FAT image to splice in as the appended partition + esp_guid (str): GPT type GUID for the appended partition (from + parse_boot_report()) + file_maps (list[tuple[Path, str]]): (src, dst) pairs added to the + ISO 9660 tree via xorriso -map """ if out_iso.exists(): out_iso.unlink() @@ -291,10 +319,10 @@ def inject_first_boot_unit( """Unpack the install squashfs, drop in a first-boot BLS-setup unit plus its helper script and a copy of u-boot.efi, and repack. - u-boot.efi travels through the install at /usr/lib/u-boot/u-boot.efi - so the first-boot unit (and the autoinstall late-commands, which - also run in-target after the squashfs has been unpacked) can copy - it onto the installed ESP. + u-boot.efi travels through the install at TARGET_UBOOT_EFI so the + first-boot unit (and the autoinstall late-commands, which also run + in-target after the squashfs has been unpacked) can copy it onto + the installed ESP. The whole unpack/modify/repack cycle runs under one fakeroot invocation so ownership survives round-tripping through a regular @@ -302,7 +330,16 @@ def inject_first_boot_unit( resulting ISO close to the source in size; that does mean the rewrite takes several minutes. - Returns the path to the modified squashfs. + Args: + iso (Path): Input ISO containing the install squashfs to extract + sqfs_in_iso (str): Path of the install squashfs inside @iso + (e.g. 'casper/minimal.squashfs') + uboot_efi (Path): U-Boot EFI app to plant inside the squashfs at + TARGET_UBOOT_EFI + work (Path): Scratch directory for the unpack/repack staging + + Returns: + Path: Modified squashfs ready to splice back into the ISO """ extracted = work / 'orig.squashfs' modified = work / 'modified.squashfs' @@ -350,6 +387,13 @@ def hash_password(password: str) -> str: Subiquity's identity.password field wants a crypt(3) hash, not a plaintext password. Shelling out to openssl keeps the script's dependency list unchanged (Python's crypt module is gone in 3.13). + + Args: + password (str): Plaintext password, fed via stdin so it never appears + on the openssl command line + + Returns: + str: SHA-512 crypt hash, with trailing newline stripped """ return command.run( 'openssl', 'passwd', '-6', '-stdin', @@ -382,6 +426,19 @@ def autoinstall_yaml( When @unattended is True, also emit identity and storage sections so subiquity can complete without user input - required for the test_distro_ubuntu_iso_install CI path. + + Args: + unattended (bool): True to emit the identity/storage/ssh sections + needed for an end-to-end autoinstall + hostname (str): Hostname for the autoinstalled system + (used only when @unattended) + username (str): Login user created by autoinstall + (used only when @unattended) + password_hash (str): SHA-512 crypt hash for that user, as + produced by hash_password() (used only when @unattended) + + Returns: + str: autoinstall.yaml document body """ head = '''\ #cloud-config -- 2.43.0
From: Simon Glass <sjg@chromium.org> The existing test_distro_ubuntu_iso_uboot exercises the live-ISO path only. It leaves the post-install half of the BLS wiring - the kernel-install plugin run by autoinstall late-commands and the first-boot systemd unit in the install squashfs - unverified. Add a companion test that drives the full unattended install. The test expects a role (efi-x86_64-uboot-iso-install) whose writer passes --autoinstall to scripts/ubuntu-iso-to-uboot.py and whose QEMU config attaches a blank target disk. It watches for: - U-Boot's BLS bootflow and the kernel's 'Linux version' line, confirming subiquity has been handed control - a subiquity reboot marker (or the kernel's reboot line) within a 45 minute window - a second BLS bootflow after reboot, this time picking up the /loader/entries/ kernel-install has written onto the installed ESP rather than the ISO's casper entry - a getty 'login: ' prompt on ttyS0, so the test only passes once the installed system is actually up Signed-off-by: Simon Glass <sjg@chromium.org> --- test/py/tests/test_distro.py | 53 ++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/test/py/tests/test_distro.py b/test/py/tests/test_distro.py index ed3f6d86249..6acc4c200b5 100644 --- a/test/py/tests/test_distro.py +++ b/test/py/tests/test_distro.py @@ -99,6 +99,59 @@ def test_distro_arm_app_extlinux(ubman): ubman.restart_uboot() +@pytest.mark.boardspec('efi-x86_app64') +@pytest.mark.role('efi-x86_64-uboot-iso-install') +@pytest.mark.restart +def test_distro_ubuntu_iso_install(ubman): + """Run an unattended Ubuntu install through U-Boot + BLS. + + Needs a role whose UBootWriterDriver passes --autoinstall to + scripts/ubuntu-iso-to-uboot.py and whose QEMU config attaches a blank virtio + disk as the install target. The rewritten ISO boots straight into subiquity + (autoinstall on the BLS cmdline), subiquity partitions the target disk and + installs Ubuntu, then kernel-install populates /boot/loader/entries/ on the + installed ESP. On reboot, OVMF hands control back to U-Boot (still living on + the ISO ESP), whose BOOTMETH_BLS picks up the new entries from the installed + disk and boots that system. + """ + with ubman.log.section('boot-installer'): + with ubman.temporary_timeout(120 * 1000): + ubman.run_command('boot', wait_for_prompt=False) + ubman.expect([r"Booting bootflow '[^']+' with bls"]) + ubman.expect(['Linux version']) + + # Subiquity sends its reporting events through journald; with + # forward_to_console=1 on the cmdline they appear on ttyS0. Each event is + # logged as 'start: <path>: <description>' (or 'finish: ...'), so wait for + # the start of the curtin_install step, which follows partitioning and + # marks the point where curtin begins copying the system to disk. + with ubman.log.section('subiquity-install'): + with ubman.temporary_timeout(10 * 60 * 1000): + ubman.expect( + [r'start: +subiquity/Install/install/curtin_install:']) + + # The clean end-of-install marker: subiquity finishes, systemd tears the + # session down and the kernel prints 'Restarting system' on the way out. + # 'quiet' has been stripped so this is visible. + with ubman.log.section('subiquity-reboot'): + with ubman.temporary_timeout(45 * 60 * 1000): + ubman.expect([r'reboot: Restarting system']) + + # After reboot OVMF re-runs U-Boot from the ISO ESP. BLS now sees the + # installed disk's /loader/entries/ and should pick the newly written entry + # rather than the ISO's casper one. + with ubman.log.section('boot-installed'): + with ubman.temporary_timeout(300 * 1000): + ubman.expect([r"Booting bootflow '[^']+' with bls"]) + ubman.expect(['Linux version']) + + with ubman.log.section('installed-ubuntu'): + with ubman.temporary_timeout(300 * 1000): + ubman.expect([r' login: ']) + + ubman.restart_uboot() + + @pytest.mark.boardspec('efi-x86_app64') @pytest.mark.role('efi-x86_64-uboot-iso') @pytest.mark.restart -- 2.43.0
From: Simon Glass <sjg@chromium.org> test_distro_ubuntu_iso_install runs a full unattended install of Ubuntu through U-Boot + BLS, which takes 30 to 45 minutes under QEMU. It needs its own stage so the sjg-lab matrix is not held up waiting for it, and its own opt-in variable so SJG_LAB consumers do not pull it in by accident. Introduce a sjg-lab-slow stage running after sjg-lab and a matching .sjg_lab_slow_template. The new template inherits sjg_lab_dfn's script and artifacts but replaces its rules, gating on a new SJG_LAB_SLOW variable rather than SJG_LAB so slow jobs can be opted into without also pulling in the rest of the sjg-lab matrix: - run unconditionally when SJG_LAB_SLOW matches the role name (or is set to '1' for every slow job) - auto-run on merge_request_event, the same as sjg_lab_dfn - on other pipelines (push, schedule), appear as a manual-trigger button with allow_failure so they never block a pipeline - honour [skip-sjg-slow] in the MR description as a slow-only escape hatch (independent of the sjg-lab [skip-sjg] hatch) Add the efi-x86_64-uboot-iso-install job using the new template and set timeout: 2h to cover the install plus first-boot slack. Signed-off-by: Simon Glass <sjg@chromium.org> --- .gitlab-ci.yml | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2d0277f9c27..b850e14a49f 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -21,6 +21,7 @@ variables: WORLD: "1" SAGE_LAB: "" SJG_LAB: "" + SJG_LAB_SLOW: "" PLATFORM: linux/amd64,linux/arm64 GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_RUNNER_ID/$CI_CONCURRENT_ID/u-boot @@ -40,6 +41,7 @@ stages: - test.py - sage-lab - sjg-lab + - sjg-lab-slow - world_build - version_bump - release @@ -899,6 +901,29 @@ coreboot test.py: - "build/${ROLE}/multiplexed_log.css" expire_in: 1 week +# Slow lab jobs (e.g. full distro installs). Same script/artifacts as +# sjg_lab_dfn but in a later stage and gated on its own SJG_LAB_SLOW variable so +# slow jobs can be opted into without pulling in the rest of the sjg-lab matrix. +# Auto-runs on merge_request_event like sjg_lab_dfn; on other pipelines (push, +# schedule) it appears as a manual-trigger button with allow_failure so it never +# blocks a pipeline. Set [skip-sjg-slow] in the MR description to skip just the +# slow jobs (independent of the sjg-lab [skip-sjg] hatch). +.sjg_lab_slow_template: &sjg_lab_slow_dfn + <<: *sjg_lab_dfn + stage: sjg-lab-slow + rules: + - if: $CI_MERGE_REQUEST_DESCRIPTION =~ /\[skip-sjg-slow\]/ + when: never + - if: $SJG_LAB_SLOW == $ROLE || $SJG_LAB_SLOW == "1" + when: always + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + when: always + - if: $SJG_LAB_SLOW != "" && $SJG_LAB_SLOW != "1" && $SJG_LAB_SLOW != $ROLE + when: never + - if: $SJG_LAB_SLOW == "" + when: manual + allow_failure: true + rpi3: variables: ROLE: rpi3 @@ -1062,6 +1087,18 @@ efi-x86_64-uboot-iso: TEST_PY_TEST_SPEC: "and test_distro_ubuntu_iso_uboot" <<: *sjg_lab_dfn +# Full unattended install of Ubuntu: the role's writer builds the ISO with +# --autoinstall, subiquity installs onto a blank target disk, and BLS is +# expected to pick up the kernel entries kernel-install creates on the installed +# ESP. Slow (~30-45 min), so lives in sjg-lab-slow: manual-trigger only on +# MRs; set SJG_LAB_SLOW to the role name (or '1') to force an automatic run. +efi-x86_64-uboot-iso-install: + variables: + ROLE: efi-x86_64-uboot-iso-install + TEST_PY_TEST_SPEC: "and test_distro_ubuntu_iso_install" + timeout: 2h + <<: *sjg_lab_slow_dfn + # NVIDIA Jetson TK1 tk1: variables: -- 2.43.0
participants (1)
-
Simon Glass