On Debian armhf, the imx_sdma driver is currently built-in. It
requests firmware and has the corresponding module metadata, and in
tests we currently generate a warning that this is missing. Any
output to stderr results in a test failure by default. It is possible
to override that, but I think we do generally want this behaviour.
imx_sdma could be built as a module (as it is on arm64), and it might
also make sense to remove the firmware warnings, but those will take
some time to do. As a quick fix to let initramfs-tools to go into
testing, work around this by creating dummy files for any firmware
listed in built-in module metadata.
This fixes the other half of #1078168.
Signed-off-by: Ben Hutchings <benh@debian.org>
The name of the default (currently only) kernel flavour on Debian
ppc64el is "powerpc64le", not "powerpc64". Correct this in the
SUPPORTED_FLAVOURS definitions for various tests.
This fixes half of #1078168.
Signed-off-by: Ben Hutchings <benh@debian.org>
The hook functions and script might be executeted by busybox. Mark those
files as POSIX sh for shellcheck.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Move shellcheck configuration to `.shellcheckrc`. That way editors like
VSCode will pick up the configuration as well.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Debian bug #1076539 (and its many duplicates) involved a firmware
package with a directory structure like:
vendor/model1/
vendor/model1/firmware.bin
vendor/model2 -> model1
vendor/model3 -> model1
...
and the associated kernel module referred to firmware names like
'vendor/model1/firmware.bin', 'vendor/model2/firmware.bin', etc.
copy_file currently checks whether the source itself is a symlink, and
all of the filenames resolve to a regular file, so copy_file would
happily install multiple copies of that.
The particular problem with firmware was fixed by the switch to
dracut-install, but the same problem could occur with other files.
Also, that change is not suitable for backporting.
To handle directory symlinks as well, replace this check with a
comparison of:
- The canonicalised filename with symlinks dereferenced, as produced
by 'readlink -f'.
- The canonicalised filename without symlinks dereferenced, as
produced by 'realpath -s'.
Signed-off-by: Ben Hutchings <benh@debian.org>
We shouldn't need to change the system configuration to test changes to
initramfs-tools. It's possible to use the -d argument with a custom
config directory this purpose, but testing different MODULES settings is
necessary frequently enough that it deserves its own shortcut. Add a -m
argument to mkinitramfs that overrides the MODULES setting.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
Add a gbp.conf that tries to match the existing changelog format, based
on options and commit message format in docs/maintainer_notes. This lets
anyone just run `gbp dch` and get a changelog mostly consistent with
existing entries.
There are two main differences from how it's currently done: this wraps
lines to 70 characters instead of something closer to 80 (due to a
python default), and it's not possible to directly sign off the git
commit via gbp dch.
To get a better sense of how the changelog differs, try:
- git checkout v0.143^
- git cherry-pick <this-commit>
- gbp dch --commit --release --distribution=experimental
- git diff HEAD v0.143
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
The add_loaded_modules() function is used to add clk, phy and pinctrl
modules to the initramfs with MODULES=dep. But devices bound to these
can have fw_devlink suppliers which also need to be included. One
example is a `clk_mt8186_mm` being supplied by `mtk-mmsys` on a MT8186
Chromebook, which breaks the boot process if the latter is missing.
For each loaded module that we add, also walk the links to bound devices
under `/sys/module/<module>/drivers/<driver>/<device>` paths for
suppliers using the sys_walk_mod_add() function. It's still possible
that we don't have any devices bound to the module, so keep calling
manual_add_modules() on module names.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
The onboard_usb_hub module can handle power regulators for USB devices
it supports. On a MT8186 Chromebook, probing it power-cycles the onboard
USB ports, which causes I/O errors and breaks the boot process if we
have a USB disk mounted as root since it will be forcibly disconnected.
Although it sounds like a kernel bug to power-cycle a hub that is
already in use, other hardware could have onboard hubs that are powered
off before the bootloader starts the kernel, so initializing these hubs
through this module could be necessary for them to boot from USB.
Add the module to the initramfs, so the power-cycle can happen before we
mount disks. It's being renamed to onboard_usb_dev with Linux v6.10, so
add that name to the module list as well.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
`update-initramfs` might be unable to identify the file system type of
the root file system:
```
$ update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-1059-kvm
W: Couldn't identify type of root file system for fsck hook
```
This error message can be confusing and misleading for the user. The
Ubuntu bug reported initially assumed it to be a bug in initramfs-tools
but later found out that the file system missed the label.
Mention the file system name in failed identification warning to help
the user finding the reason.
Bug-Ubuntu: https://launchpad.net/bugs/2068077
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
The `hid-multitouch` kernel module is needed for the keyboard of the
Surface Pro 4.
LP: #1772094
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Storing the intermediate main cpio archive in a temporary file is not
needed. The intermediate cpio archive can be directly piped into the
compressor command. This reduces the execution time a bit and reduces
the risk of the temporary directory (normally in `/var/tmp`) to run out
of space.
Benchmarks
----------
Benchmark on my desktop machine with an AMD Ryzen 7 5700G in a tmpfs
chroot with Debian unstable on 2024-06-03. initramfs-tools 0.143:
```
$ schroot-wrapper -p initramfs-tools=0.143,initramfs-tools-core=0.143,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@desktop:~# hyperfine "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 11.255 s ± 0.026 s [User: 12.467 s, System: 5.728 s]
Range (min … max): 11.194 s … 11.282 s 10 runs
(unstable)root@desktop:~# ls -l /boot/initrd.img-*
-rw-r--r-- 2 root root 67758475 Jun 3 10:54 /boot/initrd.img-6.8.12-amd64
```
This commit is significant faster and the generated initrd is roughly
the same size:
```
$ schroot-wrapper -p ../initramfs-tools_0.143_all.deb,../initramfs-tools-core_0.143_all.deb,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@desktop:~# hyperfine "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 10.283 s ± 0.029 s [User: 11.620 s, System: 5.785 s]
Range (min … max): 10.230 s … 10.335 s 10 runs
(unstable)root@desktop:~# ls -l /boot/initrd.img-*
-rw-r--r-- 1 root root 67757960 Jun 3 10:53 /boot/initrd.img-6.8.12-amd64
```
Benchmark on a Raspberry Pi Zero 2W with Debian unstable on 2024-06-03
and `COMPRESSLEVEL=1` set in `/etc/initramfs-tools/conf.d/compress.conf`
due to the low memory. initramfs-tools 0.143:
```
$ schroot-wrapper -p initramfs-tools=0.143,initramfs-tools-core=0.143,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@zero2w:~# hyperfine -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 206.167 s ± 4.261 s [User: 53.232 s, System: 112.503 s]
Range (min … max): 199.821 s … 210.960 s 10 runs
(unstable)root@zero2w:~# ls -l /boot/initrd.img-*
-rw-r--r-- 1 root root 83542031 Jun 3 12:13 /boot/initrd.img-6.8.12-arm64
```
Same result here. This commit is significant faster and the generated
initrd is roughly the same size:
```
$ schroot-wrapper -p ./initramfs-tools_0.143_all.deb,./initramfs-tools-core_0.143_all.deb,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@zero2w:~# ls -l /boot/initrd.img-*
-rw-r--r-- 1 root root 83540544 Jun 3 11:26 /boot/initrd.img-6.8.12-arm64
(unstable)root@zero2w:~# hyperfine -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 175.592 s ± 5.514 s [User: 54.224 s, System: 112.979 s]
Range (min … max): 166.502 s … 182.111 s 10 runs
```
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Use pipefail for creating final initrd.img file. This is a preparation
for the following commit.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
Enable KVM if `/dev/kvm` is present. Ignore `ppc64el` because KVM is not
usable on the Ubuntu ppc64el autopkgtest runners.
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
dracut-install can act on kernel subdirs directly. This has no
measurable effect on the runtime.
Benchmark results
-----------------
Benchmark on my desktop machine with an AMD Ryzen 7 5700G in a tmpfs
chroot with Debian unstable on 2024-06-02. initramfs-tools 0.143 from
experimental:
```
$ schroot-wrapper -p initramfs-tools=0.143,initramfs-tools-core=0.143,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@desktop:~# hyperfine "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 11.255 s ± 0.026 s [User: 12.467 s, System: 5.728 s]
Range (min … max): 11.194 s … 11.282 s 10 runs
```
This commit is within the margin of error:
```
$ schroot-wrapper -p ../initramfs-tools_0.143_all.deb,../initramfs-tools-core_0.143_all.deb,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@desktop:~# hyperfine "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 11.248 s ± 0.031 s [User: 12.705 s, System: 5.698 s]
Range (min … max): 11.209 s … 11.304 s 10 runs
```
Benchmark on a Raspberry Pi Zero 2W with Debian unstable on 2024-06-02.
initramfs-tools 0.143 from experimental:
```
$ schroot-wrapper -p initramfs-tools=0.143,initramfs-tools-core=0.143,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@zero2w:~# hyperfine -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 206.167 s ± 4.261 s [User: 53.232 s, System: 112.503 s]
Range (min … max): 199.821 s … 210.960 s 10 runs
```
This commit is within the margin of error:
```
(unstable)root@zero2w:~# hyperfine -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 203.379 s ± 4.541 s [User: 53.187 s, System: 109.325 s]
Range (min … max): 197.522 s … 213.103 s 10 runs
```
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
dracut-install has a slow start-up time on ARM devices. Alper Nebi Yasak
wrote: "On my ARM64 Chromebooks dracut-install takes about 2-3 seconds
to start up even for a single module." [1]
Instead of `manual_add_modules` calling `dracut-install` directly,
collect the wanted kernel modules in a temporary file. Call
dracut-install at the end on all modules in one go.
Since the `vmxnet` hook from `open-vm-tools` and `bootcdinitramfshook`
from `bootcd` look at the copied kernel modules, call
`apply_add_modules` before calling the hooks to not break those hooks.
`hidden_dep_add_modules` looks at the installed kernel modules. So call
`apply_add_modules` before it. Since it is the last function that
installs kernel modules, let it call dracut-install directly.
Benchmark results
=================
All benchmarks were taken on my desktop machine in a tmpfs chroot with
Debian unstable on 2024-05-28.
initramfs-tools 0.142 from unstable:
```
$ schroot-wrapper -p initramfs-tools,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@host:~# hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 14.092 s ± 0.022 s [User: 14.008 s, System: 7.106 s]
Range (min … max): 14.070 s … 14.121 s 5 runs
```
git master (commit 78d0471a6134):
```
$ schroot-wrapper -p ../initramfs-tools_0.142_all.deb,../initramfs-tools-core_0.142_all.deb,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@host:~# hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 12.064 s ± 0.028 s [User: 12.940 s, System: 6.019 s]
Range (min … max): 12.042 s … 12.110 s 5 runs
```
This commit:
```
$ schroot-wrapper -p ../initramfs-tools_0.142_all.deb,../initramfs-tools-core_0.142_all.deb,linux-image-generic,firmware-linux,zstd,cryptsetup-initramfs,lvm2,kbd,mdadm,ntfs-3g,plymouth,console-setup,hyperfine -u root -c unstable
(unstable)root@host:~# hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 11.306 s ± 0.151 s [User: 12.464 s, System: 5.756 s]
Range (min … max): 11.206 s … 11.571 s 5 runs
```
LP: #2065180
[1] https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/102
Move the `hidden_dep_add_modules` call after all kernel modules were
copied. Hooks might add or remove kernel modules that will infuence what
kernel modules `hidden_dep_add_modules` will add.
The Ubuntu autopkgtest `qemu-net-dnsmasq` needs to call qemu with
different parameters. Move calling qemu into the separate `run-qemu`
script for better abstraction.
With MODULES=dep, we don't want to include the full graphics drivers
like i915 and amdgpu on systems that provide a simpler framebuffer that
can be used for a video console. The code that handles this relies on
the simultaneous existence of device directories for both the generic
and full drivers, but this is no longer the case with newer kernels and
this strategy no longer works to detect an unloaded graphics driver.
Instead, use some heuristics to decide if we should skip walking full
graphics drivers, under some assumptions:
- Assume we always have a working platform graphics driver on x86.
- Assume EFI framebuffer works if we have EFI variable support.
- Assume any simple-framebuffer device listed in the device-tree works.
These are not necessarily correct, but the exceptions aren't really
worth worrying about.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
The qemu tests are somehow taking 10 more seconds than usual between
running systemd-udevd and discovering virtio-blk devices, which causes
the qemu-panic-shell test to exceed the timeout and error out:
[ 12.693906] Run /init as init process
Loading, please wait...
Starting systemd-udevd version 256~rc3-2
[ 20.470569] virtio_blk virtio3: 1/0/0 default/read/poll queues
[ 20.715954] virtio_blk virtio3: [vda] 556 512-byte logical blocks (285 kB/278 KiB)
Begin: Loading essential drivers ... [ 22.057615] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
[...]
qemu-system-amd64: terminating on signal 15 from pid 17745 (timeout)
Increase the timeout to 120s, so the qemu-panic-shell test can continue
long enough to print the emergency shell the test is looking for. S390X
also has a 120s timeout, so fold that into the new default case as well.
Signed-off-by: Alper Nebi Yasak <alpernebiyasak@gmail.com>
I sprinkled mkinitramfs with `date -Ins` to see where most time is
spent. I ran the [1] test case in a chroot on my laptop. mkinitramfs
took 18.3 seconds. The most time consuming parts:
* 10.13 s (51.4%) auto_add_modules
* 7.20 s (36.5%) run_scripts_optional /usr/share/initramfs-tools/hooks
* 1.27 s (6.4%) final { cat; cpio; cpio|compress } > $outfile
The remaining 1.11 seconds are spread over the remaining parts.
Following hooks in /usr/share/initramfs-tools/hooks took the longest:
* 4.56s (63.3%) framebuffer
* 0.87s (12.1%) plymouth
* 0.81s (11.3%) cryptroot
* 0.23s (3.2%) lvm2
* 0.18s (2.5%) udev
* 0.17s (2.4%) mdadm
The remaining 0.38 seconds are spread over the remaining dozen of hooks.
The framebuffer hook just calls copy_modules_dir and manual_add_modules
multiple times. auto_add_modules calls copy_modules_dir multiple times
and manual_add_modules. copy_modules_dir calls find and then
manual_add_modules.
So most time will be spent in manual_add_modules. This function calls
modprobe on the modules, copies the modules, and add_firmware on the
firmwares from a "modinfo -F firmware" call.
Replacing manual_add_modules by a call to /usr/lib/dracut/dracut-install
speeds up the initramfs building significantly. In the chroot on my
laptop the mkinitramfs execution reduces from 18.3 seconds to 11.0 (40%
reduction).
[1] schroot-wrapper -c mantic -u root --ppa bdrung/ppa -p \
linux-image-generic,initramfs-tools,zstd,busybox-initramfs,cryptsetup-initramfs,isc-dhcp-client,kbd,lvm2,mdadm,ntfs-3g,plymouth,plymouth-theme-spinner
LP: #2031185
Signed-off-by: Benjamin Drung <benjamin.drung@canonical.com>
