Understanding U-Boot's Boot Process

Over the last couple of months, I’ve been playing with writing an operating system. This post is only peripherally about that, though. I had to do quite a bit of investigating to work out how to get my nascent OS to boot. It’s hard to work it out from the available documentation: almost everything’s about how to make existing OSes work, and they’ve mostly hidden the details away so you don’t have to worry about it.

tl;dr

Introduction

One of the things that an OS needs is a way to start. At the most primitive level, this is usually handled by some kind of firmware on the machine, which will put the processor into a consistent state, and then load some more code, either from a known location on a storage device, or simply from more ROM. This next stage code is the bootloader, and its job is to find the operating system kernel, and anything else that it needs1, load those resources into the right areas of RAM, and start executing it2.

On RISC-V, which is what I’m working with in my OS project, the first step is performed by an SBI implementation (typically OpenSBI). For RISC-V and ARM, the next stage, the bootloader, is where U-Boot comes in.

U-Boot configuration

U-Boot has a small built-in “shell” command interpreter3. It starts out with its basic configuration built in at compile time. The conceptual entry-point, however, is the bootcmd variable. This variable contains a series of semi-colon separated shell commands which are executed after U-Boot has started.

In the default configuration, U-Boot waits for two seconds before running the bootcmd commands. If you press a key during those two seconds, it will stop, and drop you into a shell. From there, you can inspect the configuration, should you so desire.

You can see individual variables using echo ${varname} (the braces are not optional). If you want to view all of the variables, do not be tempted by the showvar command – that only seems to show a small subset of the variables. Instead, use env print -a to get all the environment variables.

Let’s start by looking at the bootcmd variable::

bootcmd=run distro_bootcmd

This starts the so-called distro boot process. This is a collection of functions (implemented, like bootcmd itself, as environment variables containing commands). This can be configured by individual OS distributions, but the general process is:

U-Boot will try each approach in turn; with luck, one of those will actually boot a kernel, and your OS is running.

Scanning for devices

How does this work in detail? I’m going to dive into the shell commands and see exactly what it’s doing. Since some of these are quite long, and contain lots of commands, I’m going to split them into separate lines for readability. In practice, each variable/function is a single line of text with commands separated by semicolons. I’m also gonig to include the values of any referenced variables.

First, the distro_bootcmd function::

distro_bootcmd=
    scsi_need_init=
    setenv nvme_need_init
    virtio_need_init=
    for target in ${boot_targets}; do
        run bootcmd_${target}
    done

boot_targets=nvme0 virtio0 virtio1 scsi0 dhcp

Here we can see that it sets up a few variables, and then iterates over the device names in boot_targets, trying to run bootcmd_${target} for each one. This is the device scan part. I’m not going to look at all of the subsequent commands, as they’re mostly all the same, simply working on a different device4. Here’s one of them::

bootcmd_nvme0=
    devnum=0
    run nvme_boot

nvme_boot=
    run boot_pci_enum
    run nvme_init
    if nvme dev ${devnum}; then
        devtype=nvme
        run scan_dev_for_boot_part
    fi

boot_pci_enum=pci enum

nvme_init=
    if ${nvme_need_init}; then
        setenv nvme_need_init false
        nvme scan
    fi

This collection of functions does some device-type-specific initialisation steps (boot_pci_enum and nvme_init; the latter is flagged so that it’s only done once), then checks that the device exists, and if it does, runs a partition scan looking for a boot partition. Note that it sets devtype and devnum variables. These are used later.

Most of the other bootcmd_${target} functions do much the same thing: initialise hardware, check for a device, and partition scan the device using scan_dev_for_boot_part.

Scanning for partitions

So we come to the partition scan part of the process. Remember, this is going to happen in turn for every device found earlier::

scan_dev_for_boot_part=
    part list ${devtype} ${devnum} -bootable devplist
    env exists devplist || setenv devplist 1
    for distro_bootpart in ${devplist}; do
        if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then
            part uuid ${devtype} ${devnum}:${distro_bootpart} distro_bootpart_uuid
            run scan_dev_for_boot
        fi
    done
    setenv devplist

This gets a list of partitions marked as “bootable” for the device specified by devtype and devnum, and puts the result in the devplist variable. Then it iterates over each bootable partition (or just partition 1, if none were found). It gets the partition number into the distro_bootpart variable, the filesystem type into bootfstype, the partition UUID into distro_bootpart_uuid and then scans the device for boot components.

Scanning for bootable files

This next phase simply tries a couple of different locations on the filesystem (/ and /boot/ directories), and checks for both extlinux files and a boot script. It also checks for EFI files (which can only appear in one place, so that’s checked outside the loop).

scan_dev_for_boot=
    echo Scanning ${devtype} ${devnum}:${distro_bootpart}...
    for prefix in ${boot_prefixes}; do
        run scan_dev_for_extlinux
        run scan_dev_for_scripts
    done
    run scan_dev_for_efi
boot_prefixes=/ /boot/

Booting with extlinux

This is where, finally, things can get booted. The scan_dev_for_extlinux function checks whether there’s an extlinux/extlinux.conf file. If so, it will read the file into memory, and boot using syslinux::

scan_dev_for_extlinux=
    if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then
        echo Found ${prefix}${boot_syslinux_conf}
        run boot_extlinux
        echo EXTLINUX FAILED: continuing...
    fi
boot_syslinux_conf=extlinux/extlinux.conf

boot_extlinux=
    sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}
scriptaddr=0x8c100000

Booting with a script

Alternatively, you can write a U-Boot script file named either boot.scr.uimg or simply boot.scr, and place that in either / or /boot/::

scan_dev_for_scripts=
    for script in ${boot_scripts}; do
        if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then
            echo Found U-Boot script ${prefix}${script};
            run boot_a_script;
            echo SCRIPT FAILED: continuing...;
        fi;
    done
boot_scripts=boot.scr.uimg boot.scr

boot_a_script=
    load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script};
    source ${scriptaddr}
scriptaddr=0x8c100000

As with the extlinux case above, it checks whether either filename exists, loads the script into memory, and then executes it. Note that in this case, it uses load. This seems to require the use of a FIT packaged file (see below), so you can’t just write a plain text file to disk and have it work. However, with this method, you can do whatever setup you like within your script.

Booting with EFI

The EFI method is a bit more complicated than the other two, so I’m not going to show the whole code here. But what it does is::

FIT files

The Flat Image Tree (FIT) file format is a specification for packaging multiple files and boot configurations into a single file. It piggy-backs on the device tree specification – a FIT file uses the device tree binary format, with a specific structure and key names.

The load command in U-Boot loads a FIT file into memory, and allows you to access the data in named sections of it. In this respect, it’s a bit like an initrd as used in Linux.

Files are packaged into FIT images using the mkimage command from U-Boot. See section 5 of the FIT specification for more details on how to build an image file.

Further configuration

The U-Boot build system is quite comprehensive, and there are lots of options. For example, you can enable/disable specific shell commands. You can specify particular addresses to use for loading or running data – these are presented in the environment variables of the shell (typically as something_addr_r for the loading address).

The shell can also, if given suitable write access, store changed environment variables on disk, so that persistent configuration changes can be made, such as setting a default boot configuration after installation.

  1. Like an initial ramdisk or a device tree blob. ↩︎

  2. This isn’t the only way that things can work: U-Boot allows for secondary (and even tertiary) loaders, which take the place of the firmware, or sit between the firmware and the primary bootloader. I’m concentrating on the simple case here. ↩︎

  3. Actually, one of two shells: HUSH, which is “full-featured” and supports loops and conditional statements; and a very slimmed-down shell for use on resource-contrained embedded systems, which supports only a subset of the available commands. I’m describing a setup using HUSH. ↩︎

  4. Except for the DHCP one, which is different to the rest. I’m not going to look at DHCP booting in this article either. ↩︎

  5. Flattened Device Tree. A binary format of the device tree that describes the hardware. ↩︎