Making your own Linux distribution for the Raspberry Pi for fun and profit, part 2

In the previous post, we saw that it is possible to create programs that really only depend on the Linux kernel in order to run. Yet, we still needed some sort of Linux distribution such as Raspbian on our Raspberry Pi to actually start this program. How can we get the Raspberry to run our program without having to also install a full distribution? To answer this question, we first need to figure out how the Raspberry Pi and its Linux distributions actually work. This could be interesting!

Booting Raspbian

To install Raspbian on a Raspberry Pi, you download an image file, and write it to an SD card using a program like Etcher (which, at 84 MB, is quite a bit larger than what we will end up with at the end of this post!) or good old dd. After doing this the SD card will (perhaps surprisingly) contain two partitions: a boot partition (which, when you insert the card into a Mac or Windows computer will show up as an ordinary disk) and a root partition. The latter will likely not show up on your computer as it is (in the case of Raspbian) using the ext4 Linux filesystem, whereas the boot partition is simply an old-school “MS-DOS” FAT partition.

As the boot partition only has a capacity of about 200 MB, clearly all of Raspbians software and utilities are on the root partition. But why have two partitions at all?

To understand this, we need to have a look at how the Raspberry Pi actually boots from an SD card. Looking at the contents of the boot partition, it all seems rather cryptic. So what happens when you turn a Pi on? It turns out, the boot loader, or program that is hardcoded in the Pi’s ROM and is the first thing to be run on power-up, is actually quite smart. After setting up the very basics, it will look for an SD card and, once it has found it, look for the first FAT partition on the card. The boot loader will look for a file called start.elf and execute it. Start.elf, however, is not a normal binary: it doesn’t even run on the Pi’s ARM processor! Wait, what? Yes, you read it correctly. Behind the scenes, another processor core is actually in charge of booting the Pi (and, coincidentally, for the graphics – Raspberries use their GPU to boot).

Start.elf will, again, have a look at the boot partition, this time looking for a file called config.txt. If you open this file, you will see that it contains all sorts of settings, related to the Pi’s hardware: which ports are enabled, what should be the HDMI resolution, and so on and so forth. Crucially, it also contains a setting called “kernel”. This setting points the boot loader to the file that contains a Linux kernel, and is to be loaded and started. The config file also points to a “command line” file (usually cmdline.txt), which provides the Linux kernel additional instructions on start-up.

The boot loader loads the Linux kernel file to memory, loads the command line file to memory, and jumps right into the Linux kernel. For Raspbian, the command line specifies (among others) the option root=PARTUUID=6c586e13-02. This informs Linux that it should use the partition with the indicated UUID as its root partition (obviously this is the larger partition on the SD card). The Linux kernel, which contains drivers for the Pi’s SD card reader as well as the ext4 file system, has no problems mounting this filesystem. Finally, the kernel looks for a file called “/init” (or whatever file is specified on the command line) and executes it (this would be systemd for Rasbian). Note that Linux can’t simply use the FAT partition as its root partition again, as it needs a file system that supports Linux features, such as file ownership and permission bits, which FAT doesn’t support.

Rolling your own distribution

So, knowing this, how can we run our binary without Raspbian? The most straight forward way to do this would be to replace the “init” program on Raspbian’s root partition with our own program. In theory, this should work: our binary does not depend on any system library or daemon to be running. However, in practice, our program still depends on other things to be present than just the kernel. For one thing, our program needs somewhere to write its “Hello world!” greetings to – the ‘stdout’ file descriptor. Further on, our program might need things like /dev/random or /proc/cpuinfo (i.e. reading the Raspberry’s model number using the rppal create from Rust seems to cause reading the cpuinfo file). Now this solution may or may not work if we leave all other parts of Raspbian intact – yet we still wouldn’t know what we could throw away.

Starting from scratch however is easier than you think. What happens if you simply create an empty (ext4) root partition, and throw in our Rust program and call it “init”? Yes, this works… sort of. When you actually try this, you will first see some kernel messages scrolling by (if the loglevel set in the command line is sufficiently high) and then.. it will appear to freeze.

If you were to write a program that loops infinitely however, you would however notice that instead of freezing, the console shows a blinking dash. Aha! Our program is doing something, it’s just not showing. And we now know why: our program does not yet have anywhere to write to.

Okay, so we need to add some files to our root filesystem, at least /proc, /sys and probably some more. Let’s make some directories:

mkdir {etc,bin,sbin,usr,proc,sys,usr}

We will also need to have a way to configure things properly (e.g. mount /proc to procfs!) before our Rust program runs. On a normal Linux installation, you can use the mount command from your shell, but well… we have no shell yet. We could probably call the Linux’ API functions directly from our Rust program to do all the things we need to do, but that would be a bit tedious and would also require us to do it again if we ever wanted to use another programming language, e.g. Go or even C.

Luckily, there is a very easy way to add a shell to an otherwise empty system: busybox. Busybox is a statically linked (so, no dependencies) binary that provides a very basic shell and, more importantly, essentially all utilities you usually need. Want to mount something? Execute busybox mount. Copying files? Sure, busybox cp. Even better, you can create symlinks to busybox (e.g. have /bin/cp point to /bin/busybox) and busybox will, whenever you execute ‘cp’, automatically determine you want to copy some files around. Busybox can even make these symlinks for you!

Placing busybox in /bin/busybox and symlinking /bin/sh to it now allows us to replace our init program with a shell script. The shell script has to start with #!/bin/sh to inform the kernel that it should use /bin/sh, or rather /bin/busybox (because that’s where /bin/sh points to now) should be used to execute it. The following init script will do pretty much the basic configuration for you:

#!/bin/sh
# Install Busybox (places symlinks from /bin and others to the central /bin/busybox executable)
/bin/busybox --install -s
 
# Mount basic directories
mount -t proc proc /proc
mount -t sysfs sysfs /sys
mount -t tmpfs -o size=64k,mode=0755 tmpfs /dev
mkdir /dev/pts
mount -t devpts devpts /dev/pts
# Create elementary device nodes (the numbers can be found by stat'ing on a Raspbian install)
echo "Creating basic devices..."
mknod /dev/null c 1 3
mknod /dev/tty c 5 0

exec /rust-program

Pretty simple, right? Now, as I mentioned before, our program might need other files in the future, such as /dev/random. Also we might later want the ability to read from USB drives, use SPI or I2C devices, et cetera. To do this we could of course use mknod to create all the required device files in /dev from the init script. However, there is an easier way. As it turns out, Busybox also provides a utility called mdev. Like its bigger brother udev, which is commonly included in Linux distributions, it will automatically create device files for you. Simply create an empty file called “/etc/mdev.conf” and add “mdev -s” to the init script. You can even configure it to auto-mount partitions as they are hotplugged by adding the right lines to mdev.conf and telling the kernel to tell mdev whenever a new device is hotplugged, as follows:

echo /sbin/mdev > /proc/sys/kernel/hotplug

The init script is also the place where you should load any kernel modules you need. On the Raspberry for instance, to use SPI, you need the module spi-bcm2835 and possibly spi-bcm2835aux if you want to use other ports than the first one. You can find out which modules you need by running lsmod on an existing Raspbian install. If it is the same version, you can even copy over the modules from its /lib/modules folder. Inserting modules is as simple as insmod module.ko.

Can we make it even simpler?

In the previous post, I mentioned that the root partition doesn’t show up on my Mac when I plug in a Raspbian SD card, but the boot partition does. This is inconvenient as I now need to have a Linux computer around to fiddle with my image. It would be much nicer (also to have my colleagues quickly deploy things to a new SD card) if we could simply use one partition. Is it possible to just use the FAT boot partition? As you may expect by now, the answer is yes!

On some Linux systems, it is desirable to have the ability to perform some tasks before the root filesystem is mounted (e.g. set up a RAID array, decrypt the root file system, et cetera). To this end, you can use something called an init RAM disk or ‘initrd’. The way this works is that before booting the Linux kernel, the boot loader not only loads the Linux kernel itself to memory, but also an “initrd filesystem”. From the kernel command line, the kernel is then made aware of this filesystem and told to use it as its root filesystem (root=/dev/ram0). The kernel boots from the initrd filesystem, and (usually) the init script on the initrd filesystem at some point switches the root filesystem to a ‘real’ one. As we are only interested in running our single program, we could potentially put our entire root partition in this RAM disk and never switch to any other root filesystem!

On the Raspberry, an initrd can be supplied by adding a setting to config.txt and changing the command line (cmdline.txt). The initrd itself, like the kernel, can simply be placed on the FAT partition. The file system itself is a GZIP’ed CPIO file – more or less like a tar file. To create one (e.g. from the filesystem we created above), simply run:

find . | cpio -H newc -o | gzip > initramfs.gz

As an added benefit, the full root filesystem is now copied to memory on boot. Nothing will touch the SD card anymore (unless, well, you mount it again) – you could even pull it out after booting!

Conclusion

Our initial problem of creating a minimal Linux distribution that will run a control program for our escape puzzle box, shortening the boot time, reducing complexity and simplifying building & deployment, has been solved. We now first build a static binary (i.e. without dynamic dependencies). Using a script we then create the initrd filesystem and place our program in it. Finally, the initrd filesystem image is copied together with all other necessary files to a folder. My colleagues can simply copy-paste this folder to any old FAT-formatted SD card, and the Pi will boot it. Our one minute boot time has now been reduced to three seconds!

Source code is available here. The current version improves over the above by providing sound support (ALSA), Wi-Fi (which can be configured by putting a configuration file on the boot partition) and USB serial console. The latest version will run any binary or shell script called ‘box’ in the root of the boot drive on start-up.