Making your own Linux distribution for the Raspberry Pi for fun and profit, part 1

13-10-2019

For our Escape Box project, we had to come up with some kind of way to control all the various puzzles – the lights, buttons, servos, displays all need to be controlled somehow. Luckily there are various solutions available: the Arduino, the ESP8266, and the Raspberry Pi are all available for under $10 and are all capable of doing the things we want. As we were prototyping however, we did not want to spend time learning a new platform, and stuck with what we knew best at the time: the Raspberry. On the software side, we went with NodeJS – the various packages for the Pi make programming it quite easy and straightforward. As it turns out, the Raspberry/Node-solution has some downsides that led me down a particularly interesting rabbit hole: what if I made my own Linux distribution for the Raspberry Pi?

The problems with our initial solution: slow boots, mutability, and complexity

While the Raspberry/Node-solution works fine for prototyping, it isn’t ideal for running in production. First of all, in order to run NodeJS at all, you need to install a full Linux distribution to the Pi (i.e. Raspbian). While this is easily done (simply writing an image file to an SD card), it is a bit more involved to customize the image and put our own software on it. Additionally, Raspbian is quite slow to boot – our app would only be starting after about 60 seconds on the Pi Zero. NodeJS on the Pi is far from ideal either – with only packages for i2c, spi and wiring-pi installed, our node_modules directory clocks in at 1194 files and 9.5 MB!

Finally, and perhaps the most important reason, is that the whole thing is quite fragile. The file system is writable at any time, which means we cannot be sure that our software would each time the Pi was powered up without fully imaging the SD card each time. Even simple things like the periodic fsck-on-bootup cause issues every now and them and because Raspbian is quite complex, you never know what else could cause issues. Additionally, SD cards, being tiny computers themselves, are famous for not coping well with sudden power loss. Our software doesn’t need to write any files to the disk, so it would be ideal if we could simply mount the whole card read-only. There are ways to do this, and it works, but it is quite a bit more involved than simply doing a mount -o remount,rw / and be done with it. A read-ony root filesystem also turns out to break various assumptions here and there, assumptions that change on every update to the system.

Replacing NodeJS

As dealing with the various NodeJS packages, its dependencies and cross-compiling them became tedious quite quickly, we investigated alternatives. Rust seemed interesting for a few reasons. First, it is aimed at (fairly low-level) systems programming. Second, it appears to support the Raspberry Pi and even cross-compiling to it. Finally, it allows you to compile complex programs down to a single binary file, which makes deployment quite a bit easier.

Getting Rust to work on the Raspberry turned out to be fairly easy, though not straightforward. Installing Rust is trivial using Rustup. After installing, getting to “Hello world” on your desktop computer is trivial:

$ cargo init hello
     Created binary (application) package
$ cd hello
$ cargo run
   Compiling hello v0.1.0 (/home/tommy/hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.57s
     Running `target/debug/hello`
Hello, world! 

Okay, so how do we run this on the Pi? Well, we need to fix two things. First of all, my desktop computer has a 64-bit Intel CPU, whereas the Pi (Zero, in this case) has an ARMv6 processor. Somehow we need to tell the Rust compiler to generate ARM binaries. We turn back to Rustup to install an ARM target, e.g. a Rust compiler and associated tools that are capable of creating ARM binaries:

rustup target add arm-unknown-linux-gnueabi

Now we still need to ask Rust (or rather, cargo, the program that manages building for us) to create an ARM binary. This is done by creating a file called config in the .cargo folder in your project directory and inserting the following contents:

[build]
target = "arm-unknown-linux-gnueabi"

[target.arm-unknown-linux-gnueabi]
linker = "arm-linux-gnueabi-gcc"

The first section tells cargo to use the “arm-unknown-linux-gnueabi” toolchain we just installed using rustup. The second section tells cargo that if we are building for arm-unknown-linx-gnueabi, we want to use the GCC linker for that architecture. Running cargo build again now leaves us with an ARM binary:

$ file ./target/arm-unknown-linux-gnueabi/debug/hello
./target/arm-unknown-linux-gnueabi/debug/hello: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, BuildID[sha1]=5f6a43914c5121a1bbbca5c91884e40eef8951ba, with debug_info, not stripped

Indeed, scp’ing the binary to a Pi and running it cheerfully greets us with “Hello world!”. Still, we are not done yet. Remember in the beginning I said we needed to fix two things? Right. As it turns out, the binary we just created relies on various system libraries, which are dynamically loaded when the executable is started. This is easily verified from the command line on the Pi:

$ readelf -d ./hello 

Dynamic section at offset 0x25d20 contains 32 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [librt.so.1]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux.so.3]

While this is perfectly fine, it still requires us to have a full Linux distribution on the Pi and there could still be mismatches between our toolchain and the Pi’s environment. Can we do better? Looking closely at the output of readelf you will notice that the libraries that are loaded provide “standard C library” functionality. That’s fine, but can’t our program simply deal with the Linux kernel directly? It is an operating system, after all!

As it turns out, the Rust “arm-unknown-linux-gnueabi” toolchain we just installed assumes that the target system has a GNU C-library (or “glibc”) which can be dynamically loaded. This is usually a good assumption, as basically all programs need these libraries, and it would cost significant disk space if each program were to include their own copy. However, we will only be running one program, so we might as well bring our own! As it turns out, there is another toolchain available, called “arm-unknown-linux-musleabi”. This toolchain replaces glibc with musl, an alternative standard C library implementation that can be statically linked with our application (which is a fancy way of saying that its functionality will simply be included in our binary).

Again, we turn to rustup and ask for the musl-toolchain. We edit the .cargo/config file accordingly, to specify that we now want to use the musl toolchain and that it needs a specific linker as well (turns out you should still use the “gnueabi” linker here – if the command fails, install it using apt install gcc-multilib-arm-linux-gnueabi libc6-dev-armel-cross). After cargo build‘ing again, we are left with another “hello” executable which also happily prints “Hello world!” – though this time around, without any help of its dynamic friends:

$ readelf -d ./target/arm-unknown-linux-musleabi/debug/hello
There is no dynamic section in this file.

Theoretically, this binary will work in any ARMv6 Linux environment that has a kernel that provides all the functionality our program asks for (which at this point is obviously very basic!).

Running our new, standalone binary

Deploying this to our escape box would be easily done by adding the compiled binary to its filesystem and adding it to Raspbian’s systemd configuration as an auto-starting service. This however does not really solve any of the other problems we had, such as the long boot delay. Can we do better?

As our binary is now pretty much independent of the environment, we could consider a smaller Linux distribution. Of particular interest is piCore: the port of TinyCore Linux to the Raspberry Pi. PiCore is a very minimal distribution that boots off a read-only partition. It allows users to add extensions which it will (on boot) unpack and overlay on top of the root filesystem. Exactly what we need! After installing piCore, we simply add our binary to the /home/tc folder, and add a line executing it in /opt/bootlocal.sh. After we’ve made sure everything is set up correctly, a mere filetool.sh -b is sufficient in order to commit our changes to the SD card. Indeed this works, and is much faster than the Raspbian solution we started out with!

Can we do better?

The Rust+piCore solution is pretty nice and solves a lot of our issues. However, it is still quite complex. Our program only needs the Linux kernel, really, to do its I/O. We shouldn’t need anything else, so can we just run without all the other stuff? Turns out, we can! In the next post I will describe how you can make your own, very minimal, Linux distribution…