Minimal Viable Illumos

I've been playing with the creation of several minimal variants of illumos recently.

I looked at how little memory a minimal illumos distro could be installed and run in. Note that this was a properly built distribution - correctly packaged, most features present (if disabled), running the whole set of services using SMF.

In another dimension, I considered illumos pureboot, something that was illumos, the whole of illumos, and nothing but illumos.

Given that it was possible to boot illumos just to a shell, without all the normal SMF services running, how minimal can you make such a system.

At this point, if you're not thinking JEOS, Unikernels, or things like IncludeOS, then you should be.

So the first point is that you're always running this under a hypervisor of some sort. The range of possible hardware configurations you need to worry about is very limited - hypervisors emulate a small handful of common devices.

Secondly, the intention is never to install this. Not directly anyway. You create an image, and boot and run that. For this experiment, I'm simply running from a ramdisk. This is the way the live image boots, or you PXE boot, or even the way SmartOS boots.

First, the starting set of packages, both in Tribblix (SVR4) and IPS naming:

• SUNWcsd=SUNWcsd
• SUNWcs=SUNWcs
• TRIBsys-library=system/library
• TRIBsys-kernel=system/kernel
• TRIBdrv-ser-usbser=driver/serial/usbser
• TRIBsys-kernel-platform=system/kernel/platform
• TRIBdrv-usb=driver/usb
• TRIBsys-kernel-dtrace=system/kernel/dtrace/providers
• TRIBsys-net=system/network
• TRIBsys-lib-math=system/library/math
• TRIBsys-libdiskmgt=system/library/libdiskmgt
• TRIBsys-boot-grub=system/boot/grub
• TRIBsys-zones=system/zones
• TRIBdrv-storage-ata=driver/storage/ata
• TRIBdrv-storage-ahci=driver/storage/ahci
• TRIBdrv-i86pc-platform=driver/i86pc/platform
• TRIBdrv-i86pc-ioat=driver/i86pc/ioat
• TRIBdrv-i86pc-fipe=driver/i86pc/fipe
• TRIBdrv-net-e1000g=driver/network/e1000g
• TRIBsys-boot-real-mode=system/boot/real-mode
• TRIBsys-file-system-zfs=system/file-system/zfs

You can probably go further, but I wanted to at least allow the possibility of talking to a storage device.

There are a few packages here that you might wonder about:

• usbser is actually needed, it's a hard dependency of consconfig_dacf
• many admin commands link against the zones libraries, so I add those even though they're not strictly necessary in most scenarios
• the system will boot and run without zfs, but will panic if you run find down the dev tree
• the system will panic if the real-mode stuff is missing
• grub is needed to make the iso and boot it

It's possible to construct a bootable iso from the above components, which can then be customized.

I took two approaches to this. The simple way is to simply start chopping out the files you don't want. For example, man pages and includes. The second is to drop all of userland and only put back the files you need, one by one. I tend not to tweak the kernel much, that's non-trivial and you're only looking at marginal gains.

Working out which files are necessary is trial and error. Especially shared libraries, many of which are loaded lazily so you can't just use what the executable tells you - some of the libraries it's linked against will never be pulled in

I've put together some scripts that know how to create an image suitable for 32-bit or 64-bit hardware, we can be specific as we know exactly the environment we are going to run in - and you just build a new custom iso if things change, rather than try and build a generic image.

To be useful, the system needs to talk to something. You'll see that I've installed e1000g, which is what VirtualBox and qemu will give you by default. First, we have to get the root filesystem mounted read-write:

/etc/fs/ufs/mount -o remount,rw /devices/ramdisk:a /

Normally, there's a whole lot of network configuration handled by SMF, and it's all rather complicated. So we have to do it all by hand, which turns out to be relatively simple:

/sbin/ifconfig e1000g0 plumb
env SMF_FMRI=svc:/net/ip:d /lib/inet/ipmgmtd
/sbin/ifconfig e1000g0 inet 192.168.59.59/24 up

You need ipmgmtd running, and it's expecting to be run under SMF, but the way it checks is to look for SMF_FMRI in the environment, so it's easy to fool.

If you've got your VirtualBox VM set up with a Host-only Adapter, you should be able to communicate with the guest. Not that there's anything present to talk to yet.

So I set up a simple Node.js server. Now node itself doesn't have many external dependencies - just the gcc4 runtime - and for basic purposes you just need the node binary and a js file with a 'hello world' http server.

With that, I have 64M of data in a 22M boot archive that is put on a 25M iso that boots up in a few seconds with an accessible web server. Pretty neat.

While it's pretty specific to Tribblix and my own build environment, there's an mvi repository on github containing all the scripts I used to build this, for those interested.