How low can Tribblix go?

One of the things I wanted to do with Tribblix was to allow it to run in places that other illumos distros couldn't reach.

One possible target here is systems with less than gargantuan memory capacities.

(Now, I don't have any such systems. But VirtualBox allows you to adjust the amount of memory in a guest very easily, so that's what I'm doing.)

I started out by building a 32-bit only image. That is, I built a regular (32- and 64-bit combined) image, and simply deleted all the 64-bit pieces. You can do this slightly better by building custom 32-bit only packages, but it was much simpler to identify all the directories named amd64 and delete them.

(Why focus on 32-bit here? The default image has both a 32-bit and 64-bit kernel, and many libraries are shipped in both flavours too. So removing one of the 32-bit or 64-bit flavours will potentially halve the amount of space we need. It makes more sense to drop the 64-bit files - it's easier to do, and it's more likely that real systems with minimal memory are going to be 32-bit.)

The regular boot archive in Tribblix is 160M in size (the file on the ISO is gzip-compressed and ends up being about a third of that), but it's loaded into memory as a ramdisk so the full size is a hard limit on how much memory you're going to need to boot the ISO. You might be able to run off disk with less, as we'll see later. The 32-bit boot archive can be shrunk to 90M, and still has a little room to work in.

The other part of booting from media involves the /usr file system being a compressed lofi mount from a file. I've made a change in the upcoming release by removing perl from the live boot (it's only needed for intrd, which is disabled anyway if you're booting from media), which saves a bit of space, and the 32-bit version of /usr is about a third smaller than the regular combined 32/64-bit variant. Without any additional changes, it is about 171M.

So, the boot archive takes a fixed 90M, and the whole of /usr takes 171M. Let's call that 256M of basic footprint.

I know that regular Tribblix will boot and install quite happily with 1G of memory, and previous experience is that 768M is fine too.

So I started with a 512M setup. The ISO boots just fine. I tried an install to ZFS. The initial part of the install - which is a simple cpio of the system as booted from media - worked fine, if very slowly. The second part of the base install (necessary even if you don't add more software) adds a handful of packages. This is where it really started to struggle, it just about managed the first package and then ground completely to a halt.

Now, I'm sure you could tweak the system a little further to trim the size of both the boot archive and /usr, or tweak the amount of memory ZFS uses, but we're clearly close to the edge.

So then I tried exactly the same setup, installing to UFS instead of ZFS. And it installs absolutely fine, and goes like greased lightning. OK, the conclusion here is that if you want a minimal system with less than 512M of memory, then don't bother with ZFS but aim at UFS instead.

Reducing memory to 256M, the boot and install to UFS still work fine.

With 192M of memory, boot is still good, the install is starting to get a bit sluggish.

If I go down to 128M of memory, the ISO won't boot at all.

However, if I install with a bit more memory, and then reduce it later, Tribblix on UFS works just fine with 128M of memory. Especially if you disable a few services. (Such as autofs cron zones-monitoring zones power fmd. Not necessarily what you want to do in production, but this isn't supposed to be production.)

It looks like 128M is a reasonable practical lower limit. The system is using most of the 128M (it's starting to write data to swap, so there's clearly not much headroom).

Going lower also starts to hit real hard limits. While 120M is still good, 112M fails to boot at all (I get "do_bop_phys_alloc Out of memory" errors from the kernel - see the fakebop source). I'm sure I could go down a bit further, but I think the next step is to start removing drivers from the kernel, which will reduce both the installed boot archive size and the kernel's memory requirements.

I then started to look more closely at the boot archive. On my test machine, it was 81M in size. Removing all the drivers I felt safe with dropped it down to 77M. That still seems quite large.

Diving into the boot archive itself, and crawling through the source for bootadm, I then found that the boot archive was a ufs archive that's only 25% full. It turns out that the boot archive will be hsfs if the system finds /usr/bin/mkisofs, otherwise it uses ufs. And it looks like the size calculation is a bit off, leading to an archive that's massively oversized. After installing mkisofs and rebuilding the boot archive, I got back to something that was 17M, which is much better.

On testing with the new improved boot archive, boot with 96M, or even 88M, memory is just fine.

Down to 80M of memory, and I hit the next wall. The system looks as though it will boot reasonably well, but /etc/svc/volatile fills up and you run out of swap. I suspect this is before it's had any opportunity to add the swap partition, but once it's in that state it can't progress.

Overall, in answer to the question in the title, a 32-bit variant of Tribblix will install (using UFS) on a system with as little as 192M of memory, and run on as little as 96M.