Wednesday, October 21, 2015

Tribblix Turns Three

It's a little hard to put a fixed date on when I started work on Tribblix.

The idea - of building a custom distribution - had been floating around my mind in the latter days of OpenSolaris, and I registered the domain back in 2010.

While various bits of exploratory work had been going on in the meantime, it wasn't until the autumn of 2012 that serious development started. Eventually, after a significant number of attempts, I was able to produce a functional ISO image. That was:

-rw-r--r--   1 ptribble 493049856 Oct 21  2012 tribblix-0m0.iso

The first blog post was a few days later, but I'm going to put October 21st as the real date of birth.

Which means that Tribblix is 3 years old today!

In that time it's gone from a simple toy to a fully fledged distribution, most of the original targets I set myself have been met, it's been my primary computing environment for a while, it's proving useful as a platform for interesting experiments, and I'm looking forward to taking it even further in the next few years.

Tuesday, October 20, 2015

Minimal Viable Illumos

I've been playing with the creation of several minimal variants of illumos recently.

I looked at how little memory a minimal illumos distro could be installed and run in. Note that this was a properly built distribution - correctly packaged, most features present (if disabled), running the whole set of services using SMF.

In another dimension, I considered illumos pureboot, something that was illumos, the whole of illumos, and nothing but illumos.

Given that it was possible to boot illumos just to a shell, without all the normal SMF services running, how minimal can you make such a system.

At this point, if you're not thinking JEOS, Unikernels, or things like IncludeOS, then you should be.

So the first point is that you're always running this under a hypervisor of some sort. The range of possible hardware configurations you need to worry about is very limited - hypervisors emulate a small handful of common devices.

Secondly, the intention is never to install this. Not directly anyway. You create an image, and boot and run that. For this experiment, I'm simply running from a ramdisk. This is the way the live image boots, or you PXE boot, or even the way SmartOS boots.

First, the starting set of packages, both in Tribblix (SVR4) and IPS naming:

  • SUNWcsd=SUNWcsd
  • SUNWcs=SUNWcs
  • TRIBsys-library=system/library
  • TRIBsys-kernel=system/kernel
  • TRIBdrv-ser-usbser=driver/serial/usbser
  • TRIBsys-kernel-platform=system/kernel/platform
  • TRIBdrv-usb=driver/usb
  • TRIBsys-kernel-dtrace=system/kernel/dtrace/providers
  • TRIBsys-net=system/network
  • TRIBsys-lib-math=system/library/math
  • TRIBsys-libdiskmgt=system/library/libdiskmgt
  • TRIBsys-boot-grub=system/boot/grub
  • TRIBsys-zones=system/zones
  • TRIBdrv-storage-ata=driver/storage/ata
  • TRIBdrv-storage-ahci=driver/storage/ahci
  • TRIBdrv-i86pc-platform=driver/i86pc/platform
  • TRIBdrv-i86pc-ioat=driver/i86pc/ioat
  • TRIBdrv-i86pc-fipe=driver/i86pc/fipe
  • TRIBdrv-net-e1000g=driver/network/e1000g
  • TRIBsys-boot-real-mode=system/boot/real-mode
  • TRIBsys-file-system-zfs=system/file-system/zfs
You can probably go further, but I wanted to at least allow the possibility of talking to a storage device.

There are a few packages here that you might wonder about:

  • usbser is actually needed, it's a hard dependency of consconfig_dacf
  • many admin commands link against the zones libraries, so I add those even though they're not strictly necessary in most scenarios
  • the system will boot and run without zfs, but will panic if you run find down the dev tree
  • the system will panic if the real-mode stuff is missing
  • grub is needed to make the iso and boot it
It's possible to construct a bootable iso from the above components, which can then be customized.

I took two approaches to this. The simple way is to simply start chopping out the files you don't want. For example, man pages and includes. The second is to drop all of userland and only put back the files you need, one by one. I tend not to tweak the kernel much, that's non-trivial and you're only looking at marginal gains.

Working out which files are necessary is trial and error. Especially shared libraries, many of which are loaded lazily so you can't just use what the executable tells you - some of the libraries it's linked against will never be pulled in

I've put together some scripts that know how to create an image suitable for 32-bit or 64-bit hardware, we can be specific as we know exactly the environment we are going to run in - and you just build a new custom iso if things change, rather than try and build a generic image.

To be useful, the system needs to talk to something. You'll see that I've installed e1000g, which is what VirtualBox and qemu will give you by default. First, we have to get the root filesystem mounted read-write:

/etc/fs/ufs/mount -o remount,rw /devices/ramdisk:a /
Normally, there's a whole lot of network configuration handled by SMF, and it's all rather complicated. So we have to do it all by hand, which turns out to be relatively simple:

/sbin/ifconfig e1000g0 plumb
env SMF_FMRI=svc:/net/ip:d /lib/inet/ipmgmtd
/sbin/ifconfig e1000g0 inet up

You need ipmgmtd running, and it's expecting to be run under SMF, but the way it checks is to look for SMF_FMRI in the environment, so it's easy to fool.

If you've got your VirtualBox VM set up with a Host-only Adapter, you should be able to communicate with the guest. Not that there's anything present to talk to yet.

So I set up a simple Node.js server. Now node itself doesn't have many external dependencies - just the gcc4 runtime - and for basic purposes you just need the node binary and a js file with a 'hello world' http server.

With that, I have 64M of data in a 22M boot archive that is put on a 25M iso that boots up in a few seconds with an accessible web server. Pretty neat.

While it's pretty specific to Tribblix and my own build environment, there's an mvi repository on github containing all the scripts I used to build this, for those interested.

Thursday, October 08, 2015

Deconstructing .pyc files

I've recently been trying to work out why python was recompiling a bunch of .pyc files. I haven't solved that, but I learnt a little along the way, enough to be worth writing down.

Python will recompile a .py file onto a .pyc file if it thinks something's changed. But how does it decide something has changed? It encodes some of the pertinent details in the header of the .pyc file.

Consider a file. There's a and a foo.pyc. I open up the .pyc file in emacs and view it in hex. (ESC-x hexl-mode for those unfamiliar.)

The file starts off like this:

 03f3 0d0a c368 7955 6300 0000 ....

The first 4 bytes 03f30d0a are the magic number, and encode the version of python. There's a list of magic numbers in the source, here.

To check this, take the 03f3, reverse it to f303, which is 62211 decimal. That corresponds to 2.7a0 - this is python 2.7, so that matches. (The 0d0a is also part of the encoding of the magic number.) This check is just to see if the .pyc file is compatible with the version of python you're using. If it's not, it will ignore the .pyc file and may regenerate it.

The next bit is c3687955. Reverse this again to get the endianness right, and it's 557968c3. In decimal, that's 1434020035.

That's a timestamp, standard unix time. What does that correspond to?

perl -e '$f=localtime(1434020035); print $f'
Thu Jun 11 11:53:55 2015

And I can look at the file (on Solaris and illumos, there's a -e flag to ls to give us the time in the right format rather than the default "simplified" version).

/bin/ls -eo
-rw-r--r-- 1 root  7917 Jun 11 11:53:55 2015

As you can see, that matches the timestamp on the source file exactly. If the timestamp doesn't match, then again python will ignore it.

This has consequences for packaging. SVR4 packaging automatically preserves timestamps, with IPS you need to use pkgsend -T to do so as it's not done by default.

Tuesday, October 06, 2015

Software directions in Tribblix

Tribblix has been developing in a number of different directions. I've been working on trimming the live image, and strengthening the foundations.

Beyond this, there is a continual stream of updated packages. Generally, if I package it, I'll try and keep it up to date. (If it's downrev, it's usually for a reason.)

In the meantime I've found time for experiments in booting Tribblix in very little memory, and creating a pure illumos bootable system.

But I thought it worthwhile to highlight some of the individual packages that have gone into Tribblix recently.

The big one was adding LibreOffice, of course. Needless to say, this was a modest amount of work. (Not necessarily all that hard, but it's a big build, and the edit-compile-debug cycle is fairly long, so it took a while.) I need to go back and update LibreOffice to a more current version, but the version I now have meets all of my needs so I can invest time and energy elsewhere.

On the desktop, I added MATE, and incorporated SLiM as a login manager. Tribblix has a lot of desktop environments and window managers available, although Xfce is still the primary and best supported option. I finally added the base GTK engines and icon themes, which got rid of a lot of errors.

In terms of tools, there's now Dia, Scribus, and Inkscape.

Tribblix has always had a retro streak. I've added gopher, gophervr, and the old Mosaic browser. There are other old X11 tools that some of you may remember - xcoral, xsnow, xsol, xshisen. If only I could get xfishtank working again.

I've been keeping up with Node.js releases, of course. But the new kid on the block is Go, and that's included in Tribblix. Current versions work very well, and now we've got past the cgo problems, there's a whole raft of modern software written in Go that's now available to us. The next one up is probably Rust.

Fun with SPARC emulators

While illumos supports both SPARC and x86 platforms, it would be a fair assessment that the SPARC support is a poor relation.

There are illumos distributions that run on SPARC - OpenSXCE has for a while, Tribblix and DilOS also have SPARC images available and both are actively maintained. The mainstream distributions are x86-only.

A large part of the lack of SPARC support is quite simple - the number of users with SPARC hardware is small; the number of developers with SPARC hardware is even smaller. And you can see that the SPARC support is largely in the hands of the hobbyist part of the community. (Which is to be expected - the commercial members of the community are obviously not going to spend money on supporting hardware they neither have nor use.)

Absent physical hardware, are there any alternatives?

Perhaps the most obvious candidate is qemu. However, the sparc64 implementation is fairly immature. In other words, it doesn't work. Tribblix will start to boot, and does get a little way into the kernel before qemu crashes. I think it's generally agreed that qemu isn't there yet.

The next thing I tried is legion, which is the T1/T2 simulator from the opensparc project. Having built this, attempting to boot an iso image immediately fails with:

FATAL: virtual_disk not supported on this platform

which makes it rather useless. (I haven't investigated to see if support can be enabled, but the build system explicitly disables it.) Legion hasn't been updated in a while, and I can't see that changing.

Then I came across the M5 simulator. This supports a number of systems, not just SPARC. But it's an active project, and claims to be able to emulate a full SPARC system. I can build it easily enough, running it needs the opensparc binary download from legion (note - you need the T1 download, version 1.5, not the newer T2 version of the download). The instructions here appear to be valid.

With M5, I can try booting Tribblix for SPARC. And it actually gets a lot further than I expected! Just not far enough:

cpu Probing I/O buses

Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.

ok boot
Boot device: vdisk  File and args:
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
panic[cpu0]/thread=180e000: lgrp_traverse: No memory blocks found

Still, that's illumos bailing, there aren't any errors from M5.

Overall, I think that M5 shows some promise as a SPARC emulator for illumos.

Friday, October 02, 2015

Notifications from SMF (and FMA)

In illumos, it's possible to set the system up so that notifications are sent whenever anything happens to an SMF service.

Unfortunately, however, the illumos documentation is essentially non-existent, although looking at the Solaris documentation on the subject should be accurate.

The first thing is that you can see the current notification state by looking at svcs -n:

Notification parameters for FMA Events
    Event: problem-diagnosed
        Notification Type: smtp
            Active: true
            reply-to: root@localhost
            to: root@localhost

        Notification Type: snmp
            Active: true

        Notification Type: syslog
            Active: true

    Event: problem-repaired
        Notification Type: snmp
            Active: true

    Event: problem-resolved
        Notification Type: snmp
            Active: true

The first thing to realize here is that first line - these are the notifications sent by FMA, not SMF. There's a relationship, of course, in that if an SMF service fails and ends up in maintenance, then an FMA event will be generated, and notifications will be sent according to the above scheme.

(By the way, the configuration for this comes from the svc:/system/fm/notify-params:default service, which you can see the source for here. And you'll see that it basically matches exactly what I've shown above.)

Whether you actually receive the notifications is another matter. If you have syslogd running, which is normal, then you'll see the syslog messages ending up in the log files. To get the email or SNMP notifications relies on additional service. These are


and if these are installed and enabled, they'll send the notifications out.

You can also set up notifications inside SMF itself. There's a decent intro available for this feature, although you should note that illumos doesn't currently have any of the man pages referred to. This functionality uses the listnotify, setnotify, and delnotify subcommands to svccfg. The one thing that isn't often covered is the relationship between the SMF and the FMA notifications - it's important to understand that both exist, in a strangely mingled state, with some non-obvious overlap.

You can see the global SMF notifications with
/usr/sbin/svccfg listnotify -g
This will come back with nothing by default, so the only thing you'll see is the FMA notifications. To get SMF to email you if any service goes offline, then

/usr/sbin/svccfg setnotify -g to-offline

and you can set this up at a per-service level with

/usr/sbin/svccfg -s apache22 setnotify to-offline

Now, while the SMF notifications can be configured in a very granular manner - you can turn it on and off by service, you can control exactly which state transitions you're interested in, and you can route individual notifications to different destinations, when it comes to the FMA notifications all you have is a big hammer. It's all or nothing, and you can't be selective on where notifications end up (beyond the smtp vs snmp vs syslog channels).

This is unfortunate, because SMF isn't the only source of telemetry that gets fed into FMA. In particular, the system hardware and ZFS will generate FMA events if there's a problem. If you want to get notifications from FMA if there's a problem with ZFS, then you're also going to get notified if an SMF service breaks. In a development environment, this might happen quite a lot.

Perhaps the best compromise I've come up with is to have FMA notifications disabled in a non-global zone, and configure SMF notifications explicitly there. Then, just have FMA notifications in the global zone. This assumes you have nothing but applications in zones, and all the non-SMF events will get caught in the global zone.