Friday, October 02, 2015

Notifications from SMF (and FMA)

In illumos, it's possible to set the system up so that notifications are sent whenever anything happens to an SMF service.

Unfortunately, however, the illumos documentation is essentially non-existent, although looking at the Solaris documentation on the subject should be accurate.

The first thing is that you can see the current notification state by looking at svcs -n:

Notification parameters for FMA Events
    Event: problem-diagnosed
        Notification Type: smtp
            Active: true
            reply-to: root@localhost
            to: root@localhost

        Notification Type: snmp
            Active: true

        Notification Type: syslog
            Active: true

    Event: problem-repaired
        Notification Type: snmp
            Active: true

    Event: problem-resolved
        Notification Type: snmp
            Active: true

The first thing to realize here is that first line - these are the notifications sent by FMA, not SMF. There's a relationship, of course, in that if an SMF service fails and ends up in maintenance, then an FMA event will be generated, and notifications will be sent according to the above scheme.

(By the way, the configuration for this comes from the svc:/system/fm/notify-params:default service, which you can see the source for here. And you'll see that it basically matches exactly what I've shown above.)

Whether you actually receive the notifications is another matter. If you have syslogd running, which is normal, then you'll see the syslog messages ending up in the log files. To get the email or SNMP notifications relies on additional service. These are


and if these are installed and enabled, they'll send the notifications out.

You can also set up notifications inside SMF itself. There's a decent intro available for this feature, although you should note that illumos doesn't currently have any of the man pages referred to. This functionality uses the listnotify, setnotify, and delnotify subcommands to svccfg. The one thing that isn't often covered is the relationship between the SMF and the FMA notifications - it's important to understand that both exist, in a strangely mingled state, with some non-obvious overlap.

You can see the global SMF notifications with
/usr/sbin/svccfg listnotify -g
This will come back with nothing by default, so the only thing you'll see is the FMA notifications. To get SMF to email you if any service goes offline, then

/usr/sbin/svccfg setnotify -g to-offline

and you can set this up at a per-service level with

/usr/sbin/svccfg -s apache22 setnotify to-offline

Now, while the SMF notifications can be configured in a very granular manner - you can turn it on and off by service, you can control exactly which state transitions you're interested in, and you can route individual notifications to different destinations, when it comes to the FMA notifications all you have is a big hammer. It's all or nothing, and you can't be selective on where notifications end up (beyond the smtp vs snmp vs syslog channels).

This is unfortunate, because SMF isn't the only source of telemetry that gets fed into FMA. In particular, the system hardware and ZFS will generate FMA events if there's a problem. If you want to get notifications from FMA if there's a problem with ZFS, then you're also going to get notified if an SMF service breaks. In a development environment, this might happen quite a lot.

Perhaps the best compromise I've come up with is to have FMA notifications disabled in a non-global zone, and configure SMF notifications explicitly there. Then, just have FMA notifications in the global zone. This assumes you have nothing but applications in zones, and all the non-SMF events will get caught in the global zone.

Tuesday, September 29, 2015

Improving the foundations of Tribblix

Mostly due to history, the software packages that make up Tribblix come from 3 places.
In the first category, I'm using an essentially unmodified illumos-gate. The only change to the build is the fix for 5188 so that SVR4 packaging has no external dependencies (or the internal one of wanboot). I then create packages, applying a set of transforms (many of which simply avoid delivering individual files that I see no valid reason to ship - who needs ff or volcopy any more?).

The second category is the historical part. Tribblix was bootstrapped from another distro. Owing to the fact that the amount of time I have is rather limited, not all the bits used in the bootstrapping have yet been removed. These tend to be in the foundations of the OS, which makes them harder to replace (simply due to where they sit in the dependency tree).

In the latest update (the 0m16 prerelease) I've replaced several key components that were previously inherited. Part of this is so that I'm in control of these components (which is a good thing), another is simply that they needed upgrading to a newer version.

One key component here is perl. I've been building my own versions of perl to live separately, but decided it was time to replace the old system perl (I was at 5.10) with something current (5.22). This of itself is easy enough. I then have to rebuild illumos because it's connected to perl, and that's a slightly tricky problem - the build uses the Sun::Solaris module, which comes from the build. (Unfortunately it uses the copy on the system rather than the bits it just built.) So I had to pull the bits out from the failed build, install those on the system, and then the build goes through properly.

Another - more critical - component is libxml2. Critical because SMF uses it, so if you get that wrong you break the system's ability to boot. After much careful study of both the OmniOS and OpenIndiana build recipes, I picked a set of options and everything worked first time. Phew!

(Generally, I'll tend to the OpenIndiana way of doing things, simply because that's where the package I'm trying to replace came from. But I usually look at multiple distros for useful hints.)

A broader area was compression support. I updated zlib along with libxml2, but also went in and built my own p7zip, xz, and bzip2, and then started adding additional compression tools such as lzip and pigz.

The work isn't done yet. Two areas I need to look at are the Netscape Portable Runtime, and the base graphics libraries (tiff, jpeg, png). And then there's the whole X11 stack, which is a whole separate problem - because newer versions start to require KMS (which we don't have) or have gone 64-bit only (which is still an open question, and a leap I'm not yet prepared to take).

Monday, September 28, 2015

Trimming the Tribblix live image

When Tribblix boots from the installation ISO, it reads in two things: the root archive, as a ramdisk, and /usr mounted from solaris.zlib via lofi.

In preparation for the next update, I've spent a little time minimizing both files. Part of this was alongside my experiments on genuinely memory-constrained systems; working out what's necessary in extreme cases can guide you into better behaviour in normal circumstances. While I don't necessarily expect installing onto a 128M system to be a routine occurrence, it would be good to keep 1G or even 512M within reach.

One of the largest single contributors to /usr was perl. It turns out that the only critical part of the system that needs perl is intrd, which is disabled in the live environment anyway. So, perl's not needed.

Another significant package is GNU coreutils. On closer investigation, the only reason I needed this was for a script that generated a UUID which is set as a ZFS property on the root file system (it's used by beadm to match up which zone BE matches the system BE). Apart from the fact that this functionality has recently been integrated into illumos, using the GNU coreutils was just being lazy (perhaps it was necessary under Solaris 10, where this script originated, but the system utilities are good enough now).

I also had the gcc runtime installed. The illumos packages don't need it, but some 3rd-party packages did - compile with gcc and you tend to end up with libgcc_s being pulled in. There are a variety of tricks with -nostdlib and -static-libgcc that are necessary to avoid this. (And I wish I understood better exactly what's happening, as it's too much like magic for my liking.)

The overall impact isn't huge, but the overall footprint of the live image has been reduced by 25%, which is worthwhile. It also counteracts the seemingly inevitable growth of the base system, so I have to worry less about whether I can justify every single driver or small package that might be useful.

Friday, September 25, 2015

illumos pureboot

In my previous article, I discussed an experiment in illumos minimization.

Interestingly, there was a discussion on IRC that wandered off into realms afar, but it got me thinking about making oddball illumos images.

So then I thought - how would you build something that was pure illumos. As in illumos, the whole of illumos, and nothing but illumos.

Somewhat surprisingly, this works. For some definition of works, anyway.

The idea is pretty simple. After building illumos, you end up with the artefacts that are created by the build populating a proto area. This has the same structure as a regular system, so you can find usr/bin/ls under there, for example.

So all I do is create a bootable image that is the proto area from a build.

The script is here.

What does this do?
  • Copies the proto area to a staging area, so it can be manipulated
  • Modifies inittab
  • Sets up grub for boot
  • Copies the kernel state files from the running system (otherwise, they're blank and the kernel is clueless)
  • Creates a block device and builds a ufs file system on it
  • Copies the staging area to it
  • Compresses the block device
  • Copies it back to the platform staging area
  • Creates a bootable iso
There's a bit more detail, but those are the salient points.

My (non-debug) proto area seems to be around 464MB, so a 512MB ramdisk works just fine. You could start deleting (or, indeed, adding) stuff to tune the image you create. The ISO image is 166MB, ready for VirtualBox.

The important thing to realise is that illumos, of itself, does not create a complete operating system. Even core OS functionality requires additional third-party components, which will not be present in the image you create. In particular, libxml2, zlib, and openssl are missing. What this means is that anything depending on these will not function. The list of things that won't work includes SMF, which is an integral part of the normal boot and operations.

So instead of init launching SMF, I have it run a shell instead. (I actually do this via a script rather than directly from init, this allows me to put up a message, and also allows me to run other startup commands if so desired.)

This is what it looks like:

A surprising amount of stuff actually works in this environment. Certainly most of the standard unix commands that I've tried are just fine. Although it has to be appreciated that none of the normal boot processing has been done at this point, so almost nothing has been set up. (And / won't budge from being read-only which is a little confusing.)

Sunday, September 20, 2015

How low can Tribblix go?

One of the things I wanted to do with Tribblix was to allow it to run in places that other illumos distros couldn't reach.

One possible target here is systems with less than gargantuan memory capacities.

(Now, I don't have any such systems. But VirtualBox allows you to adjust the amount of memory in a guest very easily, so that's what I'm doing.)

I started out by building a 32-bit only image. That is, I built a regular (32- and 64-bit combined) image, and simply deleted all the 64-bit pieces. You can do this slightly better by building custom 32-bit only packages, but it was much simpler to identify all the directories named amd64 and delete them.

(Why focus on 32-bit here? The default image has both a 32-bit and 64-bit kernel, and many libraries are shipped in both flavours too. So removing one of the 32-bit or 64-bit flavours will potentially halve the amount of space we need. It makes more sense to drop the 64-bit files - it's easier to do, and it's more likely that real systems with minimal memory are going to be 32-bit.)

The regular boot archive in Tribblix is 160M in size (the file on the ISO is gzip-compressed and ends up being about a third of that), but it's loaded into memory as a ramdisk so the full size is a hard limit on how much memory you're going to need to boot the ISO. You might be able to run off disk with less, as we'll see later. The 32-bit boot archive can be shrunk to 90M, and still has a little room to work in.

The other part of booting from media involves the /usr file system being a compressed lofi mount from a file. I've made a change in the upcoming release by removing perl from the live boot (it's only needed for intrd, which is disabled anyway if you're booting from media), which saves a bit of space, and the 32-bit version of /usr is about a third smaller than the regular combined 32/64-bit variant. Without any additional changes, it is about 171M.

So, the boot archive takes a fixed 90M, and the whole of /usr takes 171M. Let's call that 256M of basic footprint.

I know that regular Tribblix will boot and install quite happily with 1G of memory, and previous experience is that 768M is fine too.

So I started with a 512M setup. The ISO boots just fine. I tried an install to ZFS. The initial part of the install - which is a simple cpio of the system as booted from media - worked fine, if very slowly. The second part of the base install (necessary even if you don't add more software) adds a handful of packages. This is where it really started to struggle, it just about managed the first package and then ground completely to a halt.

Now, I'm sure you could tweak the system a little further to trim the size of both the boot archive and /usr, or tweak the amount of memory ZFS uses, but we're clearly close to the edge.

So then I tried exactly the same setup, installing to UFS instead of ZFS. And it installs absolutely fine, and goes like greased lightning. OK, the conclusion here is that if you want a minimal system with less than 512M of memory, then don't bother with ZFS but aim at UFS instead.

Reducing memory to 256M, the boot and install to UFS still work fine.

With 192M of memory, boot is still good, the install is starting to get a bit sluggish.

If I go down to 128M of memory, the ISO won't boot at all.

However, if I install with a bit more memory, and then reduce it later, Tribblix on UFS works just fine with 128M of memory. Especially if you disable a few services. (Such as autofs cron zones-monitoring zones power fmd. Not necessarily what you want to do in production, but this isn't supposed to be production.)

It looks like 128M is a reasonable practical lower limit. The system is using most of the 128M (it's starting to write data to swap, so there's clearly not much headroom).

Going lower also starts to hit real hard limits. While 120M is still good, 112M fails to boot at all (I get "do_bop_phys_alloc Out of memory" errors from the kernel - see the fakebop source). I'm sure I could go down a bit further, but I think the next step is to start removing drivers from the kernel, which will reduce both the installed boot archive size and the kernel's memory requirements.

I then started to look more closely at the boot archive. On my test machine, it was 81M in size. Removing all the drivers I felt safe with dropped it down to 77M. That still seems quite large.

Diving into the boot archive itself, and crawling through the source for bootadm, I then found that the boot archive was a ufs archive that's only 25% full. It turns out that the boot archive will be hsfs if the system finds /usr/bin/mkisofs, otherwise it uses ufs. And it looks like the size calculation is a bit off, leading to an archive that's massively oversized. After installing mkisofs and rebuilding the boot archive, I got back to something that was 17M, which is much better.

On testing with the new improved boot archive, boot with 96M, or even 88M, memory is just fine.

Down to 80M of memory, and I hit the next wall. The system looks as though it will boot reasonably well, but /etc/svc/volatile fills up and you run out of swap. I suspect this is before it's had any opportunity to add the swap partition, but once it's in that state it can't progress.

Overall, in answer to the question in the title, a 32-bit variant of Tribblix will install (using UFS) on a system with as little as 192M of memory, and run on as little as 96M.

Sunday, September 06, 2015

Fixing SLiM

Having been using my version of SLiM as a desktop login manager for a few days, I had seen a couple of spontaneous logouts.

After minimal investigation, this was a trivial configuration error on my part. And, fortunately, easy to fix.

The slim process is managed by SMF. This ensures that it starts at boot (at the right time, I've written it so that it's dependent on the console-login service, so it launches immediately the CLI login is ready) and that it gets restarted if it exits for whatever reason.

So I had seen myself being logged out on a couple of different occasions. Once when exiting a VNC session (as another user, no less); another whilst running a configure script.

A quick look at the SMF log file, in /var/svc/log/system-slim:default.log, gives an immediate hint:
[ Sep  5 13:37:17 Stopping because process dumped core. ]
So, a process in the slim process contract - which is all processes launched from the desktop - dumped core, SMF spotted it happening, and restarted the whole desktop session. You really don't want that, especially as as a desktop session can be comprised of essentially arbitrary applications, and random core dumps are not entirely unexpected.

So, the fix is a standard one, which I had forgotten entirely. Just add the following snippet to the SMF manifest:

<property_group name='startd' type='framework'>
     <!-- sub-process core dumps shouldn't restart session -->
     <propval name='ignore_error' type='astring'
         value='core,signal' />
and everything is much better behaved.

Thursday, September 03, 2015

Tribblix Graphical Login

Up to this point, login to Tribblix has been very traditional. The system boots to a console login, you enter your username and password, and then start your graphical desktop in whatever manner you choose.

That's reasonable for old-timers such as myself, but we can do better. The question is how to do that.

OpenSolaris, and thus OpenIndiana, have used gdm, from GNOME. I don't have GNOME, and don't wish to be forever locked in dependency hell, so that's not really an option for me.

There's always xdm, but it's still very primitive. I might be retro, but I'm also in favour of style and bling.

I had a good long look at LightDM, and managed to get that ported and running a while back. (And part of that work helped get it into XStreamOS.) However, LightDM is a moving target, it's evolving off in other directions, and it's quite a complicated beast. As a result, while I did manage to get it to work, I was never happy enough to enable it.

I've gone back to SLiM, which used to be hosted at BerliOS. The current source appears to be here. It has the advantage of being very simple, with minimal dependencies.

I made a few modification and customizations, and have it working pretty well. As upstream doesn't seem terribly active, and some of my changes are pretty specific, I decided to fork the code, my repo is here.

Apart from the basic business of making it compile correctly, I've put in a working configuration file, and added an SMF manifest.

SLiM doesn't have a very good mechanism for selecting what environment you get when you log in. By default it will execute your .xinitrc (and fail horribly if you don't have one). There is a mechanism where it can look in /usr/share/xsessions for .desktop files, and you can use F1 to switch between them, but there's no way currently to filter that list, or tell it what order to show then in, or have a default. So I switched that bit off.

I already have a mechanism in Tribblix to select the desktop environment, called tribblix-session. This allows you to use the setxsession and setvncsession commands to define which session you want to run, either in regular X (via the .xinitrc file) or using VNC. So my SLiM login calls a script that hooks into and cooperates with that, and then falls back on some sensible defaults - Xfce, MATE, WindowMaker, or - if all else fails - twm.

It's been working pretty well so far. It can also do automatic login for a given user, and there are magic logins for special purposes (console, halt, and reboot, with the root password).

Now what I need is a personalized theme.