Wednesday, May 18, 2016

Installing Tribblix into an existing pool

The normal installation method for Tribblix is the live_install.sh script.

This creates a ZFS pool for installation into, creates file systems, copies the OS, adds packages, makes a few customizations, installs the bootloader, and not much else. It's designed to be simple.

(There's an alternative, to install to a UFS file system. Not much used, but it's kept just to ensure no insidiuous dependencies creep into the regular installer, and is useful for people with older underpowered systems.)

However, if you've already got an illumos distro installed, then you might already have a ZFS pool, and it might also have some useful data you would rather not wipe everything out. Is there not a way to create a brand new boot environment in the existing pool and install Tribblix to that, preserving all your data?

(Remember, also, that  ZFS encourages the separation of OS and data. So you should be able to replace the OS without disturbing the data.)

As of Tribblix Milestone 17, this will work. Booting from the ISO and logging in, you'll find a script called over_install.sh in root's home directory. You can use that instead of live_install.sh, like so:

./over_install.sh -B rpool kitchen-sink

You have to give it the name of the existing bootable pool, usually rpool. It will do a couple of sanity checks to be sure this pool is suitable, but will then create a new BE there and install to that.

Arguments after the pool name are overlays, specifying what software to install, just like the regular install.

It will update grub for you, so that you have a grub on the pool that is compatible with the version of illumos you've just added. With -B, it will update the MBR as well.

It copies some files, the minimum that define the system's identity, from the existing bootable system into the new BE. This basically copies across user accounts (group, passwd, shadow files) and the system's ssh keys, but nothing else.

Any existing zfs file systems are untouched, and will be present in the new system. You'll have to import any additional zfs pools, though.

When you boot up after this, the grub menu will just contain the new BE you just created. However, any old boot environments are still present, so you can still see them, and manipulate them, using beadm. In particular, you can mount up and old BE (in case there are important files you need to get back), and activate an old BE so you can boot into the old system if so desired.

Amongst other things, you can use this as a recovery tool, when your existing system has a functioning root pool but won't boot.

I've also used this to "upgrade" older Tribblix systems. While there is an upgrade mechanism, this really only works (a) for very recent releases, and (b) to update one release at a time. With this new mechanism, I can simply stick a new copy of Milestone 17 on an old box, and enjoy the new version while having all my data intact.

Tuesday, May 17, 2016

Updating Tribblix for SPARC

Having just released an updated version of Tribblix for x86, a little commentary on the status of the SPARC version is probably in order.

After a rather extended delay, there is an updated version of Tribblix for SPARC available for download.

This is Milestone 16, so it's at the same level as the prior x86 release. More precisely, it's built from exactly the same illumos source as the x86 Milestone 16 release was. Yes, this means that it's a bit dated, but it's consistent, and there have been a number of breaking changes for SPARC builds introduced (and fixed) in the meantime.

I did want to get a release out of the door, before bumping the version to 0m17 ready for the next x86 build. Part of this was simply to ensure that I could actually still build a release - it gets tested on x86 all the time, but as it happens I had never built a SPARC release on my current infrastructure.

In terms of additional packages, the selection is still rather sparse. I haven't had the time to build up the full list. This isn't helped by the fact that my SPARC kit is rather slow, so building anything for SPARC simply takes longer. Much longer. (Not to mention the fact that they're noisy and power-hungry.)

It's not just time that's the problem. I've had some difficulty building certain packages. I'm not talking the likes of Go and Node, which I can simply ignore as they're not ported to SPARC at all, but some reasonably common packages would fail with obscure (and unexpected) build errors. If there's a problem with one component, that blocks anything dependent on it too.

Other than expanding the breadth of available packages, the key next steps are (i) to make Tribblix on SPARC self-hosting, like the x86 version has been for a very long time, and (ii) to try and keep the SPARC release more closely aligned so it doesn't drift away from the x86 release and require additional effort to bring back in sync.

Signing Packages in Tribblix

On any computer system you want to know exactly what software is installed and running.

Tribblix uses SVR4 packaging, so you can easily see what's installed. In addition, there are mechanisms - pkgchk - to compare what's on the disk with what the packaging system thinks should be there. But that's just a consistency check, it doesn't verify that the package installed is actually the one you wanted.

Tribblix has had simple integrity checking for a while. The catalog for a package repository includes both the expected size and the md5 checksum of a package. This is largely aimed at dealing with download errors - network drops, application errors, or errant intrusion detection systems mangling the data. In practice, because the downloaded packages are actually zip files, which have inbuilt consistency checking and the catalog at the end of the file, and because SVR4 packaging has its own consistency checks on package contents, the chances of a faulty download getting installed are remote, the checking is so that the layer above can make smart decisions in the case of failure.

But you want to be sure that, not only has the package you downloaded made it across the network intact, but that the source package is legitimate. So the packages are signed using gnupg, and will be verified upon download in upcoming releases. Initially this is just a warning check while the mechanisms get sorted out.

The actual signing and verification part is the easy bit, it's all the framework around it that takes the time to write and test.

One possibility would have been to sign the package catalogs, and use that to prove that the checksum is correct. That's not enough, for a couple of reasons. First, the catalog only includes current package versions, so there would be no way to verify prior versions. Second, there's no reason somebody (or me) couldn't take a subset of packages and create a new repo using them; the modified catalog couldn't be verified. In either case, you need to be able to verify individual packages. (But the package catalog should also be signed, of course.)

It turns out there's not much of a performance hit. Downloads are a little slower, because there's an extra request to get the detached signature, but it's a tiny change overall.

With this in place, you can be sure that whatever you install on Tribblix is legitimate. But all you're doing is verifying the packages at download time. This leaves open the problem of being able to go to a system and ask whether the installed files are legitimate. Yes, there's pkgchk, but there's no validated source of information for it to use as a reference - the contents file is updated with every packaging operation, so it clearly can't be signed by me each time.

This is likely to require the additional creation of a signed manifest for each package. This partially exists already, as the pkgmap fragments for each package are saved (in the global zone, anyway), and those could be signed (as they don't change) and used as the input to pkgchk. However, the checksums in the pkgmap and contents files aren't particularly strong (to put it mildly), so that file will need to be replaced by something with much stronger checksums.

Initial support for signed packages is available starting with the Tribblix Milestone 17 release. At this point, it will check the package signatures, but not act on them, enforcement will probably come in the next release when I can be reasonably sure that everything is actually working correctly.

Saturday, April 02, 2016

Minimal illumos networking

Playing around with Minimal Viable Illumos, it becomes clear how complex a system illumos is. Normally, if you're running a full system, everything is taken care of for you. In particular, SMF starts all the right services for you. Looking through the method scripts, there's a lot of magic happening under the hood.

If you start with a bare install, booted to a shell, with no startup scripts having run, you can then bring up networking automatically.

The simplest approach would be to use ifconfig to set up a network interface. For example

ifconfig e1000g0 plumb 10.0.0.1 up

This won't work. It will plumb the interface, but not assign the address. To get the address assigned, you need to have ipmgmtd running.

env SMF_FMRI=svc/net/ip:d /lib/inet/ipmgmtd

Note that SMF_FMRI is set. This is because ipmgmtd expects to be run under SMF, and checks for SMF_FMRI being set as the determinant. It doesn't matter what the value is, it just needs to exist.

OK, so if you know your network device and your address, that's pretty much it.

If you use dladm to look for interfaces, then you'll see nothing. In order for dladm show-phys to enumerate the available interfaces, you need to start up dlmgmtd and give it a poke

env SMF_FMRI=svc/net/dl:d /sbin/dlmgmtd
dladm init-phys

Note that SMF_FMRI is set here, just like for ipmgmtd.

At this point, dladm show-phys will give you back the list of interfaces as normal.

You don't actually need to run dladm init-phys. If you know you have a given interface, plumbing it will poke enough of the machinery to make it show up in the list.

If you don't know your address, then you might think of using dhcp to assign it. This turns out to require a bit more work.

The first thing you need to do is bring up the loopback.

ifconfig lo0 plumb 127.0.0.1 up
ifconfig lo0 inet6 plumb ::1 up

This is required because dhcpagent wants to bind to localhost:4999, so you need the loopback set up for it to be able to do that. (You don't necessarily need the IPv6 version, that's for completeness.)

Then

ifconfig e1000g0 plumb
ifconfig e1000g0 auto-dhcp primary

Ought to work. Sometimes it does, sometimes it doesn't.

I sometimes found that I needed to

dladm init-secobj

in order to quieten errors that occasionally appear, and sometimes need to run

ifconfig -adD4 auto-revarp netmask + broadcast + up

although I can't even imagine what that would do to help, I presume that it kicks enough of the machinery to make sure everything is properly initialized.

Within the context of mvi, setting a static IP address means you need to craete a new image for each IP address, which isn't actually too bad (creating mvi images is really quick), but isn't sustainable in the large. DHCP would give it an address, but then you need to track what address a given instance has been allocated. Is there a way of doing static IP by poking the values from outside?

A tantalizing hint is present at the end of /lib/svc/method/net-physical, where it talks about boot properties being passed in. The implementation there appears to be aimed at Xen, I'll have to investigate if it's possible to script variables into the boot menu.

Monday, March 28, 2016

Running illumos in 48M of RAM

Whilst tweaking mvi recently, I went back and had another look at just how minimal an illumos install I could make.

And, given sufficiently aggressive use of the rm command, the answer appears to be that it's possible to boot illumos in 48 meg of RAM.

No, it's not April 1st. 48 meg of RAM is pretty low. It's been a long time since I've seen a system that small.

I've added some option scripts to the mvi repo. (The ones with -fix.sh as part of their names.) You don't have to run mvi to see what these do.

First, I start with mvix.sh, which is fairly minimal up front.

Then I go 32bit, which halves the size.

Then I apply the extreme option, which removes zfs (it's the single biggest driver left), along with a bunch of crypto and other unnecessary files. And I clean up lots of bits of grub that aren't needed.

I then abstracted out a nonet and nodisk option. The nonet script removes anything that looks like networking from the kernel, and the bits of userland that I add in order to be able to configure an interface. The nodisk script removes all the remaining storage device drivers (I only included the ones that you normally see when running under a hypervisor in the first place), along with the underlying scsi and related driver frameworks.

What you end up with is a 17M root file system, which compresses down to a 6.8M root archive, which gets packaged up in an 8.7M iso.

For those interested the iso is here. It should run in VirtualBox - in 32-bit mode, and you should be able to push the memory allocated by VirtualBox down to 48M. Scary.

Of course, it doesn't do much. It boots to a prompt, the only tools you have are ksh, ls, and du.

(Oh, and if you run du against the devices tree, it will panic.)

While doing this, I found that there are a lot of dependencies between modules in the illumos kernel. Not all of them are obvious, and trying to work out what's needed and what can be removed has involved large amounts of trial and error. That said, it takes less than 5 seconds to create an iso, and it takes longer for VirtualBox to start than for this iso to boot, so the cycle is pretty quick.

Sunday, March 27, 2016

Almost there with Solarus

Every so often I'll have a go at getting some games running on Tribblix. It's not a gaming platform, but having the odd distraction can't hurt.

I accidentally stumbled across Solarus, and was immediately intrigued. Back in the day, I remember playing Zelda on the SNES. A few years ago it was released for the Game Boy Advance, and I actually went out and bought the console and the game. I haven't added any more games, nothing seemed compelling, although we had a stock of old Game Boy games (all mono) and those still worked, which is great.

So I thought I would try and build Solarus for Tribblix. Having had a quick look at the prerequisites, most of those would be useful anyway so the effort wouldn't be wasted in any event.

First up was SDL 2.0. I've had the older 1.2.15 version for a while, but hadn't had anything actually demand version 2 yet. That was easy enough, and because all the filenames are versioned, it can happily be installed alongside the older version.

While I was at it, I installed the extras SDL_image, SDL_net, SDL_ttf, smpeg, and SDL_mixer. The only build tweak needed was to supply LIBS="-lsocket -lnsl" to the configure script for smpeg and SDL_net. I needed to version the plaympeg binary installed by smpeg to avoid a conflict with the older version of smpeg, but that was about it.

Then came OpenAL, which turned out to be a bit tricky to find. Some of the obvious search results didn't seem appropriate. I have no idea whether the SourceForge site is the current one, but it appears to have the source code I needed.

Another thing that looks abandoned is modplug, where the Solarus folks have a github mirror of the copy that's right for them.

Next up, PhysicsFS. This isn't quite what you expect from the name, it's an abstraction layer that replaces traditional file system access.

On to Solarus itself. This uses cmake to build, and I ended up with the following incantation in a new empty build directory:

cmake ../solarus-1.4.5 \
 -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=NO \
 -DCMAKE_INSTALL_RPATH=/opt/solarus/lib \
 -DSOLARUS_USE_LUAJIT=OFF \
 -DCMAKE_INSTALL_PREFIX=/opt/solarus

Let's go through that. The CMAKE_INSTALL_PREFIX is fairly obvious - that's where I'm going to install it. And SOLARUS_USE_LUAJIT is necessary because I've got Lua, but have never had LuaJIT working successfully.

The two RPATH lines are necessary because of the strange way that cmake handles RPATH. Usually, it builds with RPATH set to the build location, then installs with RPATH stripped. This is plain stupid, but works when you simply dump everything into a single swamp. So you need to manually force it to put the correct RPATH into the binary (which is the sort of thing you would actually want a build system to get right on its own).

Unfortunately, it doesn't actually work properly. There are two problems which I haven't really had a chance to look at - the first is that it fails fatally with an Xlib error if I move the mouse (which is a little odd as it doesn't actually use the mouse); the second is that it runs an order of magnitude or two slower than useful, so I suspect a timing error.

Still, the build is pretty simple and it's so close to working that it would be nice to finish the job.

Saturday, March 26, 2016

Tweaking MVI

A few months ago I first talked about minimal viable illumos, an attempt to construct a rather more minimalist bootable copy of illumos than the gigabyte-size image that are becoming the norm.

I've made a couple of changes recently, which are present in the mvi repository.

The first is to make it easier for people who aren't me (and myself when I'm not on my primary build machine) to actually use mvi. The original version had the locations of the packages hardcoded to the values on my build machine. Now I've abstracted out package installation, which eliminates quite a lot of code duplication. And then I added an alternative package installation script which uses zap to retrieve and install packages from the Tribblix repo, just like the regular system does. So you can much more easily run mvi from a vanilla Tribblix system.

What I would like to add is a script that can run on pretty much any (illumos) system. This isn't too hard, but would involve copying most of the functionality of zap into the install script. I'm holding off for a short while, hoping that a better mechanism presents itself. (By better, what I mean is that I've actually got a number of image creation utilities, and it would be nice to rationalise them rather than keep creating new ones.)

The second tweak was to improve the way that the size of the root archive is calculated, to give better defaults and adapt to variations more intelligently.

There are two slightly different mechanisms used to create the image. In mvi.sh, I install packages, and then delete what I'm sure I don't need; with mvix.sh I install packages and then only take the files I do need. The difference in size is considerable - for the basic installation mvi.sh is 127M and mvix.sh is 57M.

Rather than a common base image size of 192M, I've set mvi.sh to 160M and mvix.sh to 96M. These sizes give a reasonable amount of free space - enough that adding the odd package doesn't require the sizes to be adjusted.

I then have standard scripts to construct 32-bit and 64-bit images. A little bit of experimentation indicates that the 32-bit image ends up being half the size of the base, whereas the 64-bit image comes in at two thirds. (The difference is that in the 32-bit image, you can simply remove all 64-bit files. For a 64-bit kernel, you still need both 32-bit and 64-bit userland.) So I've got those scripts to simply scale the image size, rather than try and pick a new number out of the air.

I also have a sample script to install Node.js. This again modifies the image size, just adding the extra space that Node needs. I've had to calculate this more accurately, as reducing the size of the base archive gave me less margin for error.

(As an aside, adding applications doesn't really work well in general with mvix.sh, as it doesn't know what dependencies applications might need - it only installs the bare minimum the OS needs to boot. Fortunately Node is fairly self-contained, but other applications are much less so.)

Sunday, March 13, 2016

Software selection - choice or constraint?

In Tribblix, software is preferentially managed using overlays rather than packages.

Overlays comprise a group of packages bundled together to supply a given software need - the question should be "what do you want to do?", and packages (and packaging) are merely an implementation artifact in providing the answer to that question.

Part of the idea was that, not only would the overlays match a user's mental model of what they want to install, but that there would be many fewer overlays than packages, and so it's much easier for the human brain to track that smaller number of items.

Now, it's true that there are fewer overlays than available packages. As of right now, there are 91 overlays and 1237 packages available for Tribblix. So that's better than an order of magnitude reduction in the number of things, and an enormous reduction in the possible combinations of items. However, it's no longer small in the absolute sense.

(Ideally, small for me means that it can be all seen on screen at once. If you have to page the screen to see all the information, your brain is also paging stuff in and out.)

So I've been toying with the idea of defining a more constrained set of overlays. Maybe along the lines of a desktop and server split, with small, large, and developer instances of each.

This would certainly help dramatically in that it would lead to significant simplification. However, after trying this out I'm unconvinced. The key point is that genuine needs are rather more complicated than can be addressed by half a dozen neat pigeonholes. (Past experience with the install metaclusters in Solaris 10 was also that they were essentially of no use to any particular user, they always needed to be customised.)

By removing choice, you're seriously constraining what users can do with the system. Worse, by crippling overlays you force users back into managing packages which is one of the things I was trying to avoid in the first place.

So, I'm abandoning the idea of removing choice, and the number of overlays is going to increase as more applications are added. Which means that I'm going to have to think a lot harder about the UX aspect of overlay management.