Minimizing apache and PHP

Recently I was looking at migrating a simple website in which every page but one was static.

The simplest thing here would be to use nginx. It's simple, fast, modern, and should make it dead easy to get an A+ on the Qualys SSL test.

But that non-static page? A trivial contact form. Fill in a box, the back-end sends the content of the box as an email message.

The simplest thing here in days gone by would have been to put together a trivial CGI script. Only nginx doesn't do CGI, at least not directly. Not only that, but writing the CGI script and doing it well is pretty hard.

So, what about PHP? Now, PHP has gotten itself a not entirely favourable reputation on the security front. Given the frequent security updates, not entirely undeserved. But could it be used for this?

For such a task, all you need is the mail() function. Plus maybe a quick regex and some trivial string manipulation. All that is in core, so you don't need very much of PHP at all. For example, you could use the follwoing flags to build:

--disable-all
--disable-cli
--disable-phpdbg

So, no modules. Far less to go wrong. On top  of that, you can disable a bunch of things in php.ini:

file_uploads = Off [change]
allow_url_fopen = Off [change]
allow_url_include = Off [default]
display_errors = Off [default]
expose_php=Off [change]

Furthermore, you could start disabling functions to your heart's content:

disable_functions = php_uname, getmyuid, getmypid, passthru, leak, listen, diskfreespace, tmpfile, link, ignore_user_abord, shell_exec, dl, set_time_limit, exec, system, highlight_file, source, show_source, fpaththru, virtual, posix_ctermid, posix_getcwd, posix_getegid, posix_geteuid, posix_getgid, posix_getgrgid, posix_getgrnam, posix_getgroups, posix_getlogin, posix_getpgid, posix_getpgrp, posix_getpid, posix, _getppid, posix_getpwnam, posix_getpwuid, posix_getrlimit, posix_getsid, posix_getuid, posix_isatty, posix_kill, posix_mkfifo, posix_setegid, posix_seteuid, posix_setgid, posix_setpgid, posix_setsid, posix_setuid, posix_times, posix_ttyname, posix_uname, proc_open, proc_close, proc_get_status, proc_nice, proc_terminate, phpinfo

Once you've done that, you end up with a pretty hardened PHP install. And if all it does is take in a request and issue a redirect to a static target page, it doesn't even need to create any html output.

Then, how to talk to PHP?  The standard way to integrate PHP with nginx is using FPM. Certainly, if this was a high or even moderate traffic site, then that would be fine. But that involves leaving FPM running permanently, and is a bit of a pain and a resource hog for one page that might get used once a week.

So how about forwarding to apache? Integration using mod_php is an absolute doddle. OK, it's still running permanently, but you can dial down the process count and it's pretty lightweight. But we have a similar issue to the one we faced with PHP - the default build enables a lot of things we don't need. I normally build apache with:

--enable-mods-shared=most
--enable-ssl

but in this case you can reduce that to:

--enable-modules=few
--disable-ssl

now, there is the option of --enable-modules=none, but I couldn't actually get apache to start at all with that - some modules appear to be essential (mod_authz_host, mod_dir, mod_mime, and mod_log_config at least), and going below the "few" setting is entering unsupported territory.

You can restrict apache even further with configuration, just enable PHP, return an error for any other page, listen only on localhost. (I like the concept of the currently experimental mod_allowmethods as we might only want POST for this case. Normally disabling methods with current apache version involves mod_rewrite, which is one of the more complex modules.)

In the end, we elected to solve the problem a different way, but it was still an instructive exercise.

The above would be suitable for one particular use case. For a general service, it would be completely useless. Most providers and distributions tend to build with the kitchen sink enabled, because you don't know what your users or customers might need at runtime. They might build everything as a shared module, and could package every module in a separate package (although this ends up being a pain to manage); or you rely on the user to explicitly enable and disable modules as necessary.

In Tribblix, I've tended to avoid breaking something like apache or PHP up into multiple packages.  There's one exception, which is that the PHP interface to postgresql is split out into a separate package. This is simply because it links against the postgresql shared library, so I ship that part separately to avoid forcing postgresql to be installed as a dependency.

Building Tribblix packages

Software in Tribblix is delivered in packages, which come from one of three sources - an illumos build, a bootstrap distribution (OpenIndiana or OpenSXCE depending on hardware architecture), and native Tribblix packages.

The illumos packages are converted from the IPS repo created during a build of illumos-gate using the repo2svr4 script in the tribblix-build repo. There's also a script ips2svr4 in the same repo that's used to construct an SVR4 package from that installed on a system using IPS packaging, such as OpenIndiana.  The OpenSXCE packages are shipped as-is.

(The use of another distro to provide components was expedient during early bootstrapping. Over time, the fraction of the OS provided by that other distribution has shrunk dramatically. At the present time, it's mostly X11.)

What of the other packages, those natively built on Tribblix?

Those are described in the build repo.

In the build repo, there are a number of top-level scripts. Key of these is dobuild, which is the primary software builder. Basically, it unpacks a source archive, runs configure, make, and make install. It can apply patches, run scripting before and after the configure step, and knows how to handle most things that are driven by autoconf.

There are some other scripts of note. The genpkg and create_pkg scripts go from a build to a package. The pkg_tarball script is an easy way to do a straight conversion of an archive to a package. There are scripts to create the package catalogs.

For each package, there is a directory named after the package, contains files used in the build.

At the very minimum, you need a pkginfo file (this is a fragment, the rpocess creates the rest of the actual pkginfo file). There's the possibility of using fixit and fixinstall scripts to fix up any errant behaviour from the make install step before actually creating packages. There are depend files listing package dependencies, and alias files listing user-friendly aliases for packages.

However, how do you know how a package was actually built? Even for packages created with the dobuild script, there are a lot of flags that could have been provided. And a lot of software doesn't fit into the configure style of build in any case.

What I actually did was have a big text file containing the commands I used to build each package. Occasionally with some very unprintable comments about some of the steps I had to take to get things to build. (So simply adding that file to the repo was never going to be a sensible way forward.)

So what I've done is split those notes up and created a file build.sh for each package, which contains the instructions used to create that package. It assumes that the THOME environment variable points to the parent of the build repo, and that there's a parallel tarballs directory containing the archives. (Many of the scripts, unfortunately, assume a certain value for that location, which is the location on my own machine. Yes, that should be fixed.)

There are a number of caveats here.

The first is that some packages don't have a build.sh file. Yet. Some of these are my own existing packages, which were built outside Tribblix. Some go back to the very earliest days, and notes as to how they were built have been lost in the mists of time - these will be added whenever that package is next built.

The second is that the build recipe was valid at the time it was last used. If you were to run the recipe now, it might not work, due to changes in the underlying system - packages are not rebuilt unless they need to be, so the recipes can go all the way back to the very first release. It might not generate the same output. (This is really autoconf, which gropes around the system looking for things it can use, so running it again might pull in additional dependencies. Occasionally this causes problems and I need to explicitly enable or disable certain features. In some cases, you have to uninstall packages to make the build run in a sane manner.)

The third is that, while the build recipe looks like a shell script, and in many cases will actually function as such, it's really a recipe that you cut and paste into a terminal. At least, that's what I do. Sometimes it's necessary, because there was some manual hacky workaround I needed that's just in the build script as a comment.

This has been an outstanding TODO item for a while now, so I'm glad to have got it out of the way.

Getting to grips with Docker

A while ago, I described how we took an existing application build script and managed to run it inside Docker.

Having played with this inside Docker a little more, it's probably worth scribbling down a few notes I happened to stumble across on the way.

I'm looking at having 2 basic images: as a foundation, Ubuntu with all the packages we want added; then an image that inherits FROM that with our application stack built and installed (but not configured). The idea behind this layering is simply to separate the underlying OS, which is fairly standard, from the unique stuff that is all ours.

Then, you create an instance image from the application image, simply by running a configuration script that you COPY in. Once you've got a configured application instance, you create a volume container from it, and then run the application image using the volume(s) from the instance image. You keep that volume container around, just as a home for your data, essentially forever. And you can run multiple application instances from the same base image, you just need to configure and create a volume container for each instance.

That's a brief overview of the workflow, now some tweaks and pitfalls.

We're using Ubuntu, so the first step is to run apt-get with our list of packages. This originally created a 965MB image. It's not going to be small, we need both java and a full development stack to create our application.

However, some of the stuff installed we'll never need. Using the --no-install-recommends flag to apt-get saved us about 150M. The recommends list is stuff that might be useful, but not essential. But remember - our Docker container is only ever going to run a fixed set of applications, so we'll never need any of the optional stuff. The only thing to be careful of here is if you accidentally depend on something in the recommends list without realizing you're only getting it indirectly.

We can do slightly better in terms of saving space. We use postgresql, but get it to store the database files in our own locations, so we can remove /var/lib/postgresql/9.X and what's underneath it, saving almost another 40M.

One thing to be aware of is that the list of packages in the official Ubuntu Docker image isn't quite the same as you would get from a regular Ubuntu install. There are one or two packages we didn't bother adding because they were there in a regular install that we need to add with Docker. Things like sudo and wget are on this list, so I needed to add those to the apt-get list.

Another thing to be aware of is that because you're building images afresh each time, you aren't guaranteed that new users will always get the same uid and gid. If you change the list of packages (even by just adding --no-install-recommends), this might change which users exist, and that affects the uid assigned to later users. I got burnt when a later base build ended up giving the postgres user a different uid, so it didn't own its database files on the persistent volume any more. I think the long term fix here is to create the users you need by hand before installing any packages, forcing the uid and gid to known values.

In order to keep image sizes small, you'll often see "rm -rf /var/lib/apt/lists/*" in a Dockerfile. In general, deleting temporary files is a good idea. This includes any files created by your own software deployment stage. Cleaning that up properly saved me another 200M or so in the final image. (Remember to clean up /tmp, that's part of the image too.)

It isn't strictly related to Docker, but I hit an ongoing problem - in some environments I ended up blocking on /dev/random. Search around and you'll find a lot of problems reported, especially related to java and SecureRandom (or, in our case, jruby). Running Docker on my Mac was fine, running it on a server in the cloud gave me 15-minute startup times. The solution here is to add -Djava.security.egd=file:///dev/urandom or -Djava.security.egd=file:/dev/./urandom to your java startup (or JAVA_OPTIONS).

(And, by the way, this illustrates that while Docker can guarantee that your app is the same in all environments, it doesn't magically protect you from differences in the underlying environment that can have a massive impact on your application.)

My application listens on ports 8080 and 8443, which I map on the host to the common ports, with

docker run -p 80:8080 -p 443:8443 ...

This works fine for me in testing, when I'm only running one copy and simply point a browser at the host. Networking gets a whole lot more complicated with multiple containers, although I think something like a load-balancer in front might work.

I've been using the Docker for Mac beta for some of this - while at times it's been beta in terms of stability, generally I can say it's a very impressive piece of work.

Data Destruction and illumos

When disposing of  a computer, you would like to be sure that it has no data on its storage that could be accessed by the direct recipient (or any future recipient). It would be somewhat embarrassing for personal photos to be retrieved; it would be far worse if financial or business data were to be left accessible.

The keywords you're looking for here are data remanence and disk sanitization.

There are three methods to remove data from a disk. Total physical destruction, degaussing, and overwriting the data. The effectiveness of these methods is up for debate; as is the feasibility of a sufficiently determined and well-funded attacker being able to retrieve data.

Here I'm just going to discuss overwriting the disk. For a lot of casual and home purposes that'll be enough, and is a lot better than not bothering at all, or simply reformatting the drive (or reinstalling on OS on it) which will leave a lot of disk sectors untouched and amenable to simply being read off in software.

The standard here seems to be DBAN. However, it's not seen much activity in a while, and was sold to a company that offers a commercial product that's claimed to be much better.

Basically, all DBAN is doing is scribbling over every sector on a drive. That's not hard.

In Solarish systems, format/analyze/purge does essentially the same thing. It's the documented method for wiping hard drives on Solarish style systems.

However, it's a little fiddly to use and requires a modest level of expertise to get that far. You can't purge the disk you're booted from, the solution proposed there is to boot from installation media, drop to a shell, and run format from there. That has a couple of problems - it's still very manual, and the install (or live) media are rather large and can take an age to boot.

So I started to think, how hard could it be to create a minimalist illumos boot media that just contains the format command, and a simple script around it to make it easy to run?

I've already done most of the work, as part of the minimal viable illumos project. It was pretty easy to create a new variant.

The idea is to erase disk drives, so the intended target is physical hardware rather than a hypervisor. So I added a number of common storage drivers to the image. (As an aside, I really have no idea as to what storage HBAs are actually in common use, so which drivers to put in this list or on the Tribblix install iso is largely guesswork.)

There should be no need for networking. You really don't want a mechanism for any external access to the system while the disks are being wiped, so networking is simply not there.

And I added a simple wrapper script that enumerates disk drives and runs the appropriate format commands. If you want to see how this works, just look at the wrapper script. All this is in the mvi repo, see the files with "wipe" in their names.

And there's the (14M in size) iso image I created also available.

(Why is such a small image good, you might ask? Apart from simply being sure that it's only capable of doing the one function that it's advertised for, if you're trying to wipe a remote system mounting the image over the network, then the smaller the better.)

I tested this in VirtualBox, which exposed a few quirks. For one, the defect list switching you'll see in the docs doesn't work there (I have no idea if it's going to work on any real hardware). The other is that the disk image I was using was a file on a compressed zfs file system. The purge process writes a repeating pattern, which is very compressible, so the 1G disk image I was testing only takes up 16M of disk space.

While I don't think it's really a proper alternative to DBAN, I think it's useful as a real-world example of how to use mvi.

Connecting to legacy Sun ILOM with modern clients

The bane of many a system administrator's existence is the remote management capability on their servers. In Sun's case, I'm talking about the ILOM.

(Of course, Sun have had RSC and ALOM and eLOM and maybe some other abominations over time.)

Now, for many purposes, you can just ssh to the ILOM and you're done. On Sun boxes anyway, where you often have serial console redirection and the OS using the serial console.

However, if you want to manage the system fully, you need a proper client. There are a couple of common cases. First, if you need the VGA console (either for a broken OS, or to interact with the BIOS), or if you want to do storage redirection (in other words, you want to remotely present a bootable image).

That's where the fun starts, and you get in a tangled relationship with Java. Often, it ends up being a tale of woe.

And that's on the best of days. With legacy hardware - such as the X4150 - it gets a whole lot more interesting.

Now, while the X4150 is legacy and well past end of life now, it turns out that there was an updated firmware release in 2015. (For POODLE, I think.) If you can, apply this, as it should fix some of the UI compatibility issues with newer browsers. (Not all, I suspect, but if you've tried using a current browser and only got half the GUI then you know what I'm talking about.)

However, that doesn't necessarily mean that the Java application is going to work. There are actually a couple of issues here.

The first is that the application is a signed jar, and the certificate used to sign it has expired. Worse, due to Java's rather chequered security history, current versions have draconian checks in place which you'll run into. To fix, go to the Java Control Panel, down to "Perform signed code verification checks" and change it to "Do not check". Generally, disabling security like this is a bad idea, but in this case it's necessary.

Next, if you start up the application, click through the remaining security dialogs, and try to connect to the console, you'll get a cipher suite mismatch failure. The ILOM is pretty old, and uses SSLv3 which is disabled by default in current Java. You'll need to edit the java.security file (in ${JAVA_HOME}/jre/lib/security/java.security[*]) and comment out two lines - the ones with jdk.certpath.disabledAlgorithms and jdk.tls.disabledAlgorithms, then run the application again.

With luck, that will at least enable you to get to the console.

If you want storage redirection, then you're in for more fun. For starters, you need to be running Solaris, Linux, or Windows. If you're on a Mac, it's not going to work. You'll need to get yourself another machine, or run a VM with something else installed.

And the other thing is that you need to be running a 32-bit Java Virtual Machine. If you're running Solaris, this rules out Java 8 - you'll have to go back to Java 7. On other platforms, you'll have to make sure you have a 32-bit JVM, which might not be the default and you might have to manually install it.

Oh, and if you're on Linux or Solaris and running OpenJDK (rather than the Oracle builds) then you'll need IcedTea to get the javaws integration. At least with IcedTea you can ignore the Java Control Panel stuff.

[*: On my Mac I discovered that I had 2 different installations of Java. The one that you get if you type "java" isn't the same one used for browser integration and javaws launching. Running /usr/libexec/java_home gave me the wrong one; I ended up looking at the ps output when running the Control Panel to find out the location of the one I really needed.]

What's present in libc on illumos?

Over time, operating systems such as illumos gain new functionality. For examples, new functions are added to libc. But how do you know what's there, when was it added, and whether a newer version contains something that's missing?

So perhaps the first place to start is with elfdump. If you run

elfdump -v /lib/libc.so

then you'll get a bunch of lines like so:

     index  version                     dependency
       [1]  libc.so.1                                        [ BASE ]
       [2]  ILLUMOS_0.13                ILLUMOS_0.12        
       [3]  ILLUMOS_0.12                ILLUMOS_0.11        
       [4]  ILLUMOS_0.11                ILLUMOS_0.10        
 ...
      [51]  SUNW_0.8                    SUNW_0.7            
      [52]  SUNW_0.7                    SYSVABI_1.3         
      [53]  SYSVABI_1.3                                     
      [54]  SUNWprivate_1.1                                 

So, these lines correspond to the various released versions of libc. Every time you add a new function to libc, that means a new version of the library. And each version depends on the one before.

These versions are listed in a mapfile, at usr/src/lib/libc/port/mapfile-vers in the illumos source. So, what you can see here is that ILLUMOS_0.13 (which is the version shipped with Tribblix 0m16) is where eventfd got added. If you want strerror_l (which you do if you're building vlc), then you need ILLUMOS_0.14; if you want pthread_attr_get_np (which you need for QT5) then you need ILLUMOS_0.21. (Unfortunately, Tribblix 0m17 only picks up ILLUMOS_0.19.) Looking back, you can see everything exposed by libc and what version of Solaris or illumos it was added in.

Another trick is to run elfdump against a binary. For example:

elfdump -v /usr/gnu/bin/tar

will tell us which versions of which libraries are required. Part of this is:

Version Needed Section:  .SUNW_version
     index  file                        version
       [2]  libnsl.so.1                 SUNW_0.7            
       [3]  libc.so.1                   ILLUMOS_0.8         
       [4]                              ILLUMOS_0.1          [ INFO ]
       [5]                              SUNW_1.23            [ INFO ]
       [6]                              SUNW_1.22.6          [ INFO ]

This is quite informative. It tells you that  gtar calls the newlocale stuff from ILLUMOS_0.8, and a bunch of older stuff. But the point here is that it is calling illumos additions to libc, so this binary won't work on Solaris 10 (and probably not on Solaris 11 either). If you're building binaries for distribution across distros, you can use this information to confirm that you haven't accidentally pulled in functions that might not be available everywhere.

Installing Tribblix into an existing pool

The normal installation method for Tribblix is the live_install.sh script.

This creates a ZFS pool for installation into, creates file systems, copies the OS, adds packages, makes a few customizations, installs the bootloader, and not much else. It's designed to be simple.

(There's an alternative, to install to a UFS file system. Not much used, but it's kept just to ensure no insidiuous dependencies creep into the regular installer, and is useful for people with older underpowered systems.)

However, if you've already got an illumos distro installed, then you might already have a ZFS pool, and it might also have some useful data you would rather not wipe everything out. Is there not a way to create a brand new boot environment in the existing pool and install Tribblix to that, preserving all your data?

(Remember, also, that  ZFS encourages the separation of OS and data. So you should be able to replace the OS without disturbing the data.)

As of Tribblix Milestone 17, this will work. Booting from the ISO and logging in, you'll find a script called over_install.sh in root's home directory. You can use that instead of live_install.sh, like so:

./over_install.sh -B rpool kitchen-sink

You have to give it the name of the existing bootable pool, usually rpool. It will do a couple of sanity checks to be sure this pool is suitable, but will then create a new BE there and install to that.

Arguments after the pool name are overlays, specifying what software to install, just like the regular install.

It will update grub for you, so that you have a grub on the pool that is compatible with the version of illumos you've just added. With -B, it will update the MBR as well.

It copies some files, the minimum that define the system's identity, from the existing bootable system into the new BE. This basically copies across user accounts (group, passwd, shadow files) and the system's ssh keys, but nothing else.

Any existing zfs file systems are untouched, and will be present in the new system. You'll have to import any additional zfs pools, though.

When you boot up after this, the grub menu will just contain the new BE you just created. However, any old boot environments are still present, so you can still see them, and manipulate them, using beadm. In particular, you can mount up and old BE (in case there are important files you need to get back), and activate an old BE so you can boot into the old system if so desired.

Amongst other things, you can use this as a recovery tool, when your existing system has a functioning root pool but won't boot.

I've also used this to "upgrade" older Tribblix systems. While there is an upgrade mechanism, this really only works (a) for very recent releases, and (b) to update one release at a time. With this new mechanism, I can simply stick a new copy of Milestone 17 on an old box, and enjoy the new version while having all my data intact.

Updating Tribblix for SPARC

Having just released an updated version of Tribblix for x86, a little commentary on the status of the SPARC version is probably in order.

After a rather extended delay, there is an updated version of Tribblix for SPARC available for download.

This is Milestone 16, so it's at the same level as the prior x86 release. More precisely, it's built from exactly the same illumos source as the x86 Milestone 16 release was. Yes, this means that it's a bit dated, but it's consistent, and there have been a number of breaking changes for SPARC builds introduced (and fixed) in the meantime.

I did want to get a release out of the door, before bumping the version to 0m17 ready for the next x86 build. Part of this was simply to ensure that I could actually still build a release - it gets tested on x86 all the time, but as it happens I had never built a SPARC release on my current infrastructure.

In terms of additional packages, the selection is still rather sparse. I haven't had the time to build up the full list. This isn't helped by the fact that my SPARC kit is rather slow, so building anything for SPARC simply takes longer. Much longer. (Not to mention the fact that they're noisy and power-hungry.)

It's not just time that's the problem. I've had some difficulty building certain packages. I'm not talking the likes of Go and Node, which I can simply ignore as they're not ported to SPARC at all, but some reasonably common packages would fail with obscure (and unexpected) build errors. If there's a problem with one component, that blocks anything dependent on it too.

Other than expanding the breadth of available packages, the key next steps are (i) to make Tribblix on SPARC self-hosting, like the x86 version has been for a very long time, and (ii) to try and keep the SPARC release more closely aligned so it doesn't drift away from the x86 release and require additional effort to bring back in sync.

Signing Packages in Tribblix

On any computer system you want to know exactly what software is installed and running.

Tribblix uses SVR4 packaging, so you can easily see what's installed. In addition, there are mechanisms - pkgchk - to compare what's on the disk with what the packaging system thinks should be there. But that's just a consistency check, it doesn't verify that the package installed is actually the one you wanted.

Tribblix has had simple integrity checking for a while. The catalog for a package repository includes both the expected size and the md5 checksum of a package. This is largely aimed at dealing with download errors - network drops, application errors, or errant intrusion detection systems mangling the data. In practice, because the downloaded packages are actually zip files, which have inbuilt consistency checking and the catalog at the end of the file, and because SVR4 packaging has its own consistency checks on package contents, the chances of a faulty download getting installed are remote, the checking is so that the layer above can make smart decisions in the case of failure.

But you want to be sure that, not only has the package you downloaded made it across the network intact, but that the source package is legitimate. So the packages are signed using gnupg, and will be verified upon download in upcoming releases. Initially this is just a warning check while the mechanisms get sorted out.

The actual signing and verification part is the easy bit, it's all the framework around it that takes the time to write and test.

One possibility would have been to sign the package catalogs, and use that to prove that the checksum is correct. That's not enough, for a couple of reasons. First, the catalog only includes current package versions, so there would be no way to verify prior versions. Second, there's no reason somebody (or me) couldn't take a subset of packages and create a new repo using them; the modified catalog couldn't be verified. In either case, you need to be able to verify individual packages. (But the package catalog should also be signed, of course.)

It turns out there's not much of a performance hit. Downloads are a little slower, because there's an extra request to get the detached signature, but it's a tiny change overall.

With this in place, you can be sure that whatever you install on Tribblix is legitimate. But all you're doing is verifying the packages at download time. This leaves open the problem of being able to go to a system and ask whether the installed files are legitimate. Yes, there's pkgchk, but there's no validated source of information for it to use as a reference - the contents file is updated with every packaging operation, so it clearly can't be signed by me each time.

This is likely to require the additional creation of a signed manifest for each package. This partially exists already, as the pkgmap fragments for each package are saved (in the global zone, anyway), and those could be signed (as they don't change) and used as the input to pkgchk. However, the checksums in the pkgmap and contents files aren't particularly strong (to put it mildly), so that file will need to be replaced by something with much stronger checksums.

Initial support for signed packages is available starting with the Tribblix Milestone 17 release. At this point, it will check the package signatures, but not act on them, enforcement will probably come in the next release when I can be reasonably sure that everything is actually working correctly.

Minimal illumos networking

Playing around with Minimal Viable Illumos, it becomes clear how complex a system illumos is. Normally, if you're running a full system, everything is taken care of for you. In particular, SMF starts all the right services for you. Looking through the method scripts, there's a lot of magic happening under the hood.

If you start with a bare install, booted to a shell, with no startup scripts having run, you can then bring up networking automatically.

The simplest approach would be to use ifconfig to set up a network interface. For example

ifconfig e1000g0 plumb 10.0.0.1 up

This won't work. It will plumb the interface, but not assign the address. To get the address assigned, you need to have ipmgmtd running.

env SMF_FMRI=svc/net/ip:d /lib/inet/ipmgmtd

Note that SMF_FMRI is set. This is because ipmgmtd expects to be run under SMF, and checks for SMF_FMRI being set as the determinant. It doesn't matter what the value is, it just needs to exist.

OK, so if you know your network device and your address, that's pretty much it.

If you use dladm to look for interfaces, then you'll see nothing. In order for dladm show-phys to enumerate the available interfaces, you need to start up dlmgmtd and give it a poke

env SMF_FMRI=svc/net/dl:d /sbin/dlmgmtd
dladm init-phys

Note that SMF_FMRI is set here, just like for ipmgmtd.

At this point, dladm show-phys will give you back the list of interfaces as normal.

You don't actually need to run dladm init-phys. If you know you have a given interface, plumbing it will poke enough of the machinery to make it show up in the list.

If you don't know your address, then you might think of using dhcp to assign it. This turns out to require a bit more work.

The first thing you need to do is bring up the loopback.

ifconfig lo0 plumb 127.0.0.1 up
ifconfig lo0 inet6 plumb ::1 up

This is required because dhcpagent wants to bind to localhost:4999, so you need the loopback set up for it to be able to do that. (You don't necessarily need the IPv6 version, that's for completeness.)

Then

ifconfig e1000g0 plumb
ifconfig e1000g0 auto-dhcp primary

Ought to work. Sometimes it does, sometimes it doesn't.

I sometimes found that I needed to

dladm init-secobj

in order to quieten errors that occasionally appear, and sometimes need to run

ifconfig -adD4 auto-revarp netmask + broadcast + up

although I can't even imagine what that would do to help, I presume that it kicks enough of the machinery to make sure everything is properly initialized.

Within the context of mvi, setting a static IP address means you need to craete a new image for each IP address, which isn't actually too bad (creating mvi images is really quick), but isn't sustainable in the large. DHCP would give it an address, but then you need to track what address a given instance has been allocated. Is there a way of doing static IP by poking the values from outside?

A tantalizing hint is present at the end of /lib/svc/method/net-physical, where it talks about boot properties being passed in. The implementation there appears to be aimed at Xen, I'll have to investigate if it's possible to script variables into the boot menu.

Running illumos in 48M of RAM

Whilst tweaking mvi recently, I went back and had another look at just how minimal an illumos install I could make.

And, given sufficiently aggressive use of the rm command, the answer appears to be that it's possible to boot illumos in 48 meg of RAM.

No, it's not April 1st. 48 meg of RAM is pretty low. It's been a long time since I've seen a system that small.

I've added some option scripts to the mvi repo. (The ones with -fix.sh as part of their names.) You don't have to run mvi to see what these do.

First, I start with mvix.sh, which is fairly minimal up front.

Then I go 32bit, which halves the size.

Then I apply the extreme option, which removes zfs (it's the single biggest driver left), along with a bunch of crypto and other unnecessary files. And I clean up lots of bits of grub that aren't needed.

I then abstracted out a nonet and nodisk option. The nonet script removes anything that looks like networking from the kernel, and the bits of userland that I add in order to be able to configure an interface. The nodisk script removes all the remaining storage device drivers (I only included the ones that you normally see when running under a hypervisor in the first place), along with the underlying scsi and related driver frameworks.

What you end up with is a 17M root file system, which compresses down to a 6.8M root archive, which gets packaged up in an 8.7M iso.

For those interested the iso is here. It should run in VirtualBox - in 32-bit mode, and you should be able to push the memory allocated by VirtualBox down to 48M. Scary.

Of course, it doesn't do much. It boots to a prompt, the only tools you have are ksh, ls, and du.

(Oh, and if you run du against the devices tree, it will panic.)

While doing this, I found that there are a lot of dependencies between modules in the illumos kernel. Not all of them are obvious, and trying to work out what's needed and what can be removed has involved large amounts of trial and error. That said, it takes less than 5 seconds to create an iso, and it takes longer for VirtualBox to start than for this iso to boot, so the cycle is pretty quick.

Almost there with Solarus

Every so often I'll have a go at getting some games running on Tribblix. It's not a gaming platform, but having the odd distraction can't hurt.

I accidentally stumbled across Solarus, and was immediately intrigued. Back in the day, I remember playing Zelda on the SNES. A few years ago it was released for the Game Boy Advance, and I actually went out and bought the console and the game. I haven't added any more games, nothing seemed compelling, although we had a stock of old Game Boy games (all mono) and those still worked, which is great.

So I thought I would try and build Solarus for Tribblix. Having had a quick look at the prerequisites, most of those would be useful anyway so the effort wouldn't be wasted in any event.

First up was SDL 2.0. I've had the older 1.2.15 version for a while, but hadn't had anything actually demand version 2 yet. That was easy enough, and because all the filenames are versioned, it can happily be installed alongside the older version.

While I was at it, I installed the extras SDL_image, SDL_net, SDL_ttf, smpeg, and SDL_mixer. The only build tweak needed was to supply LIBS="-lsocket -lnsl" to the configure script for smpeg and SDL_net. I needed to version the plaympeg binary installed by smpeg to avoid a conflict with the older version of smpeg, but that was about it.

Then came OpenAL, which turned out to be a bit tricky to find. Some of the obvious search results didn't seem appropriate. I have no idea whether the SourceForge site is the current one, but it appears to have the source code I needed.

Another thing that looks abandoned is modplug, where the Solarus folks have a github mirror of the copy that's right for them.

Next up, PhysicsFS. This isn't quite what you expect from the name, it's an abstraction layer that replaces traditional file system access.

On to Solarus itself. This uses cmake to build, and I ended up with the following incantation in a new empty build directory:

cmake ../solarus-1.4.5 \
 -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=NO \
 -DCMAKE_INSTALL_RPATH=/opt/solarus/lib \
 -DSOLARUS_USE_LUAJIT=OFF \
 -DCMAKE_INSTALL_PREFIX=/opt/solarus

Let's go through that. The CMAKE_INSTALL_PREFIX is fairly obvious - that's where I'm going to install it. And SOLARUS_USE_LUAJIT is necessary because I've got Lua, but have never had LuaJIT working successfully.

The two RPATH lines are necessary because of the strange way that cmake handles RPATH. Usually, it builds with RPATH set to the build location, then installs with RPATH stripped. This is plain stupid, but works when you simply dump everything into a single swamp. So you need to manually force it to put the correct RPATH into the binary (which is the sort of thing you would actually want a build system to get right on its own).

Unfortunately, it doesn't actually work properly. There are two problems which I haven't really had a chance to look at - the first is that it fails fatally with an Xlib error if I move the mouse (which is a little odd as it doesn't actually use the mouse); the second is that it runs an order of magnitude or two slower than useful, so I suspect a timing error.

Still, the build is pretty simple and it's so close to working that it would be nice to finish the job.

Tweaking MVI

A few months ago I first talked about minimal viable illumos, an attempt to construct a rather more minimalist bootable copy of illumos than the gigabyte-size image that are becoming the norm.

I've made a couple of changes recently, which are present in the mvi repository.

The first is to make it easier for people who aren't me (and myself when I'm not on my primary build machine) to actually use mvi. The original version had the locations of the packages hardcoded to the values on my build machine. Now I've abstracted out package installation, which eliminates quite a lot of code duplication. And then I added an alternative package installation script which uses zap to retrieve and install packages from the Tribblix repo, just like the regular system does. So you can much more easily run mvi from a vanilla Tribblix system.

What I would like to add is a script that can run on pretty much any (illumos) system. This isn't too hard, but would involve copying most of the functionality of zap into the install script. I'm holding off for a short while, hoping that a better mechanism presents itself. (By better, what I mean is that I've actually got a number of image creation utilities, and it would be nice to rationalise them rather than keep creating new ones.)

The second tweak was to improve the way that the size of the root archive is calculated, to give better defaults and adapt to variations more intelligently.

There are two slightly different mechanisms used to create the image. In mvi.sh, I install packages, and then delete what I'm sure I don't need; with mvix.sh I install packages and then only take the files I do need. The difference in size is considerable - for the basic installation mvi.sh is 127M and mvix.sh is 57M.

Rather than a common base image size of 192M, I've set mvi.sh to 160M and mvix.sh to 96M. These sizes give a reasonable amount of free space - enough that adding the odd package doesn't require the sizes to be adjusted.

I then have standard scripts to construct 32-bit and 64-bit images. A little bit of experimentation indicates that the 32-bit image ends up being half the size of the base, whereas the 64-bit image comes in at two thirds. (The difference is that in the 32-bit image, you can simply remove all 64-bit files. For a 64-bit kernel, you still need both 32-bit and 64-bit userland.) So I've got those scripts to simply scale the image size, rather than try and pick a new number out of the air.

I also have a sample script to install Node.js. This again modifies the image size, just adding the extra space that Node needs. I've had to calculate this more accurately, as reducing the size of the base archive gave me less margin for error.

(As an aside, adding applications doesn't really work well in general with mvix.sh, as it doesn't know what dependencies applications might need - it only installs the bare minimum the OS needs to boot. Fortunately Node is fairly self-contained, but other applications are much less so.)

Software selection - choice or constraint?

In Tribblix, software is preferentially managed using overlays rather than packages.

Overlays comprise a group of packages bundled together to supply a given software need - the question should be "what do you want to do?", and packages (and packaging) are merely an implementation artifact in providing the answer to that question.

Part of the idea was that, not only would the overlays match a user's mental model of what they want to install, but that there would be many fewer overlays than packages, and so it's much easier for the human brain to track that smaller number of items.

Now, it's true that there are fewer overlays than available packages. As of right now, there are 91 overlays and 1237 packages available for Tribblix. So that's better than an order of magnitude reduction in the number of things, and an enormous reduction in the possible combinations of items. However, it's no longer small in the absolute sense.

(Ideally, small for me means that it can be all seen on screen at once. If you have to page the screen to see all the information, your brain is also paging stuff in and out.)

So I've been toying with the idea of defining a more constrained set of overlays. Maybe along the lines of a desktop and server split, with small, large, and developer instances of each.

This would certainly help dramatically in that it would lead to significant simplification. However, after trying this out I'm unconvinced. The key point is that genuine needs are rather more complicated than can be addressed by half a dozen neat pigeonholes. (Past experience with the install metaclusters in Solaris 10 was also that they were essentially of no use to any particular user, they always needed to be customised.)

By removing choice, you're seriously constraining what users can do with the system. Worse, by crippling overlays you force users back into managing packages which is one of the things I was trying to avoid in the first place.

So, I'm abandoning the idea of removing choice, and the number of overlays is going to increase as more applications are added. Which means that I'm going to have to think a lot harder about the UX aspect of overlay management.

Load balancers - improving site reliability

You want your online service to be reliable - a service that isn't working is of no use to your users or customers. Yet, the components that you're using - hardware and software - are themselves unreliable. How do you arrange things so that the overall service is more releiable than the individual components it's made up of?

This is logically 2 distinct problems. First, given a set of N systems able to provide a service, how do you maintain service if one or more of those fail? Second, given a single service, how do you make sure it's always available?

The usual solution here involves some form of load balancer. A software or hardware device that takes incoming requests and chooses a system to handle the request, only considering as candidates those systems that are actually working.

(The distinction between hardware and software here is more between prepackaged appliances and DIY. The point about the "hardware" solutions is that you buy a thing, and treat it as a black box with little access to its internals.)

For hardware appliances, most people have heard of F5. Other competitors in this space are Kemp and A10. All are relatively (sometimes eye-wateringly) expensive. Not necessarily bad value, mind, depending on your needs. At the more affordable end of the spectrum sits loadbalancer.org.

Evaluating these recently, one thing I've noticed is that there's a general tendency to move upmarket. They're no longer load balancers, there's a new term here - ADC, or Application Delivery Controllers. They may do SSL termination and add functionality such as simple firewall functionality, Web Applications Firewalls, or Intrusion Detection and Threat Management. While this is clearly to add differentiation and keep ahead of the cannibalization of the market, many of the additional functionality simply isn't relevant for me.

Then there is a whole range of open source software solutions that do load balancing. Often these are also reverse proxies.

HAProxy is well known, and very powerful and flexible. It's not just web, it's very good at handling generic TCP. Packed with features, my only criticism is that configuration is rather monolithic.

You might think of Nginx as a web server, but it's also an excellent reverse proxy and load balancer. It doesn't quite have the range of functionality of HAProxy, but most people don't need anything that powerful anyway. One thing I like about Nginx is directory-based configuration - drop a configuration fragment into a directory, signal nginx, and you're off. If you're managing a lot of sites behind it, such a configuration mode is a godsend.

There's an interesting approach used in SNI Proxy. It assumes an incoming HTTPS connection has an SNI header on it, picks that out, and uses that to decide where to forward the TCP session. By using SNI, you don't have to put certificates on the proxy host, or get it to decrypt (and possibly re-encrypt) anything.

Offering simpler configuration are Pound and Pen. Neither are very keen on virtual hosting configurations. If all your backend servers are the same and you do all the virtual hosting there, then that's fine, but if you need to route to different sets of back end servers depending on the incoming request, they aren't a good choice.

For more dynamic configurations, there's vulcand, where you put all you configuration into Etcd. If you're into microservices and containers the it's definitely worth a look.

All the above load balancers assume that they're relatively reliable (or are relatively stable) compared to the back end services they're proxying. So they give you protection against application or hardware failure, and allow you to manage replacement, upgrades, and general deployment tasks without affecting users. The operational convenience of being able to manage an application independent of it's user-facing endpoint can be a huge win.

To achieve availability of the service customers connect to needs a little extra work. What's to stop it failing?

In terms of the application failing, that should be less of a concern. Compared to a fully-fledged business application, the proxy is a fairly simple, usually stateless, so has less to fail and can be automatically restarted pretty quickly if and when it fails.

But what if the underlying system goes away? That's what you need to protect against. And what you're really doing here is trying to ensure that the IP address associated with that service is always live. If it goes away, move it someplace else and carry on.

Ignoring routing tricks and things like VRRP and anycast, some solutions here are:

UCARP is a userland implementation of the Common Address Redundancy Protocol (CARP). Basically, hosts in a group monitor each other. If the host holding the address disappears, another host in the group will bring up a virtual interface with the required IP address. The bringup/teardown is delegated to scripts, allowing you to perform any other steps you might need to as part of the failover.

Wackamole, which uses the Spread toolkit, is another implementation of the same idea. It's getting a bit old now and hasn't seen any work for a while.

A newer variation that might be seen as the logical successor to wackamole is vippy, which is built on Node. The downside here is that Node is a moving target, so vippy won't build as is on current versions of Node, and I had trouble building it at all.

As you can see, this is a pretty large subject, and I've probably only scratched the surface. If there are things I've missed, especially if they're relevant to illumos, let me know.

Supermicro - illumos compatible server configuration

We were recently in the market for some replacement servers. We run OmniOS in production, so something compatible with illumos is essential.

This is a little more tricky than it appears. Some of the things to be aware of are specific to illumos, while some are more generic. But, after spending some time reading an awful lot of spec sheets and with the help of the OmniOS mailing list, we got to a final spec that ought to be pretty good.

I'm going to talk about a subset of Supermicro systems here. While other vendors exist, it can be harder to put together a working spec.

To start with, Supermicro have a bewildering list of parts. But we started out looking at 2U storage servers, with plenty of 2.5" disk slots in the front for future expansion.

Why 2.5"? Well, it allows you twice as many slots (24 as opposed to 12) so you have more flexibility. Also, the industry is starting to move away from 3.5" drives to 2.5", but that's a slow process. More to the point, most SSDs come in the 2.5" form factor, and I was planning to go to SSDs by default. (It's a good thing now, I'm thinking that choosing spinning rust now will look a bit daft in a few years time.) If you want bulk on the cheap, then something with 3.5" slots that you can put 4TB or larger SAS HDDs in might be better, or something like the DataON 1660 JBOD.

We're also looking at current motherboards. That means the X10 at this point. The older X9 are often seen in Nexenta configurations, and those will work too. But we're planning for the long haul, so want things to not turn into antiques for as long as possible.

So there is a choice of chassis. These are:

• 216 - 24 2.5" drive slots
• 213 - 16 2.5" drive slots
• 826 - 12 3.5" drive slots
• 825 - 8 3.5" drive slots

The ones with smaller numbers of drives have space for something like a CD.

The next thing that's important, especially for illumos and ZFS, is whether it's got a direct attach backplane or whether it puts the disks behind an expander. Looking at the 216 chassis, you can have:

• SC216BAC-R920LPB - direct attach
• SC216BE1C-R920LPB - single expander
• SC216BE1C-R920LPB - dual expander

So you can see that something with A or AC is direct attach, E1C is a single expander, E2C is a dual expander. (With a dual expander setup you can use multipathing.)

Generally, for ZFS, you don't want expanders. Especially so if you have SATA drives - and many affordable SSDs are SATA. Vendors and salespeople will tell you that expanders never cause any problems, but most illumos folk appear allergic to the mere mention of them.

(I think this is one reason Nexenta-compatible configurations, and some of the preconfigured SuperServer setups, look expensive at first sight. They often have expanders, use exclusively SAS drives as a consequence, and SAS SSDs are expensive.)

So, we want the SC216BAC-R920LPB chassis. To connect up 24 drives, you'll need HBAs. We're using ZFS, so don't need (or want) any sort of hardware raid, just direct connectivity. So you're looking at the LSI 9300-8i HBA, which has 8 internal ports, and you're going to need 3 of them to connect all 24 drives.

For the motherboard, the X10 has a range of models. At this point, select how many and what speed network interfaces you want.

• X10DRi - dual 1G
• X10DRi-T - dual 10G
• X10DRi-LN4+ - quad 1G
• X10DRi-T4+ - quad 10G

The dual network boards have 16 DIMM slots, the quad network boards have 24 DIMM slots. The network cards are Intel i350 (1Gbe) or X540 (10Gbe) which are both supported by illumos.

The 2U chassis can support a little 2-drive disk bay at the back of the machine, you can put a pair of boot drives in here and wire them up directly to the SATA ports on the motherboard, giving you an extra 2 free drive slots in the front. Note, though, that this is only possible with the dual network boards, the quad network boards take up too much room in the chassis. (It's not so much the extra network ports as such, but the extra DIMM slots.)

Another little quirk is that as far as I can tell the quad 1G board has fewer USB ports, and they're all USB3. You need USB2 for illumos, and I'm not sure if you can demote those ports down to USB2 or not.

So, if you want 4 network ports (to provide, for example, a public LACP pair and a private LACP pair), you want the X10DRi-T4+.

Any E5-2600 v3 CPUs will be fine. We're not CPU bound so just went for cheaper lower-power chips, but that's a workload thing. One thing to be aware of is that you do need to populate both sockets - if you just have one then you lose half of the DIMM slots (which is fairly obvious) and most of the PCI slots (which isn't - look at the documentation carefully if you're thinking of doing this, but better to get a single-socket motherboard in the first place).

As for drives, we went for a pair of small Intel S3510 for the boot drives, those will be mirrored using ZFS. For data, larger Intel S3610, as they support more drive writes - analysis of our I/O usage indicated that we are worryingly close to the DWPD (Drive Writes Per Day) of the S3510, so the S3610 was a safer choice, and isn't that much more expensive.

Hopefully I'll be able to tell you how well we get on once they're delivered and installed.

Moving goalposts with openssl

The most recent openssl release fixed a number of security issues.

In order to mitigate against DROWN, SSLv2 was disabled by default. OK, that's a reasonable thing to do. The mere presence of the code is harmful, and improving security is a good thing. Right?

Well, maybe. Unfortunately, this breaks binary compatibility by default. Suddenly, a number of functions that used to be present in libssl.so have disappeared. In particular:

SSLv2_client_method
SSLv2_method
SSLv2_server_method

and the problem is that if you have an application that references those symbols, the linker can't find them, and your application won't even run.

This hit pkg(5) on OmniOS, which is pretty nasty.

So I had a look around on Tribblix to see what would break. It's largely what you would expect:

• curl, specifically libcurl
• wget
• neon
• the python ssl module
• the ruby ssl module
• apache
• mysql

 And, of course, by the magic of dependency hell that spreads to affect a large number of applications.

In many cases the SSLv2 code is only present if it detects the corresponding calls being present in libssl. Which leads you into a chicken and egg situation - you have to install the newer openssl, thus breaking your system, in order to rebuild the applications that are broken.

And even if a distributor rebuilds what the distro ships, there's still any 3rd-party or locally built binaries which could be broken.

For Tribblix, I'm rebuilding what I can in any case, explicitly disabling SSLv2 (because the automatic detection is wrong). I'll temporarily ship openssl with SSLv2 enabled, until I've finally nailed everything.

But this is a game changer in terms of expectations. The argument before was that you should link dynamically in order to take advantage of updates to critical libraries rather than have to rebuild everything individually. Now, you have to assume that any security fix to a library you use could break compatibility.

For critical applications, another reason to build your own stack.

Updating gcc for Tribblix

One of the items on the Tribblix Roadmap was improving the system compiler.

I'm currently shipping a copy of gcc 4.8.3. When Tribblix was originally created, I inherited the old gcc 3.4.3 build from OpenIndiana. It took a modest amount of effort to get a copy of gcc v4 that worked (at one point I bailed out completely and rolled back to gcc3), originally I had 4.7.2 but settled on 4.8.3. And for many purposes that's worked out quite well.

It's starting to show problems, though. In no particular order, some are:

• It's not fully compatible with modern C++ code. In particular, there are instances of modern code that simply keel over and complain that 'to_string isn't a member of std', and various other functions that ought to be supported.
• There are places that have the path to the gcc install encoded. Given that each compiler version is installed to a different path, publishing packages that have a hardcoded dependency on the install location of this particular compiler build doesn't work so well. In many cases I can fix up any .la files that have errant paths in them, and do so as part of my build process.
• For (bad) reasons associated with the way I transitioned from gcc3 to gcc4, the current gcc build has libssp, libgomp, and libquadmath disabled. I've recently come across builds that need the first two at least.
• I've decided that it was a mistake to put the full version in the install prefix. It would have been better to use 4.8 rather than 4.8.3. (Yes, I know that gcc has an internal directory hierarchy that uses the full version number.)

Now, gcc has moved on in the meantime. I can see some basic options:

• Update to gcc 4.8.5, fixing the build process to add the missing features and fix up any errant paths
• Update to gcc 4.9.3, fixing the build process to add the missing features and fix up any errant paths
• Update to whatever the latest gcc5 is.
• Wait for gcc6 to come out and switch to that

 I've been doing a little experimentation.

While I would like to keep as current as possible, waiting for gcc doesn't solve the problems I have now, so maybe not.

I've tried building gcc5 and had some components fail on me. With 5.1.0, the fortran build didn't work; with 5.3.0 it was gccgo that failed. Not having fortran is probably a showstopper. Losing gccgo less so, as we have golang proper. On x64, at any rate.

Another issue arguing against gcc5 is the C++ ABI changes. Not that they're a problem of themselves, but that it makes rolling back a real pain. I've already had to do a compiler rollback once, and it was painful. And given that I've seen problems with gcc5 builds (that I haven't been able to satisfactorily resolve) then I'm wary of other problems cropping up.

Building 4.9.3 has been uneventful so far. I've enabled (or, rather, not disabled) libssp, gomp, and quadmath. I added obj-c++ as it didn't seem to hurt. I've disabled libitm. Generally, this brings me much closer to the OpenIndiana and OmniOS build settings.

Testing this compiler, the 'to_string isn't a member of std' family of errors are gone, which opens up more code (LibreOffice 5 in particular). And a handful of test builds seem to work just fine.

I have one issue to resolve before I can get this rolled out. It looks like the way that libgcc is linked has changed (looking at the dumpspecs output I can see it's changed) so that every binary ends up depending on libgcc_s. That's something I really don't want. And manually forcing -static-libgcc on every build seems wrong (and besides, there are builds where it's the wrong thing to do).

Tribblix Roadmap

It seems like only yesterday, but I put together a Tribblix scorecard about a year ago.

Tribblix keeps improving, of course. There have been a number of enhancements to the available software, and this continues with a steady stream of package updates.

Actual releases have been relatively thin on the ground. I managed to get Milestone 15 out in April, and Milestone 16 in September. It's about time for another one, looking at that cadence, and I don't really want to go much slower.

What constitutes a release anyway? For Tribblix, there are 2 things that can only happen at a release.

The first is updating the underlying illumos build - this is slightly artificial, but you can't really do on the fly updates of illumos, whereas most of the applications can be updated just fine. So that leads to a distinction between updating packages and upgrading the release.

The second is any structural change to the distro itself, including the way it's packaged and that packages are managed (so if zap changes that's a new release).

Note that adding packages, or replacing them with new versions, doesn't generally involve a release. Even if it's a fairly major package, it will simply be available when it's built.

Looking forward, then, what do I have in the pipeline in terms of releases and changes aligned with them?

Overall, the general stability of the Tribblix pipeline, and the fact that I've had managed to achieve many of the objectives I set myself initially, means that I'm reasonably close to calling it ready, and putting an actual 1.0 release together. What needs doing to make that a reality?

• Fully fledged and aligned SPARC support would be good. Not that it's necessary to align the release dates or support every piece of software, but what I need to be sure of is that a SPARC release isn't going to need breaking changes to the rest of Tribblix.
• Fixing the compiler baseline. I currently have a slightly tweaked gcc 4.8.3. I need to settle on a version (possibly a newer version) and untweak the configuration.
• Making sure that upgrades are solid. At the present time, they mostly work, but are a little clumsy.

There's a lot of devil in those details, but there's nothing massive there. And the result will be an illumos distro with regular updates and lots of the functionailty I expect.

Beyond that, I'm already making plans for Tribblix2. This is the opportunity for me to play with different ideas for how illumos could be used. And this ability to experiment with new ideas is one of the reasons I built my own distribution in the first place.

Think of Tribblix2 as more of a series of concepts rather than a single release vehicle. I can try things like removing 32-bit (and do it wholesale) to see what the benefits would be (and identify any drawbacks along the way). I can see how much of the legacy we currently ship in /usr can be ditched entirely. I can look further at projects like minimal viable illumos and minimal memory systems. It would be fun to see what happens if you apply some of the ideas behind Docker and Unikernels to illumos. Of course, successful experiments could lead to improvements trickling down to regular Tribblix.

OK, so it's not really a roadmap, but it should give an idea of where I'm heading.

Do suppliers want to go out of business?

The onrushing cloud behemoth seems destined to sweep many legacy IT suppliers aside, at least if you believe the pundits.

However, it's not just down to the (often imagined) inherent superiority of cloud computing. In many cases, IT suppliers have only themselves to blame.

Now, I have no business training, but even I can understand that making it easy for customers to buy your stuff is probably a good idea.

It's clear, though, that many companies obviously don't want to sell me stuff.

Starting with a website. That's what you do now. In the old days you might have gone to a trade show or looked in a magazine, and ended up with a brochure. No more - if I want product details, I'll look at your website. If your product details are sketchy, non-existent, out of date, inconsistent, vaporware, and generally devoid of technical content, then I'll look elsewhere. If you want me to register to view your technical documentation - even something as simple as the prerequisite system requirements - then I'll likely look elsewhere. If it's impossible to even guess what ballpark your prices are in, then I'll assume I can't afford it.

Then, please tell me how to buy the stuff. If you have resellers, have a list. Make sure that your resellers have actually heard of you. And keep that list up to date and accurate. If you sell direct, say so.

And last, but certainly not least, if a potential customer emails you - either direct or via that stupid form on your website that has a little postage stamp sized box to put the query in - showing an interest in buying your stuff, actually show some interest in selling your product. Answer the email, at the very least. Do it promptly. Do it accurately.

Looking for products recently, many supplier websites simply fail, completely. In some cases, there's a confusingly jumbled list of products that you have to visit individually to work out if they meet your requirements. Other times, they've decided to randomly segment their offerings into neat customer buckets, so I have to trawl through all their sections to find the product I want. One site had a useful handy product chart - with no hyperlinks, and listing a number of models that didn't exist as far as I can tell. Being forced to use a search engine to navigate a supplier's site is not without its problems - often you land on old pages with no indication whether the product is even current.

I've been trying to get a number of quotes recently. Maybe a third of suppliers and/or manufacturers are pretty good (thank you to all of you, by the way). Another third have such a dismal web presence that they're clearly not fit for the 21st century, so why would I even think of using them? The remaining third have simply ignored all attempts to contact them. Given how difficult it is for IT companies to survive in the current rapidly changing and highly competitive landscape, it staggers me what a poor job so many companies are doing.