Remnants of closed code in illumos

One of the annoying issues with illumos has been the presence of a body of closed binaries - things that, for some reason or other, were never able to be open sourced as part of OpenSolaris.

Generally the illumos project has had some success in replacing the closed pieces, but what's left isn't entirely zero.It took me a little while to work out what's still left, but as of today the list is:

etc/security/tsol/label_encodings.gfi.single
etc/security/tsol/label_encodings.example
etc/security/tsol/label_encodings.gfi.multi
etc/security/tsol/label_encodings
etc/security/tsol/label_encodings.multi
etc/security/tsol/label_encodings.single
usr/sbin/chk_encodings
usr/xpg4/bin/more
usr/lib/raidcfg/mpt.so.1
usr/lib/raidcfg/amd64/mpt.so.1
usr/lib/iconv/646da.8859.t
usr/lib/iconv/8859.646it.t
usr/lib/iconv/8859.646es.t
usr/lib/iconv/8859.646fr.t
usr/lib/iconv/646en.8859.t
usr/lib/iconv/646de.8859.t
usr/lib/iconv/646it.8859.t
usr/lib/iconv/8859.646en.t
usr/lib/iconv/8859.646de.t
usr/lib/iconv/iconv_data
usr/lib/iconv/646fr.8859.t
usr/lib/iconv/8859.646da.t
usr/lib/iconv/646sv.8859.t
usr/lib/iconv/8859.646.t
usr/lib/iconv/646es.8859.t
usr/lib/iconv/8859.646sv.t
usr/lib/fwflash/verify/ses-SUN.so
usr/lib/fwflash/verify/sgen-SUN.so
usr/lib/fwflash/verify/sgen-LSILOGIC.so
usr/lib/fwflash/verify/ses-LSILOGIC.so
usr/lib/labeld
usr/lib/locale/POSIX
usr/lib/inet/certlocal
usr/lib/inet/certrldb
usr/lib/inet/amd64/in.iked
usr/lib/inet/certdb
usr/lib/mdb/kvm/amd64/mpt.so
usr/lib/libike.so.1
usr/lib/amd64/libike.so.1
usr/bin/pax
platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.46
platform/i86pc/kernel/cpu/amd64/cpu_ms.GenuineIntel.6.47
lib/svc/manifest/network/ipsec/ike.xml
kernel/kmdb/amd64/mpt
kernel/misc/scsi_vhci/amd64/scsi_vhci_f_asym_lsi
kernel/misc/scsi_vhci/amd64/scsi_vhci_f_asym_emc
kernel/misc/scsi_vhci/amd64/scsi_vhci_f_sym_emc
kernel/strmod/amd64/sdpib
kernel/drv/amd64/adpu320
kernel/drv/amd64/atiatom
kernel/drv/amd64/usbser_edge
kernel/drv/amd64/sdpib
kernel/drv/amd64/bcm_sata
kernel/drv/amd64/glm
kernel/drv/amd64/intel_nhmex
kernel/drv/amd64/lsimega
kernel/drv/amd64/marvell88sx
kernel/drv/amd64/ixgb
kernel/drv/amd64/acpi_toshiba
kernel/drv/amd64/mpt
kernel/drv/adpu320.conf
kernel/drv/usbser_edge.conf
kernel/drv/mpt.conf
kernel/drv/intel_nhmex.conf
kernel/drv/sdpib.conf
kernel/drv/lsimega.conf
kernel/drv/glm.conf

Actually, this isn't much. In terms of categories:

Trusted, which includes those label_encodings, and labeld. Seriously, nobody can realistically run trusted on illumos (I have, it's ... interesting). So these don't really matter.

The iconv files actually go with the closed iconv binary, which we replaced ages ago, and our copy doesn't and can't use those files. We should simply drop those (they will be removed in Tribblix next time around).

There's a set of files connected to IKE and IPSec. We should replace those, although I suspect that modern alternatives for remote access will start to obsolete all this over time.

The scsi_vhci files are to get multipathing correctly set up on some legacy SAN systems. If you have to use such a SAN, then you need them. If not, then you're in the clear.

There are a number of drivers. These are mostly somewhat aged. The sdp stuff is being removed anyway as part of IPD29, so that'll soon be gone. Chances are that very few people will need most of these drivers, although mpt was fairly widely used (there was an open mpt replacement in the works). Eventually the need for the drivers will dwindle to zero as systems with them in no longer exist (and, by the same token, we wouldn't need them for something like an aarch64 port).

Which just leaves 2 commands.

Realistically, the XPG4 more could be replaced by less. The standard was based on the behaviour of less, after all. I'm tempted to simply delete /usr/xpg4/bin/more and make it a link to less and have done with it.

As for pax, it's required by POSIX, but to be honest I've never used it, haven't seen anywhere that uses it, and read support is already present in things like libarchive and gtar. The heirloom pax is probably more than good enough.

In summary, illumos isn't quite fully open source, but it's pretty close and for almost all cases we could put together a fully functional open subset that'll work just fine.

Static Site Generators

The current Tribblix website is a bit of a hack. Technically it's using a static site generator - a simple home-grown script that constructs pages from a bit of content and boilerplate - but I wanted to be able to go a bit further.

I looked at a few options - and there are really a huge number of them - such as Hugo and Zola. (Both are packaged for Tribblix now, by the way.)

In the end I settled on nanoc. That's packaged too (and I finally got around to having a very simple - rather naive - way of packaging gems).

Why nanoc, though? In this case it was really because it could take the html page fragments I already had and create the site from those, and after tweaking it slightly I end up with exactly the same html output as before.

Other options might be better if I was starting from scratch, but it would have been much harder to retain the fidelity of the existing site.

One advantage of the new system is that I can put the site under proper source control, so the repo is here.

There's still a lot of work to be done on filling out the content, but it should be easier to evolve the Tribblix website in future.

Zones, way back when

The original big ticket feature in Solaris 10 was Zones, a simple virtualization technology that allowed a set of processes to be put aside in a separate namespace and be under the illusion that this was a separate computer system, all under a single shared kernel.

As a result of this sleight of hand, you could connect to a zone using ssh (or, remember this was way back, telnet or rsh), and from the application level you really were in a separate system - with your own file system and network namespaces. It was like magic.

Of the features in Solaris 10, Zones and DTrace were present early in the beta cycle, while SMF just made it into the last couple of beta builds, and ZFS wasn't actually available to customers until well after the first Solaris 10 release.

I ended up using zones in production quite accidentally. In the Solaris 10 Platinum Beta, we were testing the new features, just giving them a good beating, when one of our webservers (it was something like a Netra X1) died. Sure, we could have got it repaired, or reconfigured another server. But as an experiment, I simply fired up a zone on one of my beta systems, gave it the IP address of the failed server, installed apache, copied over the website, and we were back in service in about 5 minutes.

The Zones framework turns out to be incredibly flexible and powerful. I suspect most don't realize just what it's actually capable of, as Sun only gave you a canned product in two variations - whole-root and sparse-root zones. Later you saw glimpses of the power available with the first incarnation of LX zones (or SCLA - Solaris Containers for Linux Applications) and then the Solaris 8 and Solaris 9 containers, which allowed a different set of applications to run inside a zone.

Things actually became more limited in OpenSolaris and its derivatives such as Solaris 11; not only was LX removed, but so were sparse-root zones, and the diversity of potential zone types dwindled.

In illumos, some of the distributions have pushed Zones a bit further. Tribblix brought back sparse root zones, and introduced the alien brand - essentially a way to run any illumos OS or application in a zone. OmniOS has brought back LX, and it's reasonably current (in terms of keeping up with changes in the Linux world). SmartOS ran KVM in Zones, allowing double-hulled virtualization. And we now have bhyve as a fully supported offering for any illumos distribution, usually
embedded in a Zone.

Using a sparse-root zone is incredibly efficient. By sharing the main operating system files (mostly /lib and /usr, but can be others) you can save huge amounts of disk space - you only have to have one copy so that's a saving of anything for a couple of hundred megabytes to a couple of gigabytes of storage per zone. It gets better, because the read-only segments of any binaries and shared libraries are shared between zones, which dramatically reduces the additional memory footprint of each zone. Further on from that, because Solaris has this trick whereby any shared object used more that 8 times (or something like that) is kept resident in memory, all the common applications are always in memory and start incredibly quickly.

One of the things I did was use sparse-root zones and shared filesystems for a development -> test -> production setup. Basically, you create 3 zones, sparse-root ensures they're identical, and 3 filesystems - one each for development, test, and production. You share the development filesystem read-only into the test zone, so deployment from development to test is a straight copy. Likewise test to production.

One of the weaknesses of the way that zones were managed (distinct from the underlying technology framework) is that it was based around packaging. In Solaris 10, packaging and packages knew about zones, and the details about what files and packages ended up in a zone was embedded in the package metadata. Not only is this complex, it's also very rigid - you can't evolve the system without changing the packaging system and modifying all the packages. Sadly, IPS carried forward the same mistake. (In Tribblix, packaging knows nothing about zones whatsoever, but my zones understand packaging and can do the right thing with it - not only with much more flexibility but many times quicker.)

Later on in the Solaris 10 timeframe we got ZFS, which allowed you to do interesting things around sharing data and quickly creating copies of data for zones, allowing you to extend the virtual capabilities of zones from cpu and memory to storage. And the key missing piece, virtualized networking, never made it to Solaris 10 at all, but had to wait for crossbow to arrive in OpenSolaris.

Maintaining old software with no sign of retirement

There's a lot of really old software out there. Some of it has simply been abandoned; others have been replaced by new versions. But old software never really goes away, and we end up maintaining it.

This is especially tricky when old software depends on other old software, and we have to support the entire dependency tree.

There's always python2 and python 3. Some old software may never be fixed; some current software has consciously decided to stick to python 2. Distributions will be shipping python 2 for a long time yet.

Then there's PCRE and PCRE2. Some things have been updated; others haven't. Generally for this I'll keep updating, and eventually upstream might get around to migrating. But again I'll have to ship both for a while.

And then there's gtk2 and gtk3. (I find it ironic that the gimp itself is still using gtk2.) There's no end in sight of the need to ship both.

Some libraries have been deprecated entirely. the old libXp (the X printing library) is long gone. There were a couple of things built against it in Tribblix. I've just rebuilt chimera (a really old Xaw web browser if your memory doesn't go that far back) which was one consumer and now isn't; the other one was Motif (there's a convenient build flag --disable-printing to disable libXp support, which entertainingly breaks the build someplace else which I ended up having to fix).

Another example, libpng has gone through several different revisions. Each slightly incompatible, and you have to be sure to run with the same version you built against. At least you can ship all the different versions, as they have the version in the names. Mind you, linking against 2 different versions of libpng at the same time (for example, if a dependency pulls in a different version of libpng) is a bad thing, so I did have to rebuild a number of applications to avoid that. I ship the old libpng versions in a separate compat package, I think chimera was the only consumer, but I updated that to use a more current libpng.

A slightly different problem is the use of newer toolchains. Compilers are getting stricter over time, so old unmaintained software needs patches to even compile.

Don't even get me started on openssl.

Upgrading MATE on Tribblix

I spent a little time yesterday updating MATE on Tribblix, to version 1.26.

This was supposed to be part of the "next" release, but we had to make an out of sequence release for an illumos security issue, so everything gets pushed back a bit.

Updating MATE is actually fairly easy, though. The components in MATE are largely decoupled, so can be updated independently of each other. (And there isn't really a MATE framework everything has to subscribe to, so the applications can be used outside MATE without any issues.)

There's a bit of tidying up and polish that helps. For example, I delete static archives and the harmful libtool archive files. Not only does this save space, it helps maintainability down the line.

Builds have a habit of picking up dependencies from the build system. Sometimes you can control this with judicious --enable-foo or --disable-foo flags, sometime you just have to make sure that the package you don't want pulled in isn't installed. The reverse is true - if you want a feature to be enabled, you have to make sure the dependencies are installed first and the feature will usually get enabled automatically.

That's not always true. For example, you have to explicitly tell it you have OSS for audio, it doesn't work this out on its own.

I took the opportunity to make everything 64-bit. Ultimately I want to get to 64-bit only. This involves a bit of working backwards - you have to make all consumers of a library 64-bit only first.

A couple of components are held downrev. The calculator now wants to pull in mpc and mpfr, which I don't package. (They're used by gcc, but I drop a copy of mpc and mpfr into the build for gcc to find rather than packaging them separately the way that most of the other illumos distributions do.) And pluma wants gtksourceview-4 which I don't have yet. This is related to the lack of tight coupling I mentioned earlier - there really isn't any problem having the different pieces that make up MATE at different revisions.)

You stumble across bugs along the way. For example, mate-control-center actually needs GLib 2.66 or later, which I don't have yet (there's another whole set of issues behind that), but it doesn't actually check for the right version. Fortunately the requirement is fairly localized and easy to patch out.

That done, on to another set of updates...

SPARC Tribblix m26 - what's in a number?

I've just released Tribblix m26 for SPARC.

The release history on SPARC looks a little odd - m20, m20.6, m22, m25.1, and now m26. Do these release versions mean anything?

Up to and including m25.1, the illumos commit that the SPARC version was built from matched the corresponding x86 release. This is one reason there might be a gap in the release train - that commit might not build or work on SPARC.

As of m26, the version numbers start to diverge between SPARC and x86. In terms of illumos-gate, this release is closer to m25.2, but the added packages are generally fairly current, closer to m29. So it's a bit of a hybrid.

But the real reason this is a full release rather than an m25 update is to establish a new baseline, which allows me to establish compatibility guarantees and roll over versions of key components, in this case it allows me to upgrade perl.

In the future, the x86 and SPARC releases are likely to diverge further. Clearly SPARC can't track the x86 releases perfectly, as SPARC support is being removed from the mainline source following IPD 19, and many of the recent changes in illumos simply aren't relevant to SPARC anyway. So future SPARC releases are likely to simply increment independently.

How I build the Tribblix AMIs

I run Tribblix on AWS, and make some AMIs available. They're only available in London (eu-west-2) by default, because that's the only place where I use them, and it costs money to have them available in other regions. If you want to run them elsewhere, you can copy the AMI.

It's not actually that difficult to create the AMIs, once you've got the hang of it. Certainly some of the instructions you might find can seem a little daunting. So here's how I do it. Some of the details here are very specific to my own workflow, but the overall principles are fairly generic. The same method would work for any of the illumos distributions, and you could customize the install however you wish.

The procedure below assumes you're running Tribblix m29 and have bhyve installed.

The general process is to boot and install an instance into bhyve, then boot that and clean it up, save that disk as an image, upload to S3, and register an AMI from that image.

You need to use the minimal ISO (I actually use a custom, even more minimal ISO, but that's just a convenience for myself). Just launch that as root:

zap create-zone -t bhyve -z bhyve1 \
-x 192.168.0.236  \
-I /var/tmp/tribblix-0m29-minimal.iso \
-V 8G

Note that this creates an 8G zvol, which is the starting size of the AMI.

Then run socat as root to give you a VNC socket to talk to

socat TCP-LISTEN:5905,reuseaddr,fork UNIX-CONNECT:/export/zones/bhyve1/root/tmp/vm.vnc

and as yourself, run the vnc viewer

vncviewer :5

Once it's finished booting, log in as root and install with the ec2-baseline overlay which is what makes sure it's got the pieces necessary to work on EC2.

./live_install.sh -G c1t0d0 ec2-baseline

Back as root on the host, ^C to get out of socat, remove the ISO image and reboot, so it will boot from the newly installed image.

zap remove-cd -z bhyve1 -r

Restart socat and vncviewer, and log in to the guest again.

What I then do is to remove any configuration or other data from the guest that we don't want in the final system. (This is similar to the old sys-unconfig that many of us used to Solaris will be familiar with.)

zap unconfigure -a

I usually also ensure that a functional resolv.conf exists, just in case dhcp doesn't create it correctly.

echo "nameserver    8.8.8.8" > /etc/resolv.conf

Back on the host, shut the instance down by shutting down the bhyve zoned it's running in:

zoneadm -z bhyve1 halt

Now the zfs volume you created contains a suitable image. All you have to do is get it to AWS. First copy the image into a plain file:

dd if=/dev/zvol/rdsk/rpool/bhyve1_bhvol0 of=/var/tmp/tribblix-m29.img bs=1048576

At this point you don't need the zone any more so you can get rid of it:

zap destroy-zone -z bhyve1

The raw image isn't in a form you can use, and needs converting. There's a useful tool - the VMDK stream converter (there's also a download here) - just untar it and run it on the image:

python2 ./VMDK-stream-converter-0.2/VMDKstream.py /var/tmp/tribblix-m29.img /var/tmp/tribblix-m29.vmdk

Now copy that vmdk file (and it's also a lot smaller than the raw img file) up to S3, in the following you need to adjust the bucket name from mybucket to something of yours:

aws s3 cp --cli-connect-timeout 0 --cli-read-timeout 0 \
/var/tmp/tribblix-m29.vmdk s3://mybucket/tribblix-m29.vmdk

Now you can import that image into a snapshot:

aws ec2 import-snapshot --description "Tribblix m29" \
--disk-container file://m29-import.json

where the file m29-import.json looks like this:

{
    "Description": "Tribblix m29 VMDK",
    "Format": "vmdk",
    "UserBucket": {
        "S3Bucket": "mybucket",
        "S3Key": "tribblix-m29.vmdk"
    }
}

The command will give you a snapshot id, that looks like import-snap-081c7e42756d7456b, which you can follow the progress of with

aws ec2 describe-import-snapshot-tasks --import-task-ids import-snap-081c7e42756d7456b

When that's finished it will give you the snapshot id itself, such as snap-0e0a87acc60de5394. From that you can register an AMI, with

aws ec2 register-image --cli-input-json file://m29-ami.json

where the m29-ami.json file looks like:

{
    "Architecture": "x86_64",
    "Description": "Tribblix, the retro illumos distribution, version m29",
    "EnaSupport": false,
    "Name": "Tribblix-m29",
    "RootDeviceName": "/dev/xvda",
    "BlockDeviceMappings": [
        {
            "DeviceName": "/dev/xvda",
            "Ebs": {
                "SnapshotId": "snap-0e0a87acc60de5394"
            }
        }
    ],
    "VirtualizationType": "hvm",
    "BootMode": "legacy-bios"
}

If you want to create a Nitro-enabled AMI, change "EnaSupport" from "false" to "true", and "BootMode" from "legacy-bios" to "uefi".

What, no fsck?

There was a huge amount of resistance early on to the fact that zfs didn't have an fsck. Or, rather, a separate fsck.

I recall being in Sun presentations introducing zfs and question after question was about how to repair zfs when it got corrupted.

People were so used to shoddy file systems that were so badly implemented that a separate utility was needed to repair file system errors caused by fundamental design and implementation errors in the file system itself that the idea that the file system driver itself ought to take responsibility for managing the state of the file system was totally alien.

If you think about ufs, for example, there were a number of known failure modes, and what you did was take the file system offline, run the checker against it, and it would detect the known errors and modify the bits on disk in a way that would hopefully correct the problem. (In reality, if you needed it, there was a decent chance it wouldn't work.) Doing it this way was simple laziness - it would be far better to just fix ufs so it wouldn't corrupt the data in the first place (ufs logging went a long way towards this, eventually). And you were only really protecting against known errors, where you understood exactly the sequence of events that would cause the file system to end up in a corrupted state, so that random corruption was either undetectable or unfixable, or both.

The way zfs thought about this was very different. To start with, eliminate all known behaviour that can cause corruption. The underlying copy on write design goes a long way, and updates are transactional so either complete or not. If you find a new failure mode, fix that in the file system proper. And then, correction is built in rather than separate, which means that it doesn't need manual intervention by an administrator, and all repairs can be done without taking the system offline.

Thankfully we've moved on, and I haven't heard this particular criticism of zfs for a while.

A decade of Tribblix

I seem to have just missed the anniversary, but it turns out that Tribblix has existed for slightly over a decade.

The initial blog post on Building Tribblix was published on October 24th, 2012. But the ISO image (milestone 0) was October 21st, and it looks like the packages were built on October 4th. So there's a bit of uncertainty about the actual date, and I had been playing around with some of the bits and pieces for a while before that.

There have been a lot of releases. We're now on Milestone 28, but there have been several update releases along the way, so I make it 42 distinct releases in total. That doesn't include the LX-enabled OmniTribblix variant (there have been 20 of those by the way).

The focus (given hardware availability) has been x86, naturally. But the SPARC version has seen occasional bursts of life. Now I have a decent build system, it's catching up. Will there be an ARM version? Who knows...

Over the years there have been some notable highlights. It took a few releases to become fully self-hosting; package management had to be rebuilt; LibreOffice was ported; Xfce and MATE added as fully functional desktop offerings (with a host of others); a whole family of zones, including reimplementing the traditional sparse root; made available on clouds like AWS and Digital Ocean; network install via iPXE; huge numbers of packages (it's never-ending churn); and maintaining Java by default.

And it's been (mostly) fun. Here's to the next 10 years!

TREASURE - The Remote Execution and Access Service Users Really Enjoy

Many, many years ago I worked on a prototype of a software ecosystem I called TREASURE - The Remote Execution and Access Service Users Really Enjoy.

At the time, I was running the infrastructure and application behind an international genomics service. The idea was that we could centrally manage all the software and data for genomic analysis, provide high-end compute and storage capability, and amortize the cost across 20,000 academics so that individual researchers didn't have to maintain it all individually.

Originally, access was via telnet (I did say it was a long time ago). After a while we enabled X11, so that graphical application would work (running X11 directly across the internet was fun).

Then along came the web. One of my interesting projects was to write a web server that would run with the privileges of the authenticated user. (This was before apache came along, by the way!) And clearly a web browser might be able to provide a more user-friendly and universal interface than a telnet prompt.

We added VNC as well (it came out of Cambridge and we were  aware of it well before it became public), so that users could view graphical applications more easily. This had a couple of advantages - all the hard work and complexity was at our end, where we had control, and X11 is quite latency sensitive so performance improved.

But ultimately what I wanted to do was to run the GUI on the user's machine, wit access to the user's files. Remember that the GUI is then not running where the software, genome databases, and all the compute power are located.

Hence the Remote Execution part of TREASURE - what we wanted was a system that would call across to a remote service to do the work, and return the result to the user. And the Access part was about making it accessible and transparent, which would lead to a working environment that people would enjoy using.

Ultimately, the core of TREASURE was originally a local GUI that knew how to run applications. Written in Java, it would therefore run on pretty much any client (and we had users with all sorts of Unix workstations in addition to Windows making inroads). The clever bit was to replace the java Runtime.getRuntime().exec() calls that ran applications locally with some form of remote procedure call. Being of its time, this might involve CORBA, RMI, SOAP, or JAX-WS with data marshalled as XML. In fact, I implemented pretty much every remote call mechanism available (and this did in fact come in useful as other places did make available some services using pretty random protocols). And then of course there's the server side which was effectively a CGI script.

The other key part was to work out which files needed to be sent across. Sometimes it was obvious (it's a GUI, the user has selected a file to analyse), but sometimes we needed to send across auxiliary files as well. And on the server side it ran in a little sandbox so you knew what output files had bee generated so you could return those.

Effectively, this was a production form of serverless computing running over 20 years ago. Only we called it GRID computing back then.

Another interesting feature of the architecture was the TREASURE CHEST, which was a source of applications. There were literally hundreds of possible applications you could run, and many more if you included interfaces to other providers. So rather than write all those into the app, there was a plugin system where you could download a jar file and run it as a plugin, and the TREASURE CHEST was where you could find these application. Effectively an app store, in modern terminology.

Sadly the department got closed down due to political incompetence, so the project never really got beyond the prototype stage. And while I still have bits and pieces of code, I don't appear to have a copy of the whole thing. A lot of the components would need to be replaced, but the overall concept is still sound.

Tribblix for SPARC m25.1

Following hot on the heels of the Tribblix Milestone 22 ISO for SPARC, it's possible to upgrade that to a newer version. The new version that's available is m25.1.

(If the available versions look a bit random, that's because they are. Not every release on x86 was built for SPARC, and not all of the ones that were actually worked properly. So we have what we have.)

The major jump, aside from the underlying OS, in m25.1 for SPARC is that it brings in gcc7 (for applications, illumos itself is still built with gcc4), and generally there's a bunch of more modern applications available.

To upgrade m22 to m25.1 is a manual process. This is because there are steps that are necessary, and if you don't follow them exactly the system won't boot.

The underlying cause here of the various problems in this process is that it's a big jump from m22 to m25.1 and you will hit bugs in the upgrade process that have been fixed in intermediate releases.

First, take a note of the current BE, eg tribblix. You might need it later if things go bad and you need to reboot into the current (hopefully working) release.

You can manually add available versions for upgrade with the following trick (this is just one line, despite how it might be formatted):

echo "m25.1|http://pkgs.tribblix.org/release-m25.1.sparc/TRIBzap.0.0.25.1.zap|Tribblix m25.1" >> /etc/zap/version.list

and check that's visible with

zap upgrade list

and then start the upgrade with

zap upgrade m25.1

Do not activate or reboot yet!

You MUST do the following:

beadm mount m25.1 /a
zap install -C /a TRIBshell-ksh93
pkgadm sync -q -R /a
beadm umount m25.1

and then you should be safe to reboot:

beadm activate m25.1
init 6

If it doesn't come back, you can boot into the previous release (that you took the name of earlier, remember) from the ok prompt

boot -Z rpool/ROOT/tribblix

Once you're up and running m25.1 it's time to clean up.

zap refresh

and then remove some of the old opensxce packages

zap uninstall \
SUNWfont-xorg-core \
SUNWfont-xorg-iso8859-1 \
SUNWttf-dejavu \
SUNWxorg-clientlibs \
SUNWxorg-xkb \
SUNWxvnc \
SUNWxwcft \
SUNWxwfsw \
SUNWxwice \
SUNWxwinc \
SUNWxwopt \
SUNWxwxft \
SUNWxwrtl \
SUNWxwplr \
SUNWxwplt

and then bring packages up to current

zap update-overlay -a

and this should give you a system that's in a workable state, and roughly matching my active SPARC environment.

Tribblix for SPARC m22 ISO now available

I've made available a newer ISO image for Tribblix on SPARC.

This is an m22 ISO. So it's actually relatively old compared to the mainstream x86 release.

I actually had a number of random SPARC ISO images, but for a while I've had no way of testing any of them. (And many of the problems with the SPARC ISOs in general is because I had no real way of testing them properly.)

Arrive a newish T4-1 (thanks Andy!), and I can now trivially create an LDOM, assign it a zvol for a root disk and a ISO image to boot from, and testing is trivial again. And while some of the ISO images I have are clearly so broken as to not be worth considering, the m22 version looks pretty reasonable.

In terms of available application packages, it exactly matches the old m20 release. I do have newer packages on some of my test systems, but they are built with a newer gcc and so need a proper upgrade path. But that's going to be easier now too.

There is a minor error on the m22 ISO, in that the xz package shipped appears to be wrong. To fix, simply

zap install TRIBcompress-xz

and to update to the latest available applications (the ISO is early 2020, the repo is middle of 2021)

zap refresh
zap update TRIBlib-security-openssl
zap update-overlay -a

The reason for updating openssl on its own is that a number of applications are compiled against openssl 1.1.1, so you need to be sure that gets updated first.

Next step is to push on to something newer.

DevOps as a HR problem

I wrote about one way in which HR and IT can operate more closely, but there's another interaction between IT and HR that might not be so benign.

DevOps is ultimately about breaking down silos in IT (indeed, my definition of DevOps is as a cultural structure where teams work together to meet the needs of the business rather than competing against each other to meet the needs of the team).

However, in a business, individuals and teams are actually playing a game in which the rules and criteria for success are set by HR in the shape of the (often annual) review cycle. And all too often promotions, pay rises, even restructuring, are based around individual and team performance in isolation. And who can blame individuals and teams for optimising their behaviour around the performance targets they've been set?

It's similar to Conway's Law, in which the outputs of an organization mirror its organisational structure - here, the outputs of an organisation will mirror the performance targets that have been set. If you want to improve collaboration and remove silos, then make sure that HR are on board and get them to explicitly put those into the annual performance targets.

On the intersection between IT and HR

A while ago I mentioned The three strands of Information Technology, and how this was split into an internal-facing component (IT for the business, IT for the employee) and external-facing (IT for the customer).

In a pure technology company, there's quite a mismatch, with the customer-facing component being dominant and the internal-facing parts being minimised. In this case, do you actually need an IT department, in the traditional sense?

You need a (small) team to do the work, of course. But one possibility is to assign them not to a separate IT organization but to the HR department.

Why would you do this? Well, the primary role of internal IT in a technology company is simply to make sure that new starters get the equipment and capabilities they need on day one, and that they hand stuff back and get their access removed when they leave. And if there's one part of an organisation that knows when staff are arriving and leaving, it's the HR department. Integrating internal IT directly into the rest of the onboarding and offboarding process dramatically simplifies communication.

It helps security and compliance too. One of the problems you often see with the traditional setup where IT is completely separate from HR is that it can take forever to revoke a staff member's access when they leave; integrating the two functions massively shortens that cycle.

Tribblix m28, Digital Ocean, and vioscsi

A while ago, I wrote about deploying Tribblix on Digital Ocean.

The good news is that the same process works flawlessly with the recently released m28 Tribblix release.

If you recall from the original article, adding an additional storage volume didn't work. But, we now have a vioscsi driver, so has the situation improved?

Yes!

All I did was select an additional volume when creating the droplet. Then if I run format, I see:

AVAILABLE DISK SELECTIONS:
       0. c3t0d1 <DO-Volume-2.5+ cyl 13052 alt 2 hd 255 sec 63>
          /pci@0,0/pci1af4,8@5/iport@iport0/disk@0,1
       1. c4t0d0 <Virtio-Block Device-0000-25.00GB>
          /pci@0,0/pci1af4,2@6/blkdev@0,0

That 25GB Virtio-Block device is the root device used for rpool; the other one is the 100GB additional volume. It's also visible in diskinfo:

TYPE    DISK                    VID      PID              SIZE          RMV SSD
SCSI    c3t0d1                  DO       Volume            100.00 GiB   no  no
-       c4t0d0                  Virtio   Block Device       25.00 GiB   no  no
-       c5t0d0                  Virtio   Block Device        0.00 GiB   no  no

(That empty c5t0d0 is the metadata service, by the way.)

Let's create a pool:

zpool create store c3t0d1

It just works. And performance isn't too shabby - I can read and write at 300MB/s.

There you go. More options for running illumos.

 

Tribblix and static networking on AWS

I've just made available the m27 AMIs for Tribblix. As usual, these are just available in London (eu-west-2).

One thing I've noticed repeatedly while running illumos on AWS is that network stability isn't great. The instance will occasionally drop off the network and stubbornly refuse to reclaim its IP address even if you reboot it. It's not just Tribblix, I run a whole lot of OmniOS on AWS and that does the same thing.

The problem appears to be related to DHCP not being able to pick up the address (even though I can see it ending out the correct requests and getting what look like legitimate responses).

So what I do is convert the running instance from using NWAM and being a DHCP client to having statically configured networking. On first boot it needs to use DHCP, because it cannot know what its IP address and network configuration should be until it's booted up once and used DHCP to get the details. But it's really extremely rare to take an AWS instance and change its networking - you would simply build new instances rather than modifying existing ones - so changing it to static is fine, and eliminates any possibility of DHCP failures messing you up in future.

In the past I've always done this manually, but now there's a much easier way if you're using m27 or later:

zap staticnet

will show you what the system will do, just as a sanity check, and then

zap staticnet -y

will implement the change.

Cleaning up the Java illumos port

This was originally a twitter thread, this is a more permanent expanded version.

When support for Solaris and SPARC was removed from Java, the code that was removed fell into a number of distinct classes, not all of which are equally valuable.

Solaris platform support

First, there's the support for Solaris (and thus illumos) as an operating system (the os tree in hotspot, specifically, but there are other places in the tree that have OS-specific code), which we definitely want to retain in the illumos port of openjdk.

There really isn't much difference between Solaris and illumos from Java's point of view. We have a tiny bit of illumos-specific code in the patches, but that fact is that you can take what I've done and build it on Solaris pretty easily.

Solaris x86 cpu support

Then there's support for the x86 architecture on Solaris (and thus illumos) as a specific platform (the os_cpu tree in hotspot, specifically), which I also want to retain.

SPARC cpu support

Then there's support for SPARC systems (mostly Solaris, but it took out any other OS that wanted to support SPARC too), which I've decided to pass on (if anybody wants it, they're free to do the work, but it's too large a task for me).

With SPARC removed, the remaining platform (x86) is supported by multiple other operating systems, so the basic processor port is no longer my concern. What this means, also, is that it should be relatively straightforward to add in support for arm, for example, if we ever got to the stage where illumos gained a new hardware port.

Studio toolchain support

Then there's support for the (now legacy) Studio toolchain, which was a right pain, was a great motivator for the removal, and I can only say good riddance.

There's some junk around the fact that Studio uses different command line flags to gcc. Sometimes the code has to differ based on idiosyncracies of the Studio compiler. A larger part is the way that Studio and gcc handle inline assembler, which I've almost (but not quite) fully cleaned out.

Support for legacy Solaris versions

Then, when you look closer, there's actually bits of code to support really old Solaris versions or other components that's simply no longer relevant and it would be a good thing to get rid of all that complexity.

Some of this is complex workarounds for bugs that have long been fixed. Some things use dlopen() to handle the fact that not all releases have all the shared libraries. There are still some comments mentioning old Solaris versions that you wonder if they're still relevant.

Support for various quirks and peculiarities

Finally, there's a bunch of code to simply do weird or idiosyncratic things on Solaris just because you can, and I'm actually quite keen to rip all that out and just do things the correct and easy way. Historically, there have been certain interesting peculiarities on Solaris waiting to trip up the unwary, mostly for no good reason whatsoever.

As an example of that last one, Java contains its own implementation of the way that Solaris falls back on the NIS/NIS+ domain for dns searches, and even looks for LOCALDOMAIN in the environment. Oh dear. I've eradicated that.

Another example is that there's still quite a lot of font handling code that's specifically referencing /usr/openwin, and has interesting knowledge of bugs in the OpenWindows font files.

Something I ought to look at is the way that printing is handled. Java thinks SunOS is SYS5 from the point of view of printing, which (thankfully) hasn't really been the case for a very long time. In illumos the state of printing is a mess, as we have a bunch of legacy support built in, while ideally we would just move to CUPS.

I'm also wondering whether to remove DTrace. I haven't seen it working for a while, the native DTrace in Java was always a bit problematic (the performance hit was massive and quite discouraging), and I've always disabled it in my builds.

Together, eliminating these peculiarities reduces the size of the port, makes it easier to maintain (because I don't have to keep fixing the patches as the source changes for other reasons), and makes the Java behaviour consistent with other platforms rather than being needlessly different - after all, Java is supposed to be cross-platform.

On password policies in the 21st century

 One of the scourges of corporate life was the forced monthly password change. As anyone who understands security will know, this was always a terrible idea - it leads to a culture of passwords that are weak, formulaic, and written down.

Another, more widespread scourge, is the use of devious complexity requirements.

Fortunately, the world is changing.

The NCSC called for people to update their approach to passwords.

The NIST Special Publication 800-63 on Digital Identity explicitly covers the fact that forced password changes and complexity rules shouldn't be applied. They call out the problems of bad policy - human behaviour is predictable in the face of stupid rules.

Even OWASP have got in on the act.

I think it's pretty clear that forced password changes and complexity rules are on the way out.

This is reinforced by the fact that the current UK Cyber Essentials certification requires you to follow the NCSC guidance. (Go to the Resources tab, and download the "Cyber Essentials Requirements for IT infrastructure".) Under User Access Control, it's pretty explicit - no password expiry, no password complexity requirements. Given that you need to have CE or equivalent to get UK government or NHS contracts now, this is a pretty big stick.

There's also a general push towards MFA to be used in concert with passwords.

There's another interesting question: how long do passwords need to be?

If you want to be really scared, look at the Hive Systems Password Table. You've probably seen this floating around recently. Almost all regular passwords can be trivially brute forced. If people have rainbow tables, game over.

Only it's not quite that simple. Both NIST and NCSC talk about minimum 8 characters. How is that possibly secure?

The point is that there's a huge gulf between running an optimised cracker on a GPU (incredibly quick), and trying to put a long list of passwords into a website or application. The former is 10s of billions of hashes per second; the latter is 10s of attempts per second. You're looking at a factor of a billion difference between the two attack vectors. And if you follow the recommendations for throttling and lockout, in reality an attacker will get a handful of attempts at most. If you look at the NIST guidance, it wants 8 characters for user-generated passwords, and only 6 passwords for random machine-generated passwords.

In practice, for most decent algorithms, rainbow tables normally only go up to 14-16 characters or so. But this means two things. First, that the ease of brute force and rainbow table attacks is such that you absolutely must keep the encrypted passwords protected, and assume that knowledge of the encrypted password means that the password is compromised. And second, that there's actually no benefit to a minimum password length between 8 and 16. You should allow longer (much longer) but the current attack vectors can either be met with 8 or require more than 16.

Happy passwording!

The datacentre business seems to be very much alive

Last week I went to Cloud Expo Europe at ExCeL.

(Yes, there was a tube strike. No, that didn't affect me much. Had to walk from London Bridge Station to Tower Gateway to get to the DLR, but walking past HMS Belfast, Tower Bridge, and the Tower of London isn't such an imposition.)

Now, "Cloud Expo" is the umbrella event. There are a number of co-located shows - DevOps Live, Cloud and Cyber Security Expo, Big Data and AI World, and Data Centre World.

The one absolutely conspicuous thing to take away was that, despite it being notionally a Cloud Expo, the Data Centre World part was as big as all the others put together. It's all a bit swankier than when I was designing, building, and running small datacentres too.

But the place was awash with power - generators, UPS, PDUs (power strips can be really fancy and light up in all sorts of colours now), cabling. Not to mention DCIM, inventory systems, management software, security, cages, equipment lift systems, fire suppression. The whole shooting match, as it were.

This is interesting. I've had the impression for a while that the datacentre (or colocation, as a variation) business isn't in much of a decline, and this reinforced that view. There's still a lot of on-premise compute, it's not going away.

Despite the idea being propagated by some that the only way is Cloud, it appears that Cloud is additive to on-premise. The vendors I chatted to seemed to be going strong.

The rest of Cloud Expo was really quite muted. But not only was it small, and quiet, but there was really nothing new out there. It had been 2 years since I was last out talking to vendors in person, and the impression I got was that the market is simply stagnant.

Like all business sectors, everything's cyclical, but I suspect that mourning the death of the datacentre and on-premise (including colocation) is premature.

Tribblix updates and https

One good thing to have happened recently is the rise of Let's Encrypt, bringing https to all websites without all the hassle you previously had to go through to get a certificate.

One not quite so good event recently was the switch by Let's Encrypt to certificates signed by their own ISRG X1 root, and more excitingly the expiry of the prior DST Root CA X3 signing certificate.

My experience of this is that most things just worked, but I'm still seeing odd cases where clients can't connect. Generally, browsers work just fine; CLI tools are a bigger issue.

This might be due to a couple of issues. Sometimes the software itself guesses wrong (older openssl 1.0.2 for example); sometimes the system's CA bundle of trusted root certificates needs updating.

For a while now, the Tribblix package repositories have been served over https and the zap tool for package management has been configured to use https. There are cases where it falls foul of the above issues.

This might occur on older Tribblix releases - I've seen this on m22, for example.

It turns out that curl fails, but wget works. Again, that's an example of the inconsistency in behaviour that I see. You need to update the CA bundle on m22, but if the package update tool is broken that's a bit tricky.

There's an ugly hack, though, because zap will try wget if it can't find curl. So just move curl out of the way temporarily:

mv /usr/bin/curl /usr/bin/curl.t
zap refresh
zap update TRIBca-bundle
mv /usr/bin/curl.t /usr/bin/curl

and you should be good to go again.

There's another way, of course: edit the *.repo files in /etc/zap/repositories to change the URL from https to http. That's not particularly recommended (although the packages are signed and the signatures are checked).

One thing that last hack demonstrates is the value in using simple text files.