Tribblix m28, Digital Ocean, and vioscsi

A while ago, I wrote about deploying Tribblix on Digital Ocean.

The good news is that the same process works flawlessly with the recently released m28 Tribblix release.

If you recall from the original article, adding an additional storage volume didn't work. But, we now have a vioscsi driver, so has the situation improved?

Yes!

All I did was select an additional volume when creating the droplet. Then if I run format, I see:

AVAILABLE DISK SELECTIONS:
       0. c3t0d1 <DO-Volume-2.5+ cyl 13052 alt 2 hd 255 sec 63>
          /pci@0,0/pci1af4,8@5/iport@iport0/disk@0,1
       1. c4t0d0 <Virtio-Block Device-0000-25.00GB>
          /pci@0,0/pci1af4,2@6/blkdev@0,0

That 25GB Virtio-Block device is the root device used for rpool; the other one is the 100GB additional volume. It's also visible in diskinfo:

TYPE    DISK                    VID      PID              SIZE          RMV SSD
SCSI    c3t0d1                  DO       Volume            100.00 GiB   no  no
-       c4t0d0                  Virtio   Block Device       25.00 GiB   no  no
-       c5t0d0                  Virtio   Block Device        0.00 GiB   no  no

(That empty c5t0d0 is the metadata service, by the way.)

Let's create a pool:

zpool create store c3t0d1

It just works. And performance isn't too shabby - I can read and write at 300MB/s.

There you go. More options for running illumos.

 

Tribblix and static networking on AWS

I've just made available the m27 AMIs for Tribblix. As usual, these are just available in London (eu-west-2).

One thing I've noticed repeatedly while running illumos on AWS is that network stability isn't great. The instance will occasionally drop off the network and stubbornly refuse to reclaim its IP address even if you reboot it. It's not just Tribblix, I run a whole lot of OmniOS on AWS and that does the same thing.

The problem appears to be related to DHCP not being able to pick up the address (even though I can see it ending out the correct requests and getting what look like legitimate responses).

So what I do is convert the running instance from using NWAM and being a DHCP client to having statically configured networking. On first boot it needs to use DHCP, because it cannot know what its IP address and network configuration should be until it's booted up once and used DHCP to get the details. But it's really extremely rare to take an AWS instance and change its networking - you would simply build new instances rather than modifying existing ones - so changing it to static is fine, and eliminates any possibility of DHCP failures messing you up in future.

In the past I've always done this manually, but now there's a much easier way if you're using m27 or later:

zap staticnet

will show you what the system will do, just as a sanity check, and then

zap staticnet -y

will implement the change.

Cleaning up the Java illumos port

This was originally a twitter thread, this is a more permanent expanded version.

When support for Solaris and SPARC was removed from Java, the code that was removed fell into a number of distinct classes, not all of which are equally valuable.

Solaris platform support

First, there's the support for Solaris (and thus illumos) as an operating system (the os tree in hotspot, specifically, but there are other places in the tree that have OS-specific code), which we definitely want to retain in the illumos port of openjdk.

There really isn't much difference between Solaris and illumos from Java's point of view. We have a tiny bit of illumos-specific code in the patches, but that fact is that you can take what I've done and build it on Solaris pretty easily.

Solaris x86 cpu support

Then there's support for the x86 architecture on Solaris (and thus illumos) as a specific platform (the os_cpu tree in hotspot, specifically), which I also want to retain.

SPARC cpu support

Then there's support for SPARC systems (mostly Solaris, but it took out any other OS that wanted to support SPARC too), which I've decided to pass on (if anybody wants it, they're free to do the work, but it's too large a task for me).

With SPARC removed, the remaining platform (x86) is supported by multiple other operating systems, so the basic processor port is no longer my concern. What this means, also, is that it should be relatively straightforward to add in support for arm, for example, if we ever got to the stage where illumos gained a new hardware port.

Studio toolchain support

Then there's support for the (now legacy) Studio toolchain, which was a right pain, was a great motivator for the removal, and I can only say good riddance.

There's some junk around the fact that Studio uses different command line flags to gcc. Sometimes the code has to differ based on idiosyncracies of the Studio compiler. A larger part is the way that Studio and gcc handle inline assembler, which I've almost (but not quite) fully cleaned out.

Support for legacy Solaris versions

Then, when you look closer, there's actually bits of code to support really old Solaris versions or other components that's simply no longer relevant and it would be a good thing to get rid of all that complexity.

Some of this is complex workarounds for bugs that have long been fixed. Some things use dlopen() to handle the fact that not all releases have all the shared libraries. There are still some comments mentioning old Solaris versions that you wonder if they're still relevant.

Support for various quirks and peculiarities

Finally, there's a bunch of code to simply do weird or idiosyncratic things on Solaris just because you can, and I'm actually quite keen to rip all that out and just do things the correct and easy way. Historically, there have been certain interesting peculiarities on Solaris waiting to trip up the unwary, mostly for no good reason whatsoever.

As an example of that last one, Java contains its own implementation of the way that Solaris falls back on the NIS/NIS+ domain for dns searches, and even looks for LOCALDOMAIN in the environment. Oh dear. I've eradicated that.

Another example is that there's still quite a lot of font handling code that's specifically referencing /usr/openwin, and has interesting knowledge of bugs in the OpenWindows font files.

Something I ought to look at is the way that printing is handled. Java thinks SunOS is SYS5 from the point of view of printing, which (thankfully) hasn't really been the case for a very long time. In illumos the state of printing is a mess, as we have a bunch of legacy support built in, while ideally we would just move to CUPS.

I'm also wondering whether to remove DTrace. I haven't seen it working for a while, the native DTrace in Java was always a bit problematic (the performance hit was massive and quite discouraging), and I've always disabled it in my builds.

Together, eliminating these peculiarities reduces the size of the port, makes it easier to maintain (because I don't have to keep fixing the patches as the source changes for other reasons), and makes the Java behaviour consistent with other platforms rather than being needlessly different - after all, Java is supposed to be cross-platform.

On password policies in the 21st century

 One of the scourges of corporate life was the forced monthly password change. As anyone who understands security will know, this was always a terrible idea - it leads to a culture of passwords that are weak, formulaic, and written down.

Another, more widespread scourge, is the use of devious complexity requirements.

Fortunately, the world is changing.

The NCSC called for people to update their approach to passwords.

The NIST Special Publication 800-63 on Digital Identity explicitly covers the fact that forced password changes and complexity rules shouldn't be applied. They call out the problems of bad policy - human behaviour is predictable in the face of stupid rules.

Even OWASP have got in on the act.

I think it's pretty clear that forced password changes and complexity rules are on the way out.

This is reinforced by the fact that the current UK Cyber Essentials certification requires you to follow the NCSC guidance. (Go to the Resources tab, and download the "Cyber Essentials Requirements for IT infrastructure".) Under User Access Control, it's pretty explicit - no password expiry, no password complexity requirements. Given that you need to have CE or equivalent to get UK government or NHS contracts now, this is a pretty big stick.

There's also a general push towards MFA to be used in concert with passwords.

There's another interesting question: how long do passwords need to be?

If you want to be really scared, look at the Hive Systems Password Table. You've probably seen this floating around recently. Almost all regular passwords can be trivially brute forced. If people have rainbow tables, game over.

Only it's not quite that simple. Both NIST and NCSC talk about minimum 8 characters. How is that possibly secure?

The point is that there's a huge gulf between running an optimised cracker on a GPU (incredibly quick), and trying to put a long list of passwords into a website or application. The former is 10s of billions of hashes per second; the latter is 10s of attempts per second. You're looking at a factor of a billion difference between the two attack vectors. And if you follow the recommendations for throttling and lockout, in reality an attacker will get a handful of attempts at most. If you look at the NIST guidance, it wants 8 characters for user-generated passwords, and only 6 passwords for random machine-generated passwords.

In practice, for most decent algorithms, rainbow tables normally only go up to 14-16 characters or so. But this means two things. First, that the ease of brute force and rainbow table attacks is such that you absolutely must keep the encrypted passwords protected, and assume that knowledge of the encrypted password means that the password is compromised. And second, that there's actually no benefit to a minimum password length between 8 and 16. You should allow longer (much longer) but the current attack vectors can either be met with 8 or require more than 16.

Happy passwording!

The datacentre business seems to be very much alive

Last week I went to Cloud Expo Europe at ExCeL.

(Yes, there was a tube strike. No, that didn't affect me much. Had to walk from London Bridge Station to Tower Gateway to get to the DLR, but walking past HMS Belfast, Tower Bridge, and the Tower of London isn't such an imposition.)

Now, "Cloud Expo" is the umbrella event. There are a number of co-located shows - DevOps Live, Cloud and Cyber Security Expo, Big Data and AI World, and Data Centre World.

The one absolutely conspicuous thing to take away was that, despite it being notionally a Cloud Expo, the Data Centre World part was as big as all the others put together. It's all a bit swankier than when I was designing, building, and running small datacentres too.

But the place was awash with power - generators, UPS, PDUs (power strips can be really fancy and light up in all sorts of colours now), cabling. Not to mention DCIM, inventory systems, management software, security, cages, equipment lift systems, fire suppression. The whole shooting match, as it were.

This is interesting. I've had the impression for a while that the datacentre (or colocation, as a variation) business isn't in much of a decline, and this reinforced that view. There's still a lot of on-premise compute, it's not going away.

Despite the idea being propagated by some that the only way is Cloud, it appears that Cloud is additive to on-premise. The vendors I chatted to seemed to be going strong.

The rest of Cloud Expo was really quite muted. But not only was it small, and quiet, but there was really nothing new out there. It had been 2 years since I was last out talking to vendors in person, and the impression I got was that the market is simply stagnant.

Like all business sectors, everything's cyclical, but I suspect that mourning the death of the datacentre and on-premise (including colocation) is premature.

Tribblix updates and https

One good thing to have happened recently is the rise of Let's Encrypt, bringing https to all websites without all the hassle you previously had to go through to get a certificate.

One not quite so good event recently was the switch by Let's Encrypt to certificates signed by their own ISRG X1 root, and more excitingly the expiry of the prior DST Root CA X3 signing certificate.

My experience of this is that most things just worked, but I'm still seeing odd cases where clients can't connect. Generally, browsers work just fine; CLI tools are a bigger issue.

This might be due to a couple of issues. Sometimes the software itself guesses wrong (older openssl 1.0.2 for example); sometimes the system's CA bundle of trusted root certificates needs updating.

For a while now, the Tribblix package repositories have been served over https and the zap tool for package management has been configured to use https. There are cases where it falls foul of the above issues.

This might occur on older Tribblix releases - I've seen this on m22, for example.

It turns out that curl fails, but wget works. Again, that's an example of the inconsistency in behaviour that I see. You need to update the CA bundle on m22, but if the package update tool is broken that's a bit tricky.

There's an ugly hack, though, because zap will try wget if it can't find curl. So just move curl out of the way temporarily:

mv /usr/bin/curl /usr/bin/curl.t
zap refresh
zap update TRIBca-bundle
mv /usr/bin/curl.t /usr/bin/curl

and you should be good to go again.

There's another way, of course: edit the *.repo files in /etc/zap/repositories to change the URL from https to http. That's not particularly recommended (although the packages are signed and the signatures are checked).

One thing that last hack demonstrates is the value in using simple text files.

Inside zone installation

 How do zones actually get put together on Solaris and illumos? Specifically, how does a zone get installed?

 There are various type of zones. The nomenclature here is a brand. A zone's brand defines how it gets installed and managed and its properties. Often, this is mapped to a zone template which is the default configuration for a zone of that type or brand.

 (By the way, this overlap between template and brand can be seen in the create subcommand of zonecfg. You do "create -t SUNWlx" to build a zone from a template, which is where the -t comes from. It's not the create that sets the brand, it's the template.)

 The templates are stored as xml files in /etc/zones. As are the configured zones, which is a bit confusing. So in theory, if you wanted to generate a custom template to save adding so much to your zonecfg each time, you could add your own enhanced template here. The actual zone list is in /etc/zones/index.

 In fact, Tribblix has template zones, which are sparse-root zones built from a different image to the global zone. They are implemented by building an OS image that provides the file systems to be mounted read only, and a template xml file configured appropriately.

 One of the things in the template is the brand. That maps to a directory under /usr/lib/brand. So, for example, the TRIBsparse template in /etc/zones/TRIBsparse.xml sets the brand to be sparse-root, in addition to having the normal lofs mounts for /usr, /lib, and /sbin that you expect for a sparse-root zone. There's a directory /usr/lib/brand/sparse-root that contains everything necessary to manage a sparse-root zone.

 In there you'll find a couple more xml files - platform.xml and config.xml. A lot of what's in those is internal to zones. Of the two, config.xml is the more interesting here, because it has entries that match the zoneadm subcommands. And one of those is the install entry. For TRIBsparse, it is

 /usr/lib/brand/sparse-root/pkgcreatezone -z %z -R %R

 When you invoke zoneadm install, this script gets run, and you get the zone name (-z) and zonepath (-R) passed in automatically. There's not much else that you can specify for a sparse root zone. If you look at the installopts property in config.xml, there's just an h, which means that the user can specify -h (and will get the help).

 For a whole-root zone the install entry is similar, but installopts is now o:O:h - this is like getopts, so it's saying that you can pass the -o and -O flags, and that each must have an argument. These flags are used to define what overlays get installed in a whole-root zone. Having the installopts defined here means that zoneadm can validate the install command.

 So, for a given brand, we've now seen from config.xml what command will be called when you install a zone, and what options it's allowed.

 The point is that there's nothing special here. You can build a custom brand by writing your own install script, and if you need to pass arguments to it you can easily do so as long as you set installopts to match. When building all the zone brands for Tribblix, that's all I did.

 To reiterate, the install script is completely open. For existing ones, you can see exactly what it's going to do. If you want to create one, you can have it do anything you like in order to lay down the files you want in the layout you want.

 As a crazy example, a long time ago I created a brand that built a sparse-root zone on a system using IPS packaging.

 There's a little bit of boilerplate (if you're going to create your own brands, it's probably easier to start with a copy of an existing one so you pick up the common actions that all zone installs do), but after that, the world's your oyster.

 Consider the alien-root zone in Tribblix. If you look at the installer for that, it's just dumping the contents of an iso image, tarball, or zfs send stream into the zone root. It does some cleanup afterwards, but generally it doesn't care what's in the files you give it - you can create an arbitrary software installation, tar it up, and install a zone from it.

 (In fact, I probably won't create more native zone types for Tribblix - the alien-root is sufficiently generic that I would extend that.)

 This generality in scripting goes beyond the install. For example, the prestate and poststate scripts are called before or after the zone transitions from one state to another, and you can therefore get your zone brand to do interesting things triggered by a zone transitioning state. One of the coolest uses here is the way that OmniOS implements on-demand vnics - the prestate script creates a vnic for a zone before a zone boots, and the poststate script tears it down after it halts. (Tribblix uses zap to manage vnics outside of zoneadm, so they're persistent rather than on-demand, it's just a different way of doing things.)

 As you can see, you aren't limited to the zone types supplied by your distribution. With enough imagination, you can extend zones in arbitrary ways.

Are software ecosystems a good thing?

 One way to judge the health or strength of a product might be to look at the ecosystem surrounding that product. But is this diagnostic?

 Note that there are several concepts here that are similar to the ecosystem. I'm not referring to the community, those people who might use or support the product. Nor am I talking about a marketplace, which is a source of artefacts that might be consumed by the product. Those are important in their own right, but they aren't what I mean when I'm talking about an ecosystem.

 No, an ecosystem is the set of other services or software that spring up to support or integrate with the product.

 There's one immediate problem here that's obvious if you think about it. Much of the ecosystem thus exists to address flaws or gaps in the product. Something that is more polished, more mature, and more finished will provide fewer opportunities for other products to add value.

 What this means, then, is that a thriving ecosystem is often a sign of weakness and immaturity, not strength. A good product will not need the extras and hangers on that come with an ecosystem.

 The notion of an ecosystem is tied in with that of MVP - Minimum Viable Product. The current trend is to launch a startup with just an MVP, rely on first mover advantage, and hope to actually finish the offering at a later date. By definition, an MVP cannot be complete, and will need a surrounding ecosystem in order to function at all. This is much more common now than in the past, when products - especially proprietary products - were not launched until they were in some sense done.

 Over time, too, an ecosystem will - or should - naturally diminish, as bugs are fixed and missing features filled in. The partners in the ecosystem will get frozen out, as their offerings become irrelevant (think ClusterHQ).

 As an example from the past, consider the ecosystems that built up around Windows and DOS. Whole industries were built on things like TCP stacks and centralized nameservices and storage (PC-NFS, even Netware). These were products reliant on fundamental failings of the product they supported. (Don't even get me started on antivirus software.)

 Fast forward, and I can't be the only one to recognise the CNCF landscape as a disaster area.

The three strands of Information Technology

How are IT departments structured? I've seen a variety of ways to do this. It depends on the individual business, but over the years I've come up with a way to think about this.

When thinking about Information Technology (IT), it naturally splits into 3 separate strands:

IT for the business

This is the provision of facilities for HR, Finance, Sales, and the like; basic facilities for the organisation to operate as a business

IT for the employee

This is the provision of systems and tools for employees to be able to work at all; laptops/desktops/mobile devices, and communications systems such as telephony and email, together with a way for staff to store and collaborate on documents

IT for the customer

This is the provision of services that your customers use, whether that's a product you sell in its own right, or as a mechanism to sell other products

The relative importance of these 3 strands depends on the nature of the business, of course. And very small organisations might not even have all 3 strands in any meaningful sense.

Structurally, there are two senior roles that an organisation might have, the CIO and CTO.  And the way things would naturally be laid out is that the CIO looks after IT for the business and IT for the employee, while the CTO gets IT for the customer.

Splitting things this way works because the characteristics of the strands are quite different. The responsibilities of the CIO are inward-facing, those of the CTO are outward-facing. The work of the CIO is about managing standardised commodities, while the CTO's role is to provide differentiation. Polar opposites, in a way.

There's a third role, that of the CISO, responsible for information security. This is slightly different in that it cuts across all 3 strands. As such, if you have both a CIO and a CTO, it isn't entirely obvious which of the two, if either, should take on the CISO role.

Given the different nature of these 3 strands, where does the IT department (loosely defined as those people whose job is IT) fit? Should you even have one? The job requirements for the 3 strands are sufficiently different that having different IT teams for each strand would seem to make an awful lot of sense, rather than a central IT department. And the IT team for each strand reports to the CIO or CTO as appropriate. In particular, having a product developed in the CTO part of the organisation and then thrown over the wall to be run by an operations team in the CIO organisation is one of the organisational antipatterns that never made any sense and was a major driver for DevOps.

Thus, when structuring the delivery of IT in an organisation, considering the divergent needs of the 3 different IT strands ought to be taken into account. Worst case is a single department that standardises on the same solution to deliver all 3 strands - standardisation is a common refrain of management, but what it really means here is that at least 2 strands (if not all 3) are delivered in a sub-standard way, often in a way that's actually completely unsuitable.

There is a central IT function that does cut across all 3 strands, in the same way that a CISO does at the management level. Which is a compliance function or security office. But for most other functions, you're really looking at providing distinct deliveries for each strand.

The cost of cloud

Putting your IT infrastructure into the cloud seems to be the "in" thing. It's been around for a while, of course. And, like most things related to IT, there are tradeoffs to be made.

My rough estimate is that the unit cost of provisioning a service on AWS is about 3 times that of a competent IT organization providing a similar service in house. Other people have come to the same number, and it hasn't really changed much over the last decade. (If you don't think 3x is right, consider what AWS' gross margin is.)

Some services offered by AWS deviate from that simple 3x formula. The two obvious ones are network costs, which as Cloudflare have argued are many times higher than you would expect, and S3, which you're going to struggle to beat. (Although if you're going to use S3 as a distribution site then the network costs will get you, think about Wasabi for that.)

And yet, many organizations move to the cloud to "save money". I'm going to ignore the capex versus opex part of that, and simply note that many IT organizations have in-house operations that are neither very efficient nor cost-effective. In particular, traditional legacy IT infrastructures are ridiculously overpriced. (If you're using commercial virtualization platforms and/or SAN storage, then you're overpaying by as much as a factor of 10, and getting an inferior product into the bargain - so that while many organizations could save a huge amount of money by moving to the cloud, they could save even more by running their internal operations better.)

Often the cost saving associated with a migration - not just cloud, this applies to other transitions too - comes about not because the new solution is cheaper, but because a migration gives a business leverage to introduce better practices. Practices that, if used for your on-premise deployments, would save far more than the cloud ever could. Sometimes, you need to do an end run round an entrenched legacy IT empire.

Another consideration is that the cloud has often been touted as something where you pay for what you use, which isn't always quite correct. For many services, you pay for what you configure. And some services are nowhere near as elastic as you might wish.

Capacity planning doesn't go away either, it's actually more important to get the sizing right, and while you can easily buy more capacity, you have to ensure you have the financial capacity to pay the bills.

Note that I'm not saying you should always run your systems on-premise, nor that it will always be cheaper.

Below a certain scale, doing it yourself isn't financially beneficial. There's a minimum configuration of infrastructure you need in order to get something that works, and many small organizations have needs below that. But generally, the smaller providers are likely to be a better option in that case than full-on cloud offerings.

Having the operational capability to support your infrastructure is also crucial. If you're going to support your own hardware, you really need a team, which is going to set a minimum scale at which operations are worthwhile.

This becomes even more true if you need to deploy globally. It's difficult to do that in-house with a small team, and you have to be pretty large to be able to staff multiple teams in different geographies. A huge advantage of using the cloud for this is that you can deploy into pretty much any location without driving your costs insane. Who wants to hire a full team in every country you operate in? And operationally, it's the same wherever you go, which makes things a lot easier.

In recent times, the Coronavirus pandemic has also had an impact. End user access to colocation facilities has been restricted - we've been able to do repairs, recently, but we've had to justify any datacenter visits as essential.

There are certain workloads that are well matched to the cloud, of course, Anything highly variable, with spikes above 3x the background, will be cheaper in the cloud where you can deploy capacity just for the spike than it would be in house where you either overprovision for peak load or accept that there's a spike you can't handle.

The cloud is also great for experimentation. You can try any number of memory and CPU configurations to see what works well. Much easier than trying to guess and buying equipment that isn't optimal. (This sort of sizing exercise is far less relevant if you have decent virtualization like zones.)

You can even spin up a range of entirely different systems. I do this when testing, just run each of a whole range of Linux distros for an hour or so each.

What the above cases say is that even if the unit cost of cloud resources is high, the cloud gives you more of an opportunity to optimize the number of units. And, when it comes to scaling, this means the ability to scale down is far more important than the ability to scale up.

I use AWS for a lot of things, but I strongly regard the cloud as just another tool, to be used as occasion demands, rather than because the high priests say you should.

Keeping Java alive on illumos

Back in 2019, a new JEP (JDK Enhancement Proposal) appeared.

JEP 362: Deprecate the Solaris and SPARC Ports

Of course, for those of us running Solaris or illumos (which is the same platform as far as Java is concerned), this was a big deal. Losing support for a major language on the platform was potentially a problem.

The stated reason for removal was:

Dropping support for these ports will enable contributors in the OpenJDK Community to accelerate the development of new features that will move the platform forward.

Clearly, this reflected a belief that maintaining Solaris and/or SPARC was a millstone dragging Java down. Still, it's their project, they can make whatever decisions they like, despite those of us who thought it was a bad move.

Eventually, despite objections, the ports were removed, towards the end of the JDK15 cycle.

At which point I simply carried on building OpenJDK. All I did was take the patch from the commit that removed Solaris support, applied that backwards, and added on top the pkgsrc patches that Jonathan Perkin had originally developed to support a gcc port on Solaris and illumos - patches we had already been using extensively from JDK11 onwards.

At that point I wasn't quite sure how sustainable this was. My aim was to support it as long as it wasn't proving too onerous or difficult, and my most optimistic hope was that we might be able to get to Java 17 which was planned as the next LTS release.

The modus operandi was really very simple. Every week a new tag is created. Download the tag, apply the patches, fix any errors in the patch set, try a build, hopefully fix any problems breaking the build.

Rinse and repeat, every week. The idea is that by doing it every week, it's a relatively small and manageable set of changes each time. Some weeks, it's just line number noise in the patches. Other weeks, it could be a more significant change. By loitering on the mailing lists, you become aware of what changes are coming up, which gives you a good idea of where to look when the build breaks.

Along the way, I've been cleaning up the patches to eliminate the SPARC code (you could put it back, but it's not a focus of this project) and most of the code to support the Studio toolchain (the version of Studio to build current Java isn't compatible with illumos anyway). So what we're left with is a straightforward Solaris/illumos+gcc port.

Most of the code changes I've needed to make are fairly straightforward procedural changes. Some functions moved namespace. Some function signatures have been changed. There's also been a lot of work to consolidate a number of functions into common posix code, rather than have each OS provide different implementations which might diverge and become hard to maintain.

Most of this was pretty simple. The only one that caused me a significant amount of work was the signal handling rewrite, which took several attempts to get to work at all.

And it's become fairly routine. Java 17 came along, eventually, and the builds were still succeeding and basic smoke-testing worked just fine. So, illumos has Java 17 available, just as I had hoped.

I originally packaged the builds on Tribblix, of course, which is where I'm doing the work. But I've also dropped tarballs of occasional builds so they can be downloaded and used on other illumos distributions.

Actually, the idea of those builds isn't so much that they're useful standalone, but they provide a bootstrap JDK that you can use to build Java yourself. Which, given that bootstrap JDK and my java patches, ought to be fairly straightforward. (There's a separate patch directory for each jdk release - the directory name ought to be obvious.)

Which means that if you want Java 17 on OmniOS, you can have just that - it's built and packaged ready for you. Not only that, Dominik fixed some problems with my signal handling fix so it works properly and without errors, which benefits everyone.

It doesn't stop there. In addition to the stream of quarterly updates (JDK 17 being an LTS release will see these for some time yet) work is continuing on mainline. JDK 18 works just fine, and as it's ramping down for release shouldn't have any breaking changes, so that's another release supported. I'm building JDK 19, although that's only about 1 build in so hasn't really had any significant changes put into it yet.

The fact that a relatively unskilled developer such as myself can maintain an out of tree Java port for a couple of years, tracking all the upstream changes, does make you wonder if supporting Solaris was really that much of a blocker to progress. At the time my belief was that it wasn't Solaris support that was the problem, but the Studio toolchain, and I think that's been borne out by my experience. Not only that, but the consolidation and simplification of the various OS-specific code into common posix code shows that supporting a variety of modern operating systems really isn't that hard.

Running Tribblix on Digital Ocean

A relatively recent feature offered by Digital Ocean is the ability to deploy your own custom image. So, can I deploy a Tribblix image to Digital Ocean?

Short answer: Yes!

For the process, read on.

I'm using Bhyve to create the image. This is a slight variation on the Installing Tribblix in Bhyve on Tribblix procedure.

The basic process looks like this:

• Boot Tribblix in Bhyve
• Install it
• Tweak the installed image for Digital Ocea
• Copy the ZFS volume to Digital Ocean

The first variation on the previous install is that I make the ZFS volume a bit smaller. We can resize it when it's deployed, so we don't need to make it too big. A 4G volume is fine; any smaller and there won't be any space for our swap partition. It doesn't actually matter too much, as we compress the image anyway

zap create-zone -t bhyve -z bhyve1 \
-x 192.168.0.236  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 4G

Install as before,

./live_install.sh -G c1t0d0

remove the cdrom and reboot as before.

We now need to set up networking. We need to do this on a temporary basis, as we don't want any of these network configurations to carry over to the installed image. So I temporarily disable nwam, and manually bring up a working network. When the image boots on Digital Ocean, all this configuration will have been forgotten and it will bring up nwam as normal.

So run the following in the newly installed guest:

svcadm disable -t network/physical:nwam
ifconfig vioif0 plumb
ifconfig vioif0 up
ifconfig vioif0 inet 192.168.0.236/24
route add net default 192.168.0.1
echo "nameserver    8.8.8.8"  > /etc/resolv.conf

Now we need to tweak the image. At some later point this will all be integrated into the installer so it will just work. But for now, we'll start by applying any updates:

zap refresh
zap update-overlay -a

Now for the little tweak. I'm going to add a metadata service that will run at boot on Digital Ocean and do the sort of things that cloud-init would do. Fortunately, there's one for illumos, and it's packaged for Tribblix, so install it:

zap install TRIBmetadata-agent

If you look with svcs you'll see that it's offline. That's not a problem (it's because we've got a temporary manual network setup) - once we boot properly on Digital Ocean we'll have nwam running and the metadata service will run just fine.

We can tidy up and save a bit of space:

zap clean-cache -a

and shut down the zone (and the newly installed instance of Tribblix):

zoneadm -z bhyve1 halt

What we want is a raw image. So all we do is dd the ZFS volume to a file.

dd if=/dev/zvol/rdsk/rpool/bhyve1_bhvol0 \
of=/var/tmp/tribblix-do-m24.1.img bs=1048576

That's a 4G file, the size of the volume. As it's stored on ZFS, and ZFS compression is on, it will actually consume a lot less space as the image is mostly empty. But what we don't want to do is upload 4G of empty space. So we can compress it:

gzip -9 /var/tmp/tribblix-do-m24.1.img

(it ends up as about 300M), or you could use bzip2, I think.

There are two options when you upload the image to Digital Ocean - you can either do a direct upload through the browser, or you can give it a URL where the image can be found and get Digital Ocean to pull it from there. I found it much easier to scp the image up to an existing webserver and get Digital Ocean to grab it, as I don't trust browsers to behave well.

Log in to Digital Ocean.

Select 'Images' from the left hand menu
Tab to 'Custom Images'
Import via URL
[Insert the URL where you've stashed the file]

You then get a dialog

Name - tribblix-do-m24.1.img.gz
Distribution - unknown (it's not on the list)
Region - London

Obviously, for me, London is conveniently local. You get warned there will be a charge, and it shows as pending for a few minutes.

Then it pops up 'your image is ready to use'.

To the right of the image in the list is a 'More' dropdown menu from which you can start a droplet. So off we go.

It selects a pretty hefty instance type by default. Reset that to the very cheapest. Choose the ssh key you're going to use, pick a useful (and shorter) hostname, and Create Droplet.

Don't bother with block storage. That's exposed by virtio-scsi, which illumos doesn't yet support.

It'll take a few moments to create the droplet, and once it's ready you'll see the IP address.

At this point, if everything has worked, you should be able to ssh in as root with the ssh key you chose. Chances are that doesn't quite work yet. If it doesn't, simply ssh in as jack, from where you can su to root.

(Remember, jack is on the live ISO, and we haven't deleted it. During experiments I tend not to, to give myself a way in if things don't work right. A proper production image would have the jack user removed and password login for root disabled.)

If the metadata service hasn't run properly (it will resize the ZFS pool, change the hostname, and add the correct key to ssh in as root) then you can restart the metadata service:

svcadm restart metadata

Now ssh to root works, the hostname is set, and the zfs pool has been expanded to the full size.

None of the above is excessively specific to Tribblix, the same general process will work for any of the illumos distributions. (Although you may have to build and install the metadata service yourself.)

Installing Tribblix in Bhyve on Tribblix

One of the big new features recently added to illumos is the Bhyve hypervisor. Rather that the shared-kernel application-level virtualization offered by zones, think of something like VirtualBox, KVM, or Qemu.

One of the things that I am using Bhyve for is to test the Tribblix ISO images and the installer. This allowed me to shrink the installer footprint slightly in recent releases (and showed that one of the tricks I tried wasn't going to work).

Using Bhyve requires a fairly modern system (my own is a little old, and has an Intel Core i7, and works fine). The instructions here also need you to be running current Tribblix, m24.1 or newer.

So, how do I install Tribblix in Bhyve on Tribblix? Most of the following needs to be done as root.

First, make sure that you have the software installed:

zap install-overlay bhyve

I'm doing this on my desktop, so I'm using VNC to connect.

zap install-overlay retro-desktop

You need the current ISO. Again, this has to be m24.1 or newer. Correct the path below to match wherever you've downloaded it to.

While you don't actually have to run bhyve in a zone, that's the standard and most convenient way. So, I use zap to create a zone:

zap create-zone -t bhyve -z bhyve1 \
-x 192.168.0.236  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 8G

Let's run through these. The -t flag says to create a bhyve zone, -z gives it a name, -x gives it an (exclusive) IP address, -I tells it where the ISO image is, and -V tells it to create a ZFS volume of the given size. The other argument that may be of interest is -m, which allows you to set the amount of memory allocated; it defaults to 1024M which is fine for Tribblix.

Then you need to be able to connect to the instance. I'm going to use VNC to connect to the console. We use socat to wire up the bhyve socket in the zone to a network port in the global zone that we can connect to.

socat TCP-LISTEN:5905,reuseaddr,fork UNIX-CONNECT:/export/zones/bhyve1/root/tmp/vm.vnc

You may need to modify the zone name in the path, and you can choose the TCP port (for VNC, it's offset by 5900, so the 5905 is :5 in VNC-speak).

 

 

 

Then, as yourself, you can start a VNC client

vncviewer :5

If that can't connect, then one possibility is that bhyve can't start because it doesn't have enough memory. One little trick to make sure there's enough headroom is to create a file in /tmp and then delete it immediately.

mkfile 1200m /tmp/1200m ; rm /tmp/1200m

You can then log in to the live environment. And do an install like so:

./live_install.sh -G c1t0d0

 

 

 

 

and the install will carry on

 

 

 

 

 

If you want to run this, you have to avoid booting off the CD. The way to do this is to remove the cdrom from the zone specification before you reboot the guest, like so:

zonecfg -z bhyve1 remove attr name=cdrom
zonecfg -z bhyve1 remove fs dir=/var/tmp/tribblix-0m24.1.iso special=/var/tmp/tribblix-0m24.1.iso
zoneadm -z bhyve1 reboot

Obviously, adjusted for the zone name and the ISO file name. I need to teach zap how to do this properly. And then start socat again (if you stopped it) and reconnect using VNC.

You may need to fiddle the networking in the installed guest. The exclusive-ip setting here will force the guest to use the given address. That's unlikely to work terribly well, as the chances of allocating the same address that DHCP would hand out are pretty remote. So to get it on the network you may have to fiddle in the guest by hand.

svcadm disable network/physical:nwam
echo "192.168.0.236/24" > /etc/hostname.vioif0
echo "192.168.0.1" > /etc/defaultrouter
svcadm enable network/physical:default

and populate /etc/resolv.conf with something relevant so that dns works

echo "nameserver 8.8.8.8" > /etc/resolv.conf

 

 

 

 

 

 

Once you're done, you can destroy the zone with

zap destroy-zone -z bhyve1

which will also destroy the ZFS volume it was using.

While the above refers to installing Tribblix inside Bhyve, the same general procedure ought to work to install other illumos distributions.

Tweaking the Tribblix ISO

For the latest release, I've made a couple of tweaks to the Tribblix ISO image.

The first is that it's now a hybrid ISO. So you should be able to write it (simply dd it) to a USB stick and use that to boot. I've never managed to both create and test a usb image, so this should be a win.

The hybrid ISO also boots under bhyve, which the old ISO never did. So that's a second win.

I didn't do any of the work here, simply taking the steps from the OmniOSce kayak build as suggested by tsoome. But the script I use is here.

The second tweak is that the man pages aren't present in their normal form on the live image. Instead, they're shipped as a compressed tarball. This saves a significant amount of space - the manuals are simple text and compress (with bzip2) really well. This doesn't make a great deal of difference to the size of the ISO, but it reduces the size of the ramdisk we boot from by 10%. And that's important because an overlarge ramdisk chews up too much memory if you're trying to install on a small (1GB or less) system.

Of course, the installer now knows to unpack the man pages during installation. This meant I had to add bzip2 to the live image, but that's tiny compared to the space saving you get in return.

I was hoping to use pbzip2, so that if you're on a multiprocessor machine the uncompression is sped up. That turns out not to be a good idea. The problem is that pbzip2 is written in C++, so needs libstdc++. I don't have that, or any of the gcc runtime, on the live image. Adding it would undo all the space savings I was trying to get, but I've not noticed any slowness.

This can be adjusted a bit. The downside is that you have no access to the manual during installation; I could easily have a small subset of the man pages available, and compress away the majority.

The two features came together, thanks to bhyve. Having an image I can boot and install on bhyve is a huge advantage, as it allows me to test the ISO and the installer much more easily. It was that testing that demonstrated that pbzip2 wouldn't work.

Running Eclipse on current illumos

You can run a slightly old version of Eclipse on illumos. The download is here.

Later Eclipse (and SWT) versions didn't have Solaris support. Or any unix variant such AIX or HPUX either, or certain hardware platforms, come to that.

However, if you try and run that eclipse on current Tribblix or OpenIndiana, you'll find it doesn't work and you get a Java crash.

The way I found to fix this is to edit the eclipse.ini file to force the use of GTK2, by adding the following 2 lines in the middle

--launcher.GTK_version
2

so that the full eclipse.ini looks like:

-startup
plugins/org.eclipse.equinox.launcher_1.3.201.v20161025-1711.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.solaris.x86_64_1.1.401.v20161122-1740
-showsplash
org.eclipse.platform
--launcher.defaultAction
openFile
--launcher.GTK_version
2
--launcher.appendVmargs
-vmargs
-Dosgi.requiredJavaVersion=1.8
-Xms256m
-Xmx1024m

The reason this is necessary is because we now have GTK3, and if you don't explicitly force GTK2 you end up with both GTK2 and GTK3 loaded and chaos ensues.

The failure with GTK3 is slightly ironic, given that the reason for dropping Solaris support in eclipse was that we were stuck on GTK2.

Installing OmniOS on Vultr with iPXE

A while ago, I wrote about booting Tribblix on Vultr using iPXE.

Naturally, the question arises - is this possible for other illumos distributions, like OmniOS?

First, though, a note about why this might be interesting. Imagine I have some servers in a remote datacenter, and I need to install an OS. Most servers have some sort of remote managamenet capbility that includes mounting an ISO image. That's fine, but if you're at home then you're having to transfer the ISO image the wrong way across an asymmetric network connection. It's very slow, quite unreliable, and sometimes plain doesn't work. On the other hand, booting iPXE is very quick and very reliable, because the bootable image is so small. And the same benefits mean you can repeat the process until you've got it right.

Based on that, Vultr is just a very convenient way to test this out.

Because of the way that the OmniOS media are built, a straight copy of the Tribblix procedure won't work. This is what you need to do instead.

First, you need an accessible web server. I'm using the Tribblix repo server, but it really can be any server. It doesn't need to be illumos, as long as it responds to http requests you're good.

In the document root, create a directory. I'm going to call it 'kayak', because that's what the OmniOS network installer is called, and that's what we're going to leverage.

There are 3 files from the OmniOS distribution or media that we need.

You can't use the boot archive from the ISO. You need a custom boot archive for this. Download the miniroot file from the https://downloads.omniosce.org/media/ location. If you look in each release directory, you should find a gzipped miniroot file. Put that in the kayak directory, and gunzip it. I normally rename this to be platform/i86pc/amd64/boot_archive. This is going to be the initrd file in your ipxe configuration.

The next thing you need is the corresponding kernel, aka unix. You need to extract this either from the ISO image or the miniroot. The ISO image might be easier; the miniroot contains a UFS filesystem. You could mount this up using lofi and copy the file out of it. You need the file /platform/i86pc/kernel/amd64/unix, and it needs to be located at platform/i86pc/kernel/amd64/unix.

Another way if you have the iso-read command installed (on Tribblix it's in the TRIBlibcdio package) is

mkdir -p platform/i86pc/kernel/amd64
iso-read -i omniosce-r151030.iso \
  -e platform/i86pc/kernel/amd64/unix \
  -o platform/i86pc/kernel/amd64/unix

The third thing you need is the image to install. This is a compressed zfs send stream. You can download these from https://downloads.omniosce.org/media/ for most releases - look for the .zfs.xz file. Make sure not to use the ones with ngz in the name - they're for zones.

If you're trying r151030, then there isn't a downloadable file, but you can pull this file out of the ISO too. For example

iso-read -i omniosce-r151030.iso \
  -e kayak/kayak_r151030.zfs.xz \
  -o kayak_r151030.zfs.xz

Then you need to create a client configuration script. The instructions at https://omniosce.org/setup/pxe.html are a good start. For example, it might be:

BuildRpool c1t0d0
RootPW '$5$JQkyMDvv$pPzEUsvP/rLwURyrpwz5i1SfVqx2QiEoIdDA9ZrG271'
SetRootPW
SetHostname omnios30
SetTimezone UTC
Postboot '/sbin/ipadm create-if vioif0'
Postboot '/sbin/ipadm create-addr -T dhcp vioif0/v4'

Obviously, you need to use the correct disk device (c1t0d0) and network interface (vioif0) appropriate to your system. These are the right values for Vultr.

The installer will look for this file using its (uppercased) MAC address, and then go for shorter parts of the MAC address. As I don't know what that is yet, I simply create a file and create symlinks 0-9 and A-F pointing to it, as it will eventuially get to the first letter of the MAC address and one of them will match. If you know the actual MAC address of the system, you can create a file with the exact match.

What you end up with is an ipxe.txt (and this is the one I'm using at http://pkgs.tribblix.org/kayak/ipxe.txt) file that looks like this:

#!ipxe
dhcp
kernel /kayak/platform/i86pc/kernel/amd64/unix -B install_media=http://18.130.117.123/kayak/kayak_r151030.zfs.xz,install_config=http://18.130.117.123/kayak
initrd /kayak/platform/i86pc/amd64/boot_archive
boot

To recap:

• The initrd file points at the location of the gunzipped miniroot file you downloaded.
• The kernel line refers to the unix file you extracted.
• The kernel line supplies boot arguments with the -B flag. There are two of them.
• NOTE: the kernel line is just one line, despite how its gets formatted here.
  
• The install_media argument is a URL to the ZFS send stream file you downloaded. Note that I'm using the IP address of the server rather than its DNS name, as you can't guarantee that DNS resolution will work.
• The install_config argument is the URL of the directory that the client configuration scripts are put in. The installer will look for files based on (a substring of) the client's MAC address in this directory

With that all set, you're ready to boot a machine on Vultr.

Go to instances, and click on 'Deploy instance'.

Choose a server. Cloud Compute is the basic starter, I normally use London as it's close to me to minimize latency.

For Server Type, go to the Upload ISO tab, choose iPXE isntaed of My
ISOs, and enter the URL of your ipxe file, ipxe.txt

 

Then select your server size - even the smallest 1024MB instance will work fine, and hit the Deploy Now button.

It should show as Installing for a few moments, and then change to Running. If you click on Running you get to the page for that instance, and the view the console link does exactly that using VNC.

The install is pretty quick:

 

(You can see it trying the variant client configuration scripts.)

And it will then reboot into the installed system.

This is a typical minimalist OmniOS install. You'll probably have to set DNS up (if you didn't do it in your client confiiguration script), add a user, and then you'll be able to ssh in remotely.

Adventures in Server Rooms

I'm a fan of the cloud. Honest. Providing someone else is paying for it, that is, given how ridiculously expensive it is.

But running stuff in the cloud is a whole lot easier for me. Someone else fixing hardware and infrastructure is a big win. No more driving to the datacenter at 3 in the morning. No more wandering into a room that's baking at 70C because the AC failed. No crawling about under the raised floor trying to trace that cable. No having a disk array in pieces because vendor A's SCSI adapter doesn't quite work right with vendor B's SCSI disk drive.

That said, some of the things that happened while I was fixing hardware do stand out.

Like the time I was putting together one of these stupid cable management arms. Feeling very satisfied with myself, I tried to step back only to discover that I had managed to velcro and cable tie myself into the damn thing. Unable to actually move the one arm, or get round it with the other, I eventually wriggled out of the stuck sweater and spent another 10 minutes rescuing my clothing from the clutches of the evil server.

Or, one morning, when I came in to find an SGI Challenge dead as a dodo. This was the under the desk model, back in the day when every machine had its own VT320 or similar perched on the desk above or alongside. I speak to SGI, and they assure me that everything's fine, I just need to replace the fuse. I dutifully do so and crawl under the desk to turn it back on, with my boss (the benefits of rank!) standing as far away as possible. One huge flash and enormous bang were followed by another bang as my head slammed into the underside of the desk. Some sharp words with the SGI support line followed.

Before we had datacenters, we had server rooms. Before that, we had cupboards. As a temporary measure, the campus router (oh, the days when you could find a Cisco AGS+ anywhere) was in a cupboard. The air conditioning unit extracted moisture from the air  into what was effectively a bucket on a scale, When the bucket was full, it was heavy enough to tip the scale, and powered off the AC unit - after all, you don't want puddles all over the floor. Depending on humidity, this wouldn't last overnight, let alone a weekend, resulting in extra trips across the county to empty the thing out of hours.

Servers are relatively benign beasts compared to the unreliable monstrosities responsible for datacenter power and cooling. In the early days at Hinxton, we had regular power cuts. The generator would decide to cut in, but there were several seconds before it had spun up to speed. It turned out that the thresholds on the generator for detecting low power had been set back in the days when mains voltage was 240V. Now, it's 220V, and the official range is technically the same, but when all the local farms brought the cows in for milking there was enough of a drop in voltage that the generator thought there was going to be loss of power and kicked in unnecessarily.

Once, when it cut over, the bus bars stuck halfway, fused, and the generator shed had a pool of molten copper on the floor. It was some hours before that had cooled far enough for the electricians to safely enter the building.

When we got a new building, it was much bigger, so the generators they put in were sized for critical equipment only. Along comes the first big mains outage, and whoever had assigned the "critical" circuits had done so essentially at random. Specifically, our servers (which had UPS protection to tide them through blips long enough for the generators to come on stream) clearly weren't getting any juice. There was a lot of yelling, and we had to shut half the stuff down to buy time, before they connected up the right circuits.

Someone higher up decided we needed fire protection. So that was put in, wired up, the idea is that it kills all power and cooling. There's also a manual switch, which is put by the door so you can hit the big red button as you're legging it out of there. Too close to the door, as it turns out, on the wall exactly where the door handle goes when you open the door.

They installed a status panel on the wall outside too. The electrician wiring this up never satisfactorily explained how he managed to cut all power, although he must have almost had a heart attack when a posse of system administrators came flying out of offices demanding to know what had happened.

Numeracy isn't known as a builder's strong point. Common sense likewise. So the datacenter has an outside door, then a ramp up to an inner door. We believed in having a decent amount of room under the raised floor - enough to crawl under if necessary. So they put in the doorframe, but hadn't allowed for the raised floor. When we went for the initial inspection we had to crawl through the door, it was that low.

Even the final version wasn't really high enough. The old Sun 42U racks are actually quite short - the power supply at the base eats into the dimensions, so the external height is just over 42U and there was really only 36U available for servers. When we ordered our first Dell 42U rack, we had to take all the servers out, half dismantle it, lay it down, and it took six of us to wangle it through the dogleg and the door (avoiding the aforementioned big red emergency cut off button!). After that, we went out and ordered custom 39U racks to be sure they would fit.

When you run a network connection to a campus, you use multiple routes - so we had one fibre running to Cambridge, another to London. Unfortunately, as it turned out, there's about 100 yards from the main building ingress to the main road, and someone cheated and laid the two fibres next to each other. The was some building work one day, and it had carefully been explained where the hole should and shouldn't be dug. Mid-afternoon, Sun Net Manager (remember that?) went totally mental. We go to the window, and there's a large hole, a JCB, and the obligatory gaggle of men in hard hats staring into the hole. My boss says "They're not supposed to be digging there!" and, indeed, they had managed to dig straight through the cables. If it had been a clean cut it wouldn't have been so bad (there's always a little slack so they can splice the fibres in the event of a break), but they had managed to pull the fibres out completely so they had to be blown back from London/Cambridge before we got back on the air.

Tribblix on Vultr with Block Storage

I wrote a while back about installing Tribblix on Vultr using iPXE.

That all still works perfectly with the m23.2 release. And it's pretty quick too. It's even quicker on the new "High Frequency" servers, that appear to be a 3.8GHz Skylake chip and NVME storage.

One of the snags with many of the smaller cloud providers is that they don't necessarily have a huge choice of instance configurations. I'm not saying I necessarily want the billions of choices that AWS provide, but the instances are fairly one dimensional - CPU, memory, and storage aren't adjustable independently. This is one of the reasons I use AWS, because I can add storage without having to pay for extra CPU or memory.

One option here is to take a small instance, and add extra storage to it. That's what I do on AWS, having a small root volume and adding a large EBS volume to give me the space I need. This isn't - yet - all that commonly available on other providers.

Vultr do have it as an option, but at the moment it's only available in New York (NJ). Let's deploy an instance, and select NY/NJ as the location.

Scroll down and choose the "Upload ISO" tab, select the iPXE radio button, and put in the ipxe.txt script location for the version of Tribblix you want to install.

I've chosen the 1GB instance size, which is ample. Deploy that, and then connect to the console.

View the console and watch the install. This is really quick if you deploy in London, and isn't too bad elsewhere in Europe, as the Tribblix server I'm loading from is in London. Transferring stuff across the Atlantic takes a bit longer.

Then run the install. This is just ./live_install.sh -G c2t0d0

It will download a number of packages to finish the install (these are normally loaded off the ISO if you boot that, but for an iPXE install it pulls them over the network).

Reboot the server and it will boot normally off disk.

Go to the Block Storage tab, and Add some block storage. I'm going to add 50GB just to play with.

Now we need to connect it to our server.

This isn't quite as obvious as I would have liked. Click on the little pencil icon and you get to the "Manage Block Storage" page. Select the instance you want to attach it to, and hit Attach. This will restart the server, so make sure you're not doing anything.

The documentation talks about /dev/vdb and the like, which isn't the way we name our devices. As we're using vioblk, this comes up as c3t0d0 (the initial boot drive is c2t0d0).

We can create a pool

zpool create -O compression=lz4 store c3t0d0

This took a little longer than I expected to complete.

And I can do all the normal things you would expect with a zfs pool.

If you go to Block Storage and click the pencil to Manage it, the size is clickable. I clicked that, and changed the size to 64GB.

Like resizing an EBS volume on AWS, there doesn't seem to be a way to persuade illumos to rescan the devices to spot that the size has changed. You have to reboot.

Only reboot doesn't appear to be enough. It says on the Vultr console "You will need to restart your server via the control panel before changes will be visible." and it appears to be correct on that.

(This is effectively power-cycling it, which is presumably necessary to propagate the change through the whole stack properly.)

After that, the additional space is visible, as you can see from the extra 14G in the EXPANDSZ column:

 

And you can expand the pool using 'online -e'

zpool online -e store c3t0d0

This caused me a little bit of trouble. This appeared to generate an I/O error, lots of messages, and a hung console. I had to ssh in, clear the pool, and run a scrub, before things looked sane. Expanding the pool then worked and things look OK.

Generally, block device resize appears to be possible, but is still a bit rough round the edges.

The state of Tribblix, 2020

It's been a funny year, has 2020.

But amongst all this, work on Tribblix continues.

I released milestone 22 back in March. That was a fairly long time in the making, as the previous relase was 9 months earlier. Part of the reason for the lengthy delay was that there wasn't all that much reason for a new release - there are a lot of updated packages, but no big items. I guess the biggest thing is that the default gcc compiler and runtime went from gcc4 to gcc7. (In places, the gcc4 name continues.)

Milestone 23 was the next full release, in July. Things start to move again here - Tribblix fully transitioned from gcc4 to gcc7, as illumos is now a gcc7 build. I updated the MATE desktop, which was the start of moving from gtk2 to gtk3. There's a prettier boot banner, which allows a bit of custom branding.

There's a long-running effort to migrate from Python 2.x to 3.x. This is slow going - there are actually quite a lot of python modules and tools (and things that use python) that still show no sign of engaging with the Python 3 shift. But I'm gradually making sure that everything that can be version 3 is, and removing the python 2 pieces where possible. This is getting a bit more complicated - as of Python 3.8 I've switched from 32-bit to 64-bit. And now they're doing time-based releases there will be a version bump to navigate every year, just to add to the work.

Most of the Tribblix releases have been full upgrades from one version to the next. With the milestone 20 series, I had update releases, which allowed a shared stream of userland packages, while allowing illumos updates to take place. The same model is true of milestone 23 - update 1 came along in September.

With Milestone 23 update 1 we fixed the bhyve CVE. Other than normal updates, I added XView, which suits the retro theme and I've had quite a few people ask for.

Immediately after that (it was supposed to be in 23.1 but wasn't quite ready) came another major update: refreshing the X server stack.

When Tribblix was created, I didn't have the resources to build everything from scratch straight away, so "borrowed" a few components from OpenIndiana (initially 151a8, then 151a9) just to make sure I had enough bits to provide a complete OS. Many of the isolated components were replaced fairly quickly over time, but the X11 stack was the big holdout. It was finally time to build Xorg and the drivers myself. It wasn't too difficult, but to be honest I have no real way to test most of it. So that will all be present in 23.2.

One reason for doing this - and my hand was forced a little here - is that I've also updated Xfce from 4.12 to 4.14. That's also a gtk2 to gtk3 switch, but Xfce 4.14 simply didn't work on the old Xorg I had before.

Something else I've put together - and these are all gtk3 based - is a lightweight desktop, using abiword, geany, gnumeric, grisbi, imagination, and netsurf. You still need a file manager to round out the set, and I really haven't found anything that's lightweight and builds successfully, so at the moment this is really an adjunct to MATE or Xfce.

Alongside all this I've been working on keeping Java OpenJDK working on illumos. They ripped out Solaris support early in the year, but I've been able to put that back. The real killer here was Studio support, and we don't want that anyway (it's not open source, and the binaries no longer run). There are other unix-like variants supported by Java, running on the x86 architecture with a gcc toolchain, just like us, so it shouldn't be that much of a mountain to climb.

Support for SPARC is currently slightly on the back burner, partly because the big changes mentioned above aren't really relevant for SPARC, partly due to less time, partly due to the weather - running SPARC boxes in the home office tends to be more of a winter than a summer pursuit, due to the heat.

Solaris 10 zones on Tribblix

One of the interesting capabilities of Solaris zones was the ability to run older versions of Solaris than that in the global zone. Marketing managed to mangle this into Containers, and it was supported for Solaris 8 and Solaris 9.

I used this extensively on one project, to lift a whole datacenter of ancient (yes, really ancient) Sun servers into zones on a couple of T5240s. Worked great. (We had to get an E450 out of the dumpster and build it specially to get a Solaris 2.6 system, however.)

Solaris 11 and illumos have dropped the Solaris 8 and 9 legacy containers, but have a Solaris 10 zone brand. On Tribblix, this can be installed with

zap install TRIBsys-zones-brand-s10

(If you're on an IPS based distro, the package name is system/zones/brand/s10.)

Installing an s10 branded zone is just like a regular zone, but you need a Solaris 10 image to install from. You could tar up a legacy system, or create a new image from the install media.

There are certain requirements for the software in the image and on the host. If, on a Solaris 10 system, you look in the directory /usr/lib/brand/solaris10, you might see a couple of files called 0 and 1. They have a little bit of text in them for explanation, but these are emulation compatibility feature flags. If you look at the illumos source, you can see them listed too. This is a basic versioning system - the host running the global zone needs to support all the features of the software in the zone. Fortunately that feature list hasn't changed, so we're good.

For the image, the s10 brand checks the SUNWcakr package and needs it to be patched to a minimum level. In practice, this means that anything S10U8 or newer will work.

During zone installation, there's some sanity checking. It turns out the installer is looking for /var/sadm/system/admin/INST_RELEASE and gives up if it can't find it. I had to manually create that file:

cat > ......./var/sadm/system/admin/INST_RELEASE
OS=Solaris
VERSION=10
REV=0
^D

There's also some processing of the zone image that gets done as part of the zone installation. And if that processing fails, then the zone install will fail too.

It has a hardcoded list of safe_dir entries for things it needs to replace. If you don't have those, it simply fails, so you need to add a bunch of packages just to make it happy. Specifically: autofs, zfs, and ipf.

The postprocessing also runs sys-unconfig, so you need to make sure that's present, from SUNWadmap.

Enough chatter. If I have my Solaris 10 media under /mnt1, and want to create an image under /export/S10, then it's going to look like this:

cd /mnt1/Solaris_10/Product
# SUNWCcs SUNWcar SUNWcakr SUNWkvm
pkgadd -d . -R /export/S10 SUNWcsr
pkgadd -d . -R /export/S10 SUNWckr
pkgadd -d . -R /export/S10 SUNWcnetr
pkgadd -d . -R /export/S10 SUNWcsd
pkgadd -d . -R /export/S10 SUNWcsl
pkgadd -d . -R /export/S10 SUNWcsu
pkgadd -d . -R /export/S10 SUNWcar.i
pkgadd -d . -R /export/S10 SUNWcakr.i
pkgadd -d . -R /export/S10 SUNWkvm.i
pkgadd -d . -R /export/S10 SUNWcslr
# SUNWCfmd
pkgadd -d . -R /export/S10 SUNWfmdr
pkgadd -d . -R /export/S10 SUNWfmd
# SUNWClexpt
pkgadd -d . -R /export/S10 SUNWlexpt
# SUNWCpicl
pkgadd -d . -R /export/S10 SUNWpiclr
pkgadd -d . -R /export/S10 SUNWpiclu
# SUNWCopenssl SUNWhea (but not man)
pkgadd -d . -R /export/S10 SUNWopensslr
pkgadd -d . -R /export/S10 SUNWopenssl-libraries
pkgadd -d . -R /export/S10 SUNWhea
pkgadd -d . -R /export/S10 SUNWopenssl-include
pkgadd -d . -R /export/S10 SUNWopenssl-commands
# SUNWCpkgcmds SUNWwbsup
pkgadd -d . -R /export/S10 SUNWproduct-registry-root
pkgadd -d . -R /export/S10 SUNWwsr2
pkgadd -d . -R /export/S10 SUNWpkgcmdsr
pkgadd -d . -R /export/S10 SUNWwbsup
pkgadd -d . -R /export/S10 SUNWpkgcmdsu
# SUNWCpr SUNWCtls
pkgadd -d . -R /export/S10 SUNWpr
pkgadd -d . -R /export/S10 SUNWtls
pkgadd -d . -R /export/S10 SUNWjss
# SUNWCfwshl
pkgadd -d . -R /export/S10 SUNWbash
pkgadd -d . -R /export/S10 SUNWtcsh
pkgadd -d . -R /export/S10 SUNWzsh
# perl
pkgadd -d . -R /export/S10 SUNWperl584core
pkgadd -d . -R /export/S10 SUNWperl584usr
# SUNWCptoo SUNWtecla SUNWesu SUNWtoo
pkgadd -d . -R /export/S10 SUNWtecla
pkgadd -d . -R /export/S10 SUNWbtool
pkgadd -d . -R /export/S10 SUNWesu
pkgadd -d . -R /export/S10 SUNWcpp
pkgadd -d . -R /export/S10 SUNWtoo
pkgadd -d . -R /export/S10 SUNWlibmr
pkgadd -d . -R /export/S10 SUNWlibm
pkgadd -d . -R /export/S10 SUNWlibmsr
pkgadd -d . -R /export/S10 SUNWlibms
pkgadd -d . -R /export/S10 SUNWsprot
# SUNWCfwcmp SUNWlibC
pkgadd -d . -R /export/S10 SUNWlibC
pkgadd -d . -R /export/S10 SUNWbzip
pkgadd -d . -R /export/S10 SUNWgzip
pkgadd -d . -R /export/S10 SUNWzip
pkgadd -d . -R /export/S10 SUNWzlib
# release and sys-unconfig
pkgadd -d . -R /export/S10 SUNWsolnm
pkgadd -d . -R /export/S10 SUNWadmr
pkgadd -d . -R /export/S10 SUNWadmlib-sysid
pkgadd -d . -R /export/S10 SUNWadmap
# autofs is needed for validation
pkgadd -d . -R /export/S10 SUNWatfsr
pkgadd -d . -R /export/S10 SUNWatfsu
# ditto zfs
pkgadd -d . -R /export/S10 SUNWlxml
pkgadd -d . -R /export/S10 SUNWsmapi
pkgadd -d . -R /export/S10 SUNWzfskr
pkgadd -d . -R /export/S10 SUNWzfsr
pkgadd -d . -R /export/S10 SUNWzfsu
# ditto ipf
pkgadd -d . -R /export/S10 SUNWipfr
pkgadd -d . -R /export/S10 SUNWipfu
# It's about 235M at this point

If you cd to /export/S10, make sure the INST_RELEASE file is there with the correct contents (see above) and then tar up what you have, you can feed that tarball to the zone installation and it should work.

If you look at documentation for s10 zones on Solaris 11, you'll see a -c option. We don't have that, but you could drop a sysidcfg file into /etc/sysidcfg in the zone so it will configure itself at boot. It will look something like this:

system_locale=C
terminal=xterm
# shared
network_interface=primary {
    hostname=${ZNAME}
}
# exclusive
network_interface=primary {
        hostname=${ZNAME}
    ip_address=192.168.0.101
        netmask=255.255.255.0
        default_route=NONE
        protocol_ipv6=no
}
security_policy=NONE
name_service=DNS {
    domain_name=special.example.com
    name_server=192.168.112.3,1.2.3.4
}
nfs4_domain=dynamic
timezone=${TZ:=UTC}
root_password=jhDYXBAnWJKr2

If you're using Tribblix, most of the zone creation is simplified, and it will be:

zap create-zone -t s10 -z s10-test4 -I /tmp/S10.tar -i 192.168.0.224

I haven't tried this on SPARC (my use case is building Java and Node.JS), but it ought to be exactly the same modulo trivial chagnes to package names.