Monday, November 21, 2022

A decade of Tribblix

I seem to have just missed the anniversary, but it turns out that Tribblix has existed for slightly over a decade.

The initial blog post on Building Tribblix was published on October 24th, 2012. But the ISO image (milestone 0) was October 21st, and it looks like the packages were built on October 4th. So there's a bit of uncertainty about the actual date, and I had been playing around with some of the bits and pieces for a while before that.

There have been a lot of releases. We're now on Milestone 28, but there have been several update releases along the way, so I make it 42 distinct releases in total. That doesn't include the LX-enabled OmniTribblix variant (there have been 20 of those by the way).

The focus (given hardware availability) has been x86, naturally. But the SPARC version has seen occasional bursts of life. Now I have a decent build system, it's catching up. Will there be an ARM version? Who knows...

Over the years there have been some notable highlights. It took a few releases to become fully self-hosting; package management had to be rebuilt; LibreOffice was ported; Xfce and MATE added as fully functional desktop offerings (with a host of others); a whole family of zones, including reimplementing the traditional sparse root; made available on clouds like AWS and Digital Ocean; network install via iPXE; huge numbers of packages (it's never-ending churn); and maintaining Java by default.

And it's been (mostly) fun. Here's to the next 10 years!

Sunday, November 20, 2022

TREASURE - The Remote Execution and Access Service Users Really Enjoy

Many, many years ago I worked on a prototype of a software ecosystem I called TREASURE - The Remote Execution and Access Service Users Really Enjoy.

At the time, I was running the infrastructure and application behind an international genomics service. The idea was that we could centrally manage all the software and data for genomic analysis, provide high-end compute and storage capability, and amortize the cost across 20,000 academics so that individual researchers didn't have to maintain it all individually.

Originally, access was via telnet (I did say it was a long time ago). After a while we enabled X11, so that graphical application would work (running X11 directly across the internet was fun).

Then along came the web. One of my interesting projects was to write a web server that would run with the privileges of the authenticated user. (This was before apache came along, by the way!) And clearly a web browser might be able to provide a more user-friendly and universal interface than a telnet prompt.

We added VNC as well (it came out of Cambridge and we were  aware of it well before it became public), so that users could view graphical applications more easily. This had a couple of advantages - all the hard work and complexity was at our end, where we had control, and X11 is quite latency sensitive so performance improved.

But ultimately what I wanted to do was to run the GUI on the user's machine, wit access to the user's files. Remember that the GUI is then not running where the software, genome databases, and all the compute power are located.

Hence the Remote Execution part of TREASURE - what we wanted was a system that would call across to a remote service to do the work, and return the result to the user. And the Access part was about making it accessible and transparent, which would lead to a working environment that people would enjoy using.

Ultimately, the core of TREASURE was originally a local GUI that knew how to run applications. Written in Java, it would therefore run on pretty much any client (and we had users with all sorts of Unix workstations in addition to Windows making inroads). The clever bit was to replace the java Runtime.getRuntime().exec() calls that ran applications locally with some form of remote procedure call. Being of its time, this might involve CORBA, RMI, SOAP, or JAX-WS with data marshalled as XML. In fact, I implemented pretty much every remote call mechanism available (and this did in fact come in useful as other places did make available some services using pretty random protocols). And then of course there's the server side which was effectively a CGI script.

The other key part was to work out which files needed to be sent across. Sometimes it was obvious (it's a GUI, the user has selected a file to analyse), but sometimes we needed to send across auxiliary files as well. And on the server side it ran in a little sandbox so you knew what output files had bee generated so you could return those.

Effectively, this was a production form of serverless computing running over 20 years ago. Only we called it GRID computing back then.

Another interesting feature of the architecture was the TREASURE CHEST, which was a source of applications. There were literally hundreds of possible applications you could run, and many more if you included interfaces to other providers. So rather than write all those into the app, there was a plugin system where you could download a jar file and run it as a plugin, and the TREASURE CHEST was where you could find these application. Effectively an app store, in modern terminology.

Sadly the department got closed down due to political incompetence, so the project never really got beyond the prototype stage. And while I still have bits and pieces of code, I don't appear to have a copy of the whole thing. A lot of the components would need to be replaced, but the overall concept is still sound.

Tuesday, November 15, 2022

Tribblix for SPARC m25.1

Following hot on the heels of the Tribblix Milestone 22 ISO for SPARC, it's possible to upgrade that to a newer version. The new version that's available is m25.1.

(If the available versions look a bit random, that's because they are. Not every release on x86 was built for SPARC, and not all of the ones that were actually worked properly. So we have what we have.)

The major jump, aside from the underlying OS, in m25.1 for SPARC is that it brings in gcc7 (for applications, illumos itself is still built with gcc4), and generally there's a bunch of more modern applications available.

To upgrade m22 to m25.1 is a manual process. This is because there are steps that are necessary, and if you don't follow them exactly the system won't boot.

The underlying cause here of the various problems in this process is that it's a big jump from m22 to m25.1 and you will hit bugs in the upgrade process that have been fixed in intermediate releases.

First, take a note of the current BE, eg tribblix. You might need it later if things go bad and you need to reboot into the current (hopefully working) release.

You can manually add available versions for upgrade with the following trick (this is just one line, despite how it might be formatted):

echo "m25.1|http://pkgs.tribblix.org/release-m25.1.sparc/TRIBzap.0.0.25.1.zap|Tribblix m25.1" >> /etc/zap/version.list

and check that's visible with

zap upgrade list

and then start the upgrade with

zap upgrade m25.1

Do not activate or reboot yet!

You MUST do the following:

beadm mount m25.1 /a
zap install -C /a TRIBshell-ksh93
pkgadm sync -q -R /a
beadm umount m25.1

and then you should be safe to reboot:

beadm activate m25.1
init 6

If it doesn't come back, you can boot into the previous release (that you took the name of earlier, remember) from the ok prompt

boot -Z rpool/ROOT/tribblix

Once you're up and running m25.1 it's time to clean up.

zap refresh

and then remove some of the old opensxce packages

zap uninstall \
SUNWfont-xorg-core \
SUNWfont-xorg-iso8859-1 \
SUNWttf-dejavu \
SUNWxorg-clientlibs \
SUNWxorg-xkb \
SUNWxvnc \
SUNWxwcft \
SUNWxwfsw \
SUNWxwice \
SUNWxwinc \
SUNWxwopt \
SUNWxwxft \
SUNWxwrtl \
SUNWxwplr \
SUNWxwplt

and then bring packages up to current

zap update-overlay -a

and this should give you a system that's in a workable state, and roughly matching my active SPARC environment.

Monday, November 14, 2022

Tribblix for SPARC m22 ISO now available

I've made available a newer ISO image for Tribblix on SPARC.

This is an m22 ISO. So it's actually relatively old compared to the mainstream x86 release.

I actually had a number of random SPARC ISO images, but for a while I've had no way of testing any of them. (And many of the problems with the SPARC ISOs in general is because I had no real way of testing them properly.)

Arrive a newish T4-1 (thanks Andy!), and I can now trivially create an LDOM, assign it a zvol for a root disk and a ISO image to boot from, and testing is trivial again. And while some of the ISO images I have are clearly so broken as to not be worth considering, the m22 version looks pretty reasonable.

In terms of available application packages, it exactly matches the old m20 release. I do have newer packages on some of my test systems, but they are built with a newer gcc and so need a proper upgrade path. But that's going to be easier now too.

There is a minor error on the m22 ISO, in that the xz package shipped appears to be wrong. To fix, simply

zap install TRIBcompress-xz

and to update to the latest available applications (the ISO is early 2020, the repo is middle of 2021)

zap refresh
zap update TRIBlib-security-openssl
zap update-overlay -a

The reason for updating openssl on its own is that a number of applications are compiled against openssl 1.1.1, so you need to be sure that gets updated first.

Next step is to push on to something newer.