Sunday, April 04, 2021

Running Tribblix on Digital Ocean

A relatively recent feature offered by Digital Ocean is the ability to deploy your own custom image. So, can I deploy a Tribblix image to Digital Ocean?

Short answer: Yes!

For the process, read on.

I'm using Bhyve to create the image. This is a slight variation on the Installing Tribblix in Bhyve on Tribblix procedure.

The basic process looks like this:

  • Boot Tribblix in Bhyve
  • Install it
  • Tweak the installed image for Digital Ocea
  • Copy the ZFS volume to Digital Ocean

The first variation on the previous install is that I make the ZFS volume a bit smaller. We can resize it when it's deployed, so we don't need to make it too big. A 4G volume is fine; any smaller and there won't be any space for our swap partition. It doesn't actually matter too much, as we compress the image anyway

zap create-zone -t bhyve -z bhyve1 \
-x  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 4G

Install as before,

./ -G c1t0d0

remove the cdrom and reboot as before.

We now need to set up networking. We need to do this on a temporary basis, as we don't want any of these network configurations to carry over to the installed image. So I temporarily disable nwam, and manually bring up a working network. When the image boots on Digital Ocean, all this configuration will have been forgotten and it will bring up nwam as normal.

So run the following in the newly installed guest:

svcadm disable -t network/physical:nwam
ifconfig vioif0 plumb
ifconfig vioif0 up
ifconfig vioif0 inet
route add net default
echo "nameserver"  > /etc/resolv.conf

Now we need to tweak the image. At some later point this will all be integrated into the installer so it will just work. But for now, we'll start by applying any updates:

zap refresh
zap update-overlay -a

Now for the little tweak. I'm going to add a metadata service that will run at boot on Digital Ocean and do the sort of things that cloud-init would do. Fortunately, there's one for illumos, and it's packaged for Tribblix, so install it:

zap install TRIBmetadata-agent

If you look with svcs you'll see that it's offline. That's not a problem (it's because we've got a temporary manual network setup) - once we boot properly on Digital Ocean we'll have nwam running and the metadata service will run just fine.

We can tidy up and save a bit of space:

zap clean-cache -a

and shut down the zone (and the newly installed instance of Tribblix):

zoneadm -z bhyve1 halt

What we want is a raw image. So all we do is dd the ZFS volume to a file.

dd if=/dev/zvol/rdsk/rpool/bhyve1_bhvol0 \
of=/var/tmp/tribblix-do-m24.1.img bs=1048576

That's a 4G file, the size of the volume. As it's stored on ZFS, and ZFS compression is on, it will actually consume a lot less space as the image is mostly empty. But what we don't want to do is upload 4G of empty space. So we can compress it:

gzip -9 /var/tmp/tribblix-do-m24.1.img

(it ends up as about 300M), or you could use bzip2, I think.

There are two options when you upload the image to Digital Ocean - you can either do a direct upload through the browser, or you can give it a URL where the image can be found and get Digital Ocean to pull it from there. I found it much easier to scp the image up to an existing webserver and get Digital Ocean to grab it, as I don't trust browsers to behave well.

Log in to Digital Ocean.

Select 'Images' from the left hand menu
Tab to 'Custom Images'
Import via URL
[Insert the URL where you've stashed the file]

You then get a dialog

Name - tribblix-do-m24.1.img.gz
Distribution - unknown (it's not on the list)
Region - London

Obviously, for me, London is conveniently local. You get warned there will be a charge, and it shows as pending for a few minutes.

Then it pops up 'your image is ready to use'.

To the right of the image in the list is a 'More' dropdown menu from which you can start a droplet. So off we go.

It selects a pretty hefty instance type by default. Reset that to the very cheapest. Choose the ssh key you're going to use, pick a useful (and shorter) hostname, and Create Droplet.

Don't bother with block storage. That's exposed by virtio-scsi, which illumos doesn't yet support.

It'll take a few moments to create the droplet, and once it's ready you'll see the IP address.

At this point, if everything has worked, you should be able to ssh in as root with the ssh key you chose. Chances are that doesn't quite work yet. If it doesn't, simply ssh in as jack, from where you can su to root.

(Remember, jack is on the live ISO, and we haven't deleted it. During experiments I tend not to, to give myself a way in if things don't work right. A proper production image would have the jack user removed and password login for root disabled.)

If the metadata service hasn't run properly (it will resize the ZFS pool, change the hostname, and add the correct key to ssh in as root) then you can restart the metadata service:

svcadm restart metadata

Now ssh to root works, the hostname is set, and the zfs pool has been expanded to the full size.

None of the above is excessively specific to Tribblix, the same general process will work for any of the illumos distributions. (Although you may have to build and install the metadata service yourself.)

Installing Tribblix in Bhyve on Tribblix

One of the big new features recently added to illumos is the Bhyve hypervisor. Rather that the shared-kernel application-level virtualization offered by zones, think of something like VirtualBox, KVM, or Qemu.

One of the things that I am using Bhyve for is to test the Tribblix ISO images and the installer. This allowed me to shrink the installer footprint slightly in recent releases (and showed that one of the tricks I tried wasn't going to work).

Using Bhyve requires a fairly modern system (my own is a little old, and has an Intel Core i7, and works fine). The instructions here also need you to be running current Tribblix, m24.1 or newer.

So, how do I install Tribblix in Bhyve on Tribblix? Most of the following needs to be done as root.

First, make sure that you have the software installed:

zap install-overlay bhyve

I'm doing this on my desktop, so I'm using VNC to connect.

zap install-overlay retro-desktop

You need the current ISO. Again, this has to be m24.1 or newer. Correct the path below to match wherever you've downloaded it to.

While you don't actually have to run bhyve in a zone, that's the standard and most convenient way. So, I use zap to create a zone:

zap create-zone -t bhyve -z bhyve1 \
-x  \
-I /var/tmp/tribblix-0m24.1.iso \
-V 8G

Let's run through these. The -t flag says to create a bhyve zone, -z gives it a name, -x gives it an (exclusive) IP address, -I tells it where the ISO image is, and -V tells it to create a ZFS volume of the given size. The other argument that may be of interest is -m, which allows you to set the amount of memory allocated; it defaults to 1024M which is fine for Tribblix.

Then you need to be able to connect to the instance. I'm going to use VNC to connect to the console. We use socat to wire up the bhyve socket in the zone to a network port in the global zone that we can connect to.

socat TCP-LISTEN:5905,reuseaddr,fork UNIX-CONNECT:/export/zones/bhyve1/root/tmp/vm.vnc

You may need to modify the zone name in the path, and you can choose the TCP port (for VNC, it's offset by 5900, so the 5905 is :5 in VNC-speak).




Then, as yourself, you can start a VNC client

vncviewer :5

If that can't connect, then one possibility is that bhyve can't start because it doesn't have enough memory. One little trick to make sure there's enough headroom is to create a file in /tmp and then delete it immediately.

mkfile 1200m /tmp/1200m ; rm /tmp/1200m

You can then log in to the live environment. And do an install like so:

./ -G c1t0d0





and the install will carry on






If you want to run this, you have to avoid booting off the CD. The way to do this is to remove the cdrom from the zone specification before you reboot the guest, like so:

zonecfg -z bhyve1 remove attr name=cdrom
zonecfg -z bhyve1 remove fs dir=/var/tmp/tribblix-0m24.1.iso special=/var/tmp/tribblix-0m24.1.iso
zoneadm -z bhyve1 reboot

Obviously, adjusted for the zone name and the ISO file name. I need to teach zap how to do this properly. And then start socat again (if you stopped it) and reconnect using VNC.

You may need to fiddle the networking in the installed guest. The exclusive-ip setting here will force the guest to use the given address. That's unlikely to work terribly well, as the chances of allocating the same address that DHCP would hand out are pretty remote. So to get it on the network you may have to fiddle in the guest by hand.

svcadm disable network/physical:nwam
echo "" > /etc/hostname.vioif0
echo "" > /etc/defaultrouter
svcadm enable network/physical:default

and populate /etc/resolv.conf with something relevant so that dns works

echo "nameserver" > /etc/resolv.conf







Once you're done, you can destroy the zone with

zap destroy-zone -z bhyve1

which will also destroy the ZFS volume it was using.

While the above refers to installing Tribblix inside Bhyve, the same general procedure ought to work to install other illumos distributions.