Building illumos-gate on AWS

Having talked about running Tribblix on AWS, one of the things that would be quite neat would be to be able to build illumos-gate.

This is interesting because it's a relatively involved process, and might require proper resources - it's not really possible to build illumos inside VirtualBox, for instance, and many laptops don't run illumos terribly well. So it's hard for the average user to put together a decent - most likely dedicated - rig capable of building or developing illumos, which is clearly a barrier to contribution.

Here's how anyone can build illumos, using Tribblix.

Build yourself an EC2 instance as documented here, with 2 changes:

1. The instance type should be m4.large or bigger - m4.xlarge or c4.xlarge would be better. The bigger the instance, the quicker the build, but m4.large is pretty much the minimum size.
2. Attach an EBS volume to the instance, at least 8G in size. If you want to do multiple builds, or do lint or debug builds, then it has to be larger. I attach the volume as /dev/sdf, which is assumed below. (You could keep the volume around to persist the data, of course.)

Once booted, log in as root. You then need to set up the zfs pool (the disk showing up as c2t5d0 below matches the /dev/sdf attachment point) and create a couple of file systems that can be used to host the build zone and store the build.

zpool create storage c2t5d0
zfs set compression=lz4 storage
zfs destroy rpool/export/home
zfs create -o mountpoint=/export/home storage/home
zfs create -o mountpoint=/export/zones storage/zones

You should then do an update to ensure packages are up to date, and install the develop overlay to get you some useful tools.

zap refresh
zap update-overlay -a
zap install-overlay develop

Then create a user, which you're going to use to do the build. For me, that is:

groupadd -g 10000 it
useradd -g it -u 11730 -c "Peter Tribble" -s /bin/tcsh \
  -d /export/home/ptribble ptribble
mkdir -p /export/home/ptribble
chown -hR ptribble:it /export/home/ptribble
passwd ptribble

Then create a build zone. It has an IP address, just pick any unused private address (I simply use the address above that of the global zone, which you can get with ifconfig or from the AWS console - note that it's the private address, not the public IP that you ssh to).

zap create-zone -z illumos-build -t whole \
  -i 172.xxx.xxx.xxx -o develop \
  -O java -O illumos-build -U ptribble

What does this do? It creates a new zone, called illumos-build. It's a whole root zone, with its own exclusive set of file systems. The IP address is 172.xxx.xxx.xxxx. The develop overlay is installed (in this case, copied from the global zone); the java and illumos-build overlays are added to this new zone (note the upper-case -O here). Finally, the user account ptribble is shared with the zone.

Give that a few seconds to boot and log in to it, then a couple of tweaks that are necessary for illumos to build without errors.

zlogin illumos-build
rm /usr/bin/cpp
cd /usr/bin ; ln -s ../gnu/bin/xgettext gxgettext

Now log out and log back in to the instance as your new user. We're going to create somewhere to store the files, and check out the source code.

mkdir Illumos
cd Illumos
git clone git://github.com/illumos/illumos-gate.git
wget -c \
  https://download.smartos.org/pub/build/illumos/on-closed-bins.i386.tar.bz2 \
  https://download.smartos.org/pub/build/illumos/on-closed-bins-nd.i386.tar.bz2

Now we set up the build.

cd illumos-gate
bzcat ../on-closed-bins.i386.tar.bz2 | tar xf -
bzcat ../on-closed-bins-nd.i386.tar.bz2 | tar xf -
cp usr/src/tools/scripts/nightly.sh .
chmod +x nightly.sh

There are two more files we need. Go to the tribblix-build repo and look in the illumos directory there. Grab one of the illumos.sh files from there and put it into your illumos-gate directory with the name illumos.sh. If you need to change how the build is done, this is the file to edit (but start from one of those files so you get one appropriate for Tribblix as the host). Also, grab Makefile.auditrecord and use it to replace usr/src/cmd/auditrecord/Makefile.

Now log in to the zone and start the build.

pfexec zlogin -l ptribble illumos-build
cd Illumos/illumos-gate
time ./nightly.sh illumos.sh

On an m4.xlarge instance, this took me just under 75 minutes. Look in the log directory and check that the mail_msg looks clean without errors, and you'll have the built files in the proto directory and an IPS repo under packages.

For more behind the scenes details on the illumos build process itself, look at the how to build illumos page.