A couple of times recently I've had interviewers ask me to quickly sketch out the design for a web scale architecture. Of course, being a scientist by training the first thing I did was to work out what sort of system requirements we're looking at, to see what sort of scale we might really need.
In both cases even my initial extreme estimate, just rounding everything up, didn't indicate a scaling problem. Most sites aren't Facebook or Google, they see limited use by a fraction of the population. The point here is that while web scale sites exist, they are the exception rather than the rule, so why does everyone think they have to go to the complexity and expense of architecting a "web scale" solution?
To set this into perspective, assume you want to track everyone in the UK's viewing habits. If everyone watches 10 programmes per day and channel-hops 3 times for each programme, and there are 30 million viewers, then that's 1 billion data points per day or 10,000 per second. Each is small, so at 100 bytes each that's 100GB/day or ~10megabits/s. So, you're still talking single server. You can hold a week's data in RAM, a year on disk.
And most businesses don't need anything like that level of traffic.
Part of the problem is that most implementations are horrifically inefficient. The site itself may be inefficient - you know the ones that have hundreds of assets on each page, multi-megabyte page weight, widgets all over, that take forever to load and look awful - if their customers bother waiting long enough. The software implementation behind the site is almost certainly inefficient (and is probably trying to do lots of stupid things it shouldn't as well).
Another trend fueling this is the "army of squirrels" approach to architecture. Rather than design an architecture that is correctly sized to do what you need, it seems all too common to simply splatter everything across a lot of little boxes. (Perhaps because the application is so badly designed it has cripplingly limited scalability so you need to run many instances.) Of course, all you've done here is simply multiplied your scaling problem, not solved it.
As an example, see this article Scalability! But at what COST? I especially like the quote that Big data systems may scale well, but this can often be just because they introduce a lot of overhead.
Don't underestimate the psychological driver, either. A lot of people want to be seen as operating at "web scale" or with "Big Data", either to make themselves or their company look good, to pad their own CV, or to appeal to unsophisticated potential employees.
There are problems that truly require web scale, but for the rest it's an ego trip combined with inefficient applications on badly designed architectures.
Sunday, November 15, 2015
Thursday, November 12, 2015
On the early web
I was browsing around, as one does, when I came across a list of early websites. Or should I say, a seriously incomplete list of web servers of the time.
This was November 1992, and I had been running a web server at the Institute of Astronomy in Cambridge for some time. That wasn't the only technology we were using at the time, of course - there was Gopher, the emergent Hyper-G, WAIS, ftp, fsp, USENET, and a few others that never made the cut.
Going back a bit further in time, about a year earlier, is an email regarding internet connectivity in Cambridge. I vaguely remember this - I had just arrived at the IoA at the time and was probably making rather a nuisance of myself, having come back from Canada where the internet was already a thing.
I can't remember exactly when we started playing with the web proper, but it would have been some time about Easter 1992. As the above email indicates, 1991 saw the department having the staggering bandwidth of 64k/s, and I think it would have taken the promised network upgrade for us to start advertising our site.
Graphical browsers came quite late - people might think of Mosaic (which you can still run if you like), but to start with we just had the CERN line mode browser, and things like Viola. Around this time there were other graphical browsers - there was one in the Andrew system, as I recall, and chimera was aimed at the lightweight end of the scale.
Initially we ran the CERN web server, but it was awful - it burnt seconds of cpu time to deliver every page, and as soon as the NCSA server came out we switched to that, and the old Sun 630MP that hosted all this was much the happier for it. (That was the machine called cast0 in the above email - the name there got burnt into the site URL, it took a while for people to get used to the idea of adding functional aliases to DNS.)
With the range of new tools becoming available, it wasn't entirely obvious which technologies would survive and prosper.
With my academic background I was initially very much against the completely unstructured web, preferring properly structured and indexed technologies. In fact, one of the things I remember saying at the time, as the number of sites started to grow, was "How on earth are people going to be able to find stuff?". Hm. Missed business opportunity there!
Although I have to say that even with search engines, actually finding stuff on the web now is a total lottery - Google have made a lot of money along the way, though. One thing I miss, again from the early days of the web (although we're talking later in the 90s now) is the presence of properly curated and well maintained indices of web content.
Another concern I had about the web was that, basically, any idiot could create a web page, leading to most of the content being complete and utter garbage (both in terms of what it contained and how it was coded). I think I got the results of that one dead right, but it failed to account for the huge growth that the democratization of the web allowed.
After a couple of years the web was starting to emerge as a clear front runner. OK, there were only a couple of thousand sites in total at this point (I think that up to the first thousand or so I had visited every single one), and the concept was only just starting to become known to the wider public.
One of the last things I did at the IoA, when I left in May 1994, was to set up all the computers in the building running Mosaic, with it looping through all the pages on a local website showcasing some glorious astronomical images, all for the departmental open day. This was probably the first time many of the visitors had come across the web, and the Wow factor was incredible.
This was November 1992, and I had been running a web server at the Institute of Astronomy in Cambridge for some time. That wasn't the only technology we were using at the time, of course - there was Gopher, the emergent Hyper-G, WAIS, ftp, fsp, USENET, and a few others that never made the cut.
Going back a bit further in time, about a year earlier, is an email regarding internet connectivity in Cambridge. I vaguely remember this - I had just arrived at the IoA at the time and was probably making rather a nuisance of myself, having come back from Canada where the internet was already a thing.
I can't remember exactly when we started playing with the web proper, but it would have been some time about Easter 1992. As the above email indicates, 1991 saw the department having the staggering bandwidth of 64k/s, and I think it would have taken the promised network upgrade for us to start advertising our site.
Graphical browsers came quite late - people might think of Mosaic (which you can still run if you like), but to start with we just had the CERN line mode browser, and things like Viola. Around this time there were other graphical browsers - there was one in the Andrew system, as I recall, and chimera was aimed at the lightweight end of the scale.
Initially we ran the CERN web server, but it was awful - it burnt seconds of cpu time to deliver every page, and as soon as the NCSA server came out we switched to that, and the old Sun 630MP that hosted all this was much the happier for it. (That was the machine called cast0 in the above email - the name there got burnt into the site URL, it took a while for people to get used to the idea of adding functional aliases to DNS.)
With the range of new tools becoming available, it wasn't entirely obvious which technologies would survive and prosper.
With my academic background I was initially very much against the completely unstructured web, preferring properly structured and indexed technologies. In fact, one of the things I remember saying at the time, as the number of sites started to grow, was "How on earth are people going to be able to find stuff?". Hm. Missed business opportunity there!
Although I have to say that even with search engines, actually finding stuff on the web now is a total lottery - Google have made a lot of money along the way, though. One thing I miss, again from the early days of the web (although we're talking later in the 90s now) is the presence of properly curated and well maintained indices of web content.
Another concern I had about the web was that, basically, any idiot could create a web page, leading to most of the content being complete and utter garbage (both in terms of what it contained and how it was coded). I think I got the results of that one dead right, but it failed to account for the huge growth that the democratization of the web allowed.
After a couple of years the web was starting to emerge as a clear front runner. OK, there were only a couple of thousand sites in total at this point (I think that up to the first thousand or so I had visited every single one), and the concept was only just starting to become known to the wider public.
One of the last things I did at the IoA, when I left in May 1994, was to set up all the computers in the building running Mosaic, with it looping through all the pages on a local website showcasing some glorious astronomical images, all for the departmental open day. This was probably the first time many of the visitors had come across the web, and the Wow factor was incredible.
Wednesday, October 21, 2015
Tribblix Turns Three
It's a little hard to put a fixed date on when I started work on Tribblix.
The idea - of building a custom distribution - had been floating around my mind in the latter days of OpenSolaris, and I registered the domain back in 2010.
While various bits of exploratory work had been going on in the meantime, it wasn't until the autumn of 2012 that serious development started. Eventually, after a significant number of attempts, I was able to produce a functional ISO image. That was:
The first blog post was a few days later, but I'm going to put October 21st as the real date of birth.
Which means that Tribblix is 3 years old today!
In that time it's gone from a simple toy to a fully fledged distribution, most of the original targets I set myself have been met, it's been my primary computing environment for a while, it's proving useful as a platform for interesting experiments, and I'm looking forward to taking it even further in the next few years.
The idea - of building a custom distribution - had been floating around my mind in the latter days of OpenSolaris, and I registered the domain back in 2010.
While various bits of exploratory work had been going on in the meantime, it wasn't until the autumn of 2012 that serious development started. Eventually, after a significant number of attempts, I was able to produce a functional ISO image. That was:
-rw-r--r-- 1 ptribble 493049856 Oct 21 2012 tribblix-0m0.iso
The first blog post was a few days later, but I'm going to put October 21st as the real date of birth.
Which means that Tribblix is 3 years old today!
In that time it's gone from a simple toy to a fully fledged distribution, most of the original targets I set myself have been met, it's been my primary computing environment for a while, it's proving useful as a platform for interesting experiments, and I'm looking forward to taking it even further in the next few years.
Tuesday, October 20, 2015
Minimal Viable Illumos
I've been playing with the creation of several minimal variants of illumos recently.
I looked at how little memory a minimal illumos distro could be installed and run in. Note that this was a properly built distribution - correctly packaged, most features present (if disabled), running the whole set of services using SMF.
In another dimension, I considered illumos pureboot, something that was illumos, the whole of illumos, and nothing but illumos.
Given that it was possible to boot illumos just to a shell, without all the normal SMF services running, how minimal can you make such a system.
At this point, if you're not thinking JEOS, Unikernels, or things like IncludeOS, then you should be.
So the first point is that you're always running this under a hypervisor of some sort. The range of possible hardware configurations you need to worry about is very limited - hypervisors emulate a small handful of common devices.
Secondly, the intention is never to install this. Not directly anyway. You create an image, and boot and run that. For this experiment, I'm simply running from a ramdisk. This is the way the live image boots, or you PXE boot, or even the way SmartOS boots.
First, the starting set of packages, both in Tribblix (SVR4) and IPS naming:
There are a few packages here that you might wonder about:
I took two approaches to this. The simple way is to simply start chopping out the files you don't want. For example, man pages and includes. The second is to drop all of userland and only put back the files you need, one by one. I tend not to tweak the kernel much, that's non-trivial and you're only looking at marginal gains.
Working out which files are necessary is trial and error. Especially shared libraries, many of which are loaded lazily so you can't just use what the executable tells you - some of the libraries it's linked against will never be pulled in
I've put together some scripts that know how to create an image suitable for 32-bit or 64-bit hardware, we can be specific as we know exactly the environment we are going to run in - and you just build a new custom iso if things change, rather than try and build a generic image.
To be useful, the system needs to talk to something. You'll see that I've installed e1000g, which is what VirtualBox and qemu will give you by default. First, we have to get the root filesystem mounted read-write:
You need ipmgmtd running, and it's expecting to be run under SMF, but the way it checks is to look for SMF_FMRI in the environment, so it's easy to fool.
If you've got your VirtualBox VM set up with a Host-only Adapter, you should be able to communicate with the guest. Not that there's anything present to talk to yet.
So I set up a simple Node.js server. Now node itself doesn't have many external dependencies - just the gcc4 runtime - and for basic purposes you just need the node binary and a js file with a 'hello world' http server.
With that, I have 64M of data in a 22M boot archive that is put on a 25M iso that boots up in a few seconds with an accessible web server. Pretty neat.
While it's pretty specific to Tribblix and my own build environment, there's an mvi repository on github containing all the scripts I used to build this, for those interested.
I looked at how little memory a minimal illumos distro could be installed and run in. Note that this was a properly built distribution - correctly packaged, most features present (if disabled), running the whole set of services using SMF.
In another dimension, I considered illumos pureboot, something that was illumos, the whole of illumos, and nothing but illumos.
Given that it was possible to boot illumos just to a shell, without all the normal SMF services running, how minimal can you make such a system.
At this point, if you're not thinking JEOS, Unikernels, or things like IncludeOS, then you should be.
So the first point is that you're always running this under a hypervisor of some sort. The range of possible hardware configurations you need to worry about is very limited - hypervisors emulate a small handful of common devices.
Secondly, the intention is never to install this. Not directly anyway. You create an image, and boot and run that. For this experiment, I'm simply running from a ramdisk. This is the way the live image boots, or you PXE boot, or even the way SmartOS boots.
First, the starting set of packages, both in Tribblix (SVR4) and IPS naming:
- SUNWcsd=SUNWcsd
- SUNWcs=SUNWcs
- TRIBsys-library=system/library
- TRIBsys-kernel=system/kernel
- TRIBdrv-ser-usbser=driver/serial/usbser
- TRIBsys-kernel-platform=system/kernel/platform
- TRIBdrv-usb=driver/usb
- TRIBsys-kernel-dtrace=system/kernel/dtrace/providers
- TRIBsys-net=system/network
- TRIBsys-lib-math=system/library/math
- TRIBsys-libdiskmgt=system/library/libdiskmgt
- TRIBsys-boot-grub=system/boot/grub
- TRIBsys-zones=system/zones
- TRIBdrv-storage-ata=driver/storage/ata
- TRIBdrv-storage-ahci=driver/storage/ahci
- TRIBdrv-i86pc-platform=driver/i86pc/platform
- TRIBdrv-i86pc-ioat=driver/i86pc/ioat
- TRIBdrv-i86pc-fipe=driver/i86pc/fipe
- TRIBdrv-net-e1000g=driver/network/e1000g
- TRIBsys-boot-real-mode=system/boot/real-mode
- TRIBsys-file-system-zfs=system/file-system/zfs
There are a few packages here that you might wonder about:
- usbser is actually needed, it's a hard dependency of consconfig_dacf
- many admin commands link against the zones libraries, so I add those even though they're not strictly necessary in most scenarios
- the system will boot and run without zfs, but will panic if you run find down the dev tree
- the system will panic if the real-mode stuff is missing
- grub is needed to make the iso and boot it
I took two approaches to this. The simple way is to simply start chopping out the files you don't want. For example, man pages and includes. The second is to drop all of userland and only put back the files you need, one by one. I tend not to tweak the kernel much, that's non-trivial and you're only looking at marginal gains.
Working out which files are necessary is trial and error. Especially shared libraries, many of which are loaded lazily so you can't just use what the executable tells you - some of the libraries it's linked against will never be pulled in
I've put together some scripts that know how to create an image suitable for 32-bit or 64-bit hardware, we can be specific as we know exactly the environment we are going to run in - and you just build a new custom iso if things change, rather than try and build a generic image.
To be useful, the system needs to talk to something. You'll see that I've installed e1000g, which is what VirtualBox and qemu will give you by default. First, we have to get the root filesystem mounted read-write:
/etc/fs/ufs/mount -o remount,rw /devices/ramdisk:a /Normally, there's a whole lot of network configuration handled by SMF, and it's all rather complicated. So we have to do it all by hand, which turns out to be relatively simple:
/sbin/ifconfig e1000g0 plumb
env SMF_FMRI=svc:/net/ip:d /lib/inet/ipmgmtd
/sbin/ifconfig e1000g0 inet 192.168.59.59/24 up
You need ipmgmtd running, and it's expecting to be run under SMF, but the way it checks is to look for SMF_FMRI in the environment, so it's easy to fool.
If you've got your VirtualBox VM set up with a Host-only Adapter, you should be able to communicate with the guest. Not that there's anything present to talk to yet.
So I set up a simple Node.js server. Now node itself doesn't have many external dependencies - just the gcc4 runtime - and for basic purposes you just need the node binary and a js file with a 'hello world' http server.
With that, I have 64M of data in a 22M boot archive that is put on a 25M iso that boots up in a few seconds with an accessible web server. Pretty neat.
While it's pretty specific to Tribblix and my own build environment, there's an mvi repository on github containing all the scripts I used to build this, for those interested.
Thursday, October 08, 2015
Deconstructing .pyc files
I've recently been trying to work out why python was recompiling a bunch of .pyc files. I haven't solved that, but I learnt a little along the way, enough to be worth writing down.
Python will recompile a .py file onto a .pyc file if it thinks something's changed. But how does it decide something has changed? It encodes some of the pertinent details in the header of the .pyc file.
Consider a file. There's a foo.py and a foo.pyc. I open up the .pyc file in emacs and view it in hex. (ESC-x hexl-mode for those unfamiliar.)
The file starts off like this:
The first 4 bytes 03f30d0a are the magic number, and encode the version of python. There's a list of magic numbers in the source, here.
To check this, take the 03f3, reverse it to f303, which is 62211 decimal. That corresponds to 2.7a0 - this is python 2.7, so that matches. (The 0d0a is also part of the encoding of the magic number.) This check is just to see if the .pyc file is compatible with the version of python you're using. If it's not, it will ignore the .pyc file and may regenerate it.
The next bit is c3687955. Reverse this again to get the endianness right, and it's 557968c3. In decimal, that's 1434020035.
That's a timestamp, standard unix time. What does that correspond to?
And I can look at the file (on Solaris and illumos, there's a -e flag to ls to give us the time in the right format rather than the default "simplified" version).
As you can see, that matches the timestamp on the source file exactly. If the timestamp doesn't match, then again python will ignore it.
This has consequences for packaging. SVR4 packaging automatically preserves timestamps, with IPS you need to use pkgsend -T to do so as it's not done by default.
Python will recompile a .py file onto a .pyc file if it thinks something's changed. But how does it decide something has changed? It encodes some of the pertinent details in the header of the .pyc file.
Consider a file. There's a foo.py and a foo.pyc. I open up the .pyc file in emacs and view it in hex. (ESC-x hexl-mode for those unfamiliar.)
The file starts off like this:
03f3 0d0a c368 7955 6300 0000 ....
The first 4 bytes 03f30d0a are the magic number, and encode the version of python. There's a list of magic numbers in the source, here.
To check this, take the 03f3, reverse it to f303, which is 62211 decimal. That corresponds to 2.7a0 - this is python 2.7, so that matches. (The 0d0a is also part of the encoding of the magic number.) This check is just to see if the .pyc file is compatible with the version of python you're using. If it's not, it will ignore the .pyc file and may regenerate it.
The next bit is c3687955. Reverse this again to get the endianness right, and it's 557968c3. In decimal, that's 1434020035.
That's a timestamp, standard unix time. What does that correspond to?
perl -e '$f=localtime(1434020035); print $f'
Thu Jun 11 11:53:55 2015
And I can look at the file (on Solaris and illumos, there's a -e flag to ls to give us the time in the right format rather than the default "simplified" version).
/bin/ls -eo foo.py
-rw-r--r-- 1 root 7917 Jun 11 11:53:55 2015 foo.py
As you can see, that matches the timestamp on the source file exactly. If the timestamp doesn't match, then again python will ignore it.
This has consequences for packaging. SVR4 packaging automatically preserves timestamps, with IPS you need to use pkgsend -T to do so as it's not done by default.
Tuesday, October 06, 2015
Software directions in Tribblix
Tribblix has been developing in a number of different directions. I've been working on trimming the live image, and strengthening the foundations.
Beyond this, there is a continual stream of updated packages. Generally, if I package it, I'll try and keep it up to date. (If it's downrev, it's usually for a reason.)
In the meantime I've found time for experiments in booting Tribblix in very little memory, and creating a pure illumos bootable system.
But I thought it worthwhile to highlight some of the individual packages that have gone into Tribblix recently.
The big one was adding LibreOffice, of course. Needless to say, this was a modest amount of work. (Not necessarily all that hard, but it's a big build, and the edit-compile-debug cycle is fairly long, so it took a while.) I need to go back and update LibreOffice to a more current version, but the version I now have meets all of my needs so I can invest time and energy elsewhere.
On the desktop, I added MATE, and incorporated SLiM as a login manager. Tribblix has a lot of desktop environments and window managers available, although Xfce is still the primary and best supported option. I finally added the base GTK engines and icon themes, which got rid of a lot of errors.
In terms of tools, there's now Dia, Scribus, and Inkscape.
Tribblix has always had a retro streak. I've added gopher, gophervr, and the old Mosaic browser. There are other old X11 tools that some of you may remember - xcoral, xsnow, xsol, xshisen. If only I could get xfishtank working again.
I've been keeping up with Node.js releases, of course. But the new kid on the block is Go, and that's included in Tribblix. Current versions work very well, and now we've got past the cgo problems, there's a whole raft of modern software written in Go that's now available to us. The next one up is probably Rust.
Beyond this, there is a continual stream of updated packages. Generally, if I package it, I'll try and keep it up to date. (If it's downrev, it's usually for a reason.)
In the meantime I've found time for experiments in booting Tribblix in very little memory, and creating a pure illumos bootable system.
But I thought it worthwhile to highlight some of the individual packages that have gone into Tribblix recently.
The big one was adding LibreOffice, of course. Needless to say, this was a modest amount of work. (Not necessarily all that hard, but it's a big build, and the edit-compile-debug cycle is fairly long, so it took a while.) I need to go back and update LibreOffice to a more current version, but the version I now have meets all of my needs so I can invest time and energy elsewhere.
On the desktop, I added MATE, and incorporated SLiM as a login manager. Tribblix has a lot of desktop environments and window managers available, although Xfce is still the primary and best supported option. I finally added the base GTK engines and icon themes, which got rid of a lot of errors.
In terms of tools, there's now Dia, Scribus, and Inkscape.
Tribblix has always had a retro streak. I've added gopher, gophervr, and the old Mosaic browser. There are other old X11 tools that some of you may remember - xcoral, xsnow, xsol, xshisen. If only I could get xfishtank working again.
I've been keeping up with Node.js releases, of course. But the new kid on the block is Go, and that's included in Tribblix. Current versions work very well, and now we've got past the cgo problems, there's a whole raft of modern software written in Go that's now available to us. The next one up is probably Rust.
Fun with SPARC emulators
While illumos supports both SPARC and x86 platforms, it would be a fair assessment that the SPARC support is a poor relation.
There are illumos distributions that run on SPARC - OpenSXCE has for a while, Tribblix and DilOS also have SPARC images available and both are actively maintained. The mainstream distributions are x86-only.
A large part of the lack of SPARC support is quite simple - the number of users with SPARC hardware is small; the number of developers with SPARC hardware is even smaller. And you can see that the SPARC support is largely in the hands of the hobbyist part of the community. (Which is to be expected - the commercial members of the community are obviously not going to spend money on supporting hardware they neither have nor use.)
Absent physical hardware, are there any alternatives?
Perhaps the most obvious candidate is qemu. However, the sparc64 implementation is fairly immature. In other words, it doesn't work. Tribblix will start to boot, and does get a little way into the kernel before qemu crashes. I think it's generally agreed that qemu isn't there yet.
The next thing I tried is legion, which is the T1/T2 simulator from the opensparc project. Having built this, attempting to boot an iso image immediately fails with:
which makes it rather useless. (I haven't investigated to see if support can be enabled, but the build system explicitly disables it.) Legion hasn't been updated in a while, and I can't see that changing.
Then I came across the M5 simulator. This supports a number of systems, not just SPARC. But it's an active project, and claims to be able to emulate a full SPARC system. I can build it easily enough, running it needs the opensparc binary download from legion (note - you need the T1 download, version 1.5, not the newer T2 version of the download). The instructions here appear to be valid.
With M5, I can try booting Tribblix for SPARC. And it actually gets a lot further than I expected! Just not far enough:
Still, that's illumos bailing, there aren't any errors from M5.
Overall, I think that M5 shows some promise as a SPARC emulator for illumos.
There are illumos distributions that run on SPARC - OpenSXCE has for a while, Tribblix and DilOS also have SPARC images available and both are actively maintained. The mainstream distributions are x86-only.
A large part of the lack of SPARC support is quite simple - the number of users with SPARC hardware is small; the number of developers with SPARC hardware is even smaller. And you can see that the SPARC support is largely in the hands of the hobbyist part of the community. (Which is to be expected - the commercial members of the community are obviously not going to spend money on supporting hardware they neither have nor use.)
Absent physical hardware, are there any alternatives?
Perhaps the most obvious candidate is qemu. However, the sparc64 implementation is fairly immature. In other words, it doesn't work. Tribblix will start to boot, and does get a little way into the kernel before qemu crashes. I think it's generally agreed that qemu isn't there yet.
The next thing I tried is legion, which is the T1/T2 simulator from the opensparc project. Having built this, attempting to boot an iso image immediately fails with:
FATAL: virtual_disk not supported on this platform
which makes it rather useless. (I haven't investigated to see if support can be enabled, but the build system explicitly disables it.) Legion hasn't been updated in a while, and I can't see that changing.
Then I came across the M5 simulator. This supports a number of systems, not just SPARC. But it's an active project, and claims to be able to emulate a full SPARC system. I can build it easily enough, running it needs the opensparc binary download from legion (note - you need the T1 download, version 1.5, not the newer T2 version of the download). The instructions here appear to be valid.
With M5, I can try booting Tribblix for SPARC. And it actually gets a lot further than I expected! Just not far enough:
cpu Probing I/O buses
Sun Fire T2000, No Keyboard
Copyright 2005 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.0, 256 MB memory available, Serial #1122867.
[mo23723 obp4.20.0 #0]
Ethernet address 0:80:3:de:ad:3, Host ID: 80112233.
ok boot
Boot device: vdisk File and args:
hsfs-file-system
Loading: /platform/sun4v/boot_archive
ramdisk-root ufs-file-system
Loading: /platform/sun4v/kernel/sparcv9/unix
\
panic[cpu0]/thread=180e000: lgrp_traverse: No memory blocks found
Still, that's illumos bailing, there aren't any errors from M5.
Overall, I think that M5 shows some promise as a SPARC emulator for illumos.
Friday, October 02, 2015
Notifications from SMF (and FMA)
In illumos, it's possible to set the system up so that notifications are sent whenever anything happens to an SMF service.
Unfortunately, however, the illumos documentation is essentially non-existent, although looking at the Solaris documentation on the subject should be accurate.
The first thing is that you can see the current notification state by looking at svcs -n:
Notification parameters for FMA Events
Event: problem-diagnosed
Notification Type: smtp
Active: true
reply-to: root@localhost
to: root@localhost
Notification Type: snmp
Active: true
Notification Type: syslog
Active: true
Event: problem-repaired
Notification Type: snmp
Active: true
Event: problem-resolved
Notification Type: snmp
Active: true
The first thing to realize here is that first line - these are the notifications sent by FMA, not SMF. There's a relationship, of course, in that if an SMF service fails and ends up in maintenance, then an FMA event will be generated, and notifications will be sent according to the above scheme.
(By the way, the configuration for this comes from the svc:/system/fm/notify-params:default service, which you can see the source for here. And you'll see that it basically matches exactly what I've shown above.)
Whether you actually receive the notifications is another matter. If you have syslogd running, which is normal, then you'll see the syslog messages ending up in the log files. To get the email or SNMP notifications relies on additional service. These are
service/fault-management/smtp-notify
service/fault-management/snmp-notify
and if these are installed and enabled, they'll send the notifications out.
You can also set up notifications inside SMF itself. There's a decent intro available for this feature, although you should note that illumos doesn't currently have any of the man pages referred to. This functionality uses the listnotify, setnotify, and delnotify subcommands to svccfg. The one thing that isn't often covered is the relationship between the SMF and the FMA notifications - it's important to understand that both exist, in a strangely mingled state, with some non-obvious overlap.
You can see the global SMF notifications with
/usr/sbin/svccfg listnotify -gThis will come back with nothing by default, so the only thing you'll see is the FMA notifications. To get SMF to email you if any service goes offline, then
/usr/sbin/svccfg setnotify -g to-offline mailto:admin@example.com
and you can set this up at a per-service level with
/usr/sbin/svccfg -s apache22 setnotify to-offline mailto:webadmin@example.com
Now, while the SMF notifications can be configured in a very granular manner - you can turn it on and off by service, you can control exactly which state transitions you're interested in, and you can route individual notifications to different destinations, when it comes to the FMA notifications all you have is a big hammer. It's all or nothing, and you can't be selective on where notifications end up (beyond the smtp vs snmp vs syslog channels).
This is unfortunate, because SMF isn't the only source of telemetry that gets fed into FMA. In particular, the system hardware and ZFS will generate FMA events if there's a problem. If you want to get notifications from FMA if there's a problem with ZFS, then you're also going to get notified if an SMF service breaks. In a development environment, this might happen quite a lot.
Perhaps the best compromise I've come up with is to have FMA notifications disabled in a non-global zone, and configure SMF notifications explicitly there. Then, just have FMA notifications in the global zone. This assumes you have nothing but applications in zones, and all the non-SMF events will get caught in the global zone.
Tuesday, September 29, 2015
Improving the foundations of Tribblix
Mostly due to history, the software packages that make up Tribblix come from 3 places.
The second category is the historical part. Tribblix was bootstrapped from another distro. Owing to the fact that the amount of time I have is rather limited, not all the bits used in the bootstrapping have yet been removed. These tend to be in the foundations of the OS, which makes them harder to replace (simply due to where they sit in the dependency tree).
In the latest update (the 0m16 prerelease) I've replaced several key components that were previously inherited. Part of this is so that I'm in control of these components (which is a good thing), another is simply that they needed upgrading to a newer version.
One key component here is perl. I've been building my own versions of perl to live separately, but decided it was time to replace the old system perl (I was at 5.10) with something current (5.22). This of itself is easy enough. I then have to rebuild illumos because it's connected to perl, and that's a slightly tricky problem - the build uses the Sun::Solaris module, which comes from the build. (Unfortunately it uses the copy on the system rather than the bits it just built.) So I had to pull the bits out from the failed build, install those on the system, and then the build goes through properly.
Another - more critical - component is libxml2. Critical because SMF uses it, so if you get that wrong you break the system's ability to boot. After much careful study of both the OmniOS and OpenIndiana build recipes, I picked a set of options and everything worked first time. Phew!
(Generally, I'll tend to the OpenIndiana way of doing things, simply because that's where the package I'm trying to replace came from. But I usually look at multiple distros for useful hints.)
A broader area was compression support. I updated zlib along with libxml2, but also went in and built my own p7zip, xz, and bzip2, and then started adding additional compression tools such as lzip and pigz.
The work isn't done yet. Two areas I need to look at are the Netscape Portable Runtime, and the base graphics libraries (tiff, jpeg, png). And then there's the whole X11 stack, which is a whole separate problem - because newer versions start to require KMS (which we don't have) or have gone 64-bit only (which is still an open question, and a leap I'm not yet prepared to take).
- The base OS from illumos
- Some base packages from OpenIndiana (or OpenSXCE for SPARC)
- Software built afresh for Tribblix
The second category is the historical part. Tribblix was bootstrapped from another distro. Owing to the fact that the amount of time I have is rather limited, not all the bits used in the bootstrapping have yet been removed. These tend to be in the foundations of the OS, which makes them harder to replace (simply due to where they sit in the dependency tree).
In the latest update (the 0m16 prerelease) I've replaced several key components that were previously inherited. Part of this is so that I'm in control of these components (which is a good thing), another is simply that they needed upgrading to a newer version.
One key component here is perl. I've been building my own versions of perl to live separately, but decided it was time to replace the old system perl (I was at 5.10) with something current (5.22). This of itself is easy enough. I then have to rebuild illumos because it's connected to perl, and that's a slightly tricky problem - the build uses the Sun::Solaris module, which comes from the build. (Unfortunately it uses the copy on the system rather than the bits it just built.) So I had to pull the bits out from the failed build, install those on the system, and then the build goes through properly.
Another - more critical - component is libxml2. Critical because SMF uses it, so if you get that wrong you break the system's ability to boot. After much careful study of both the OmniOS and OpenIndiana build recipes, I picked a set of options and everything worked first time. Phew!
(Generally, I'll tend to the OpenIndiana way of doing things, simply because that's where the package I'm trying to replace came from. But I usually look at multiple distros for useful hints.)
A broader area was compression support. I updated zlib along with libxml2, but also went in and built my own p7zip, xz, and bzip2, and then started adding additional compression tools such as lzip and pigz.
The work isn't done yet. Two areas I need to look at are the Netscape Portable Runtime, and the base graphics libraries (tiff, jpeg, png). And then there's the whole X11 stack, which is a whole separate problem - because newer versions start to require KMS (which we don't have) or have gone 64-bit only (which is still an open question, and a leap I'm not yet prepared to take).
Monday, September 28, 2015
Trimming the Tribblix live image
When Tribblix boots from the installation ISO, it reads in two things: the root archive, as a ramdisk, and /usr mounted from solaris.zlib via lofi.
In preparation for the next update, I've spent a little time minimizing both files. Part of this was alongside my experiments on genuinely memory-constrained systems; working out what's necessary in extreme cases can guide you into better behaviour in normal circumstances. While I don't necessarily expect installing onto a 128M system to be a routine occurrence, it would be good to keep 1G or even 512M within reach.
One of the largest single contributors to /usr was perl. It turns out that the only critical part of the system that needs perl is intrd, which is disabled in the live environment anyway. So, perl's not needed.
Another significant package is GNU coreutils. On closer investigation, the only reason I needed this was for a script that generated a UUID which is set as a ZFS property on the root file system (it's used by beadm to match up which zone BE matches the system BE). Apart from the fact that this functionality has recently been integrated into illumos, using the GNU coreutils was just being lazy (perhaps it was necessary under Solaris 10, where this script originated, but the system utilities are good enough now).
I also had the gcc runtime installed. The illumos packages don't need it, but some 3rd-party packages did - compile with gcc and you tend to end up with libgcc_s being pulled in. There are a variety of tricks with -nostdlib and -static-libgcc that are necessary to avoid this. (And I wish I understood better exactly what's happening, as it's too much like magic for my liking.)
The overall impact isn't huge, but the overall footprint of the live image has been reduced by 25%, which is worthwhile. It also counteracts the seemingly inevitable growth of the base system, so I have to worry less about whether I can justify every single driver or small package that might be useful.
In preparation for the next update, I've spent a little time minimizing both files. Part of this was alongside my experiments on genuinely memory-constrained systems; working out what's necessary in extreme cases can guide you into better behaviour in normal circumstances. While I don't necessarily expect installing onto a 128M system to be a routine occurrence, it would be good to keep 1G or even 512M within reach.
One of the largest single contributors to /usr was perl. It turns out that the only critical part of the system that needs perl is intrd, which is disabled in the live environment anyway. So, perl's not needed.
Another significant package is GNU coreutils. On closer investigation, the only reason I needed this was for a script that generated a UUID which is set as a ZFS property on the root file system (it's used by beadm to match up which zone BE matches the system BE). Apart from the fact that this functionality has recently been integrated into illumos, using the GNU coreutils was just being lazy (perhaps it was necessary under Solaris 10, where this script originated, but the system utilities are good enough now).
I also had the gcc runtime installed. The illumos packages don't need it, but some 3rd-party packages did - compile with gcc and you tend to end up with libgcc_s being pulled in. There are a variety of tricks with -nostdlib and -static-libgcc that are necessary to avoid this. (And I wish I understood better exactly what's happening, as it's too much like magic for my liking.)
The overall impact isn't huge, but the overall footprint of the live image has been reduced by 25%, which is worthwhile. It also counteracts the seemingly inevitable growth of the base system, so I have to worry less about whether I can justify every single driver or small package that might be useful.
Friday, September 25, 2015
illumos pureboot
In my previous article, I discussed an experiment in illumos minimization.
Interestingly, there was a discussion on IRC that wandered off into realms afar, but it got me thinking about making oddball illumos images.
So then I thought - how would you build something that was pure illumos. As in illumos, the whole of illumos, and nothing but illumos.
Somewhat surprisingly, this works. For some definition of works, anyway.
The idea is pretty simple. After building illumos, you end up with the artefacts that are created by the build populating a proto area. This has the same structure as a regular system, so you can find usr/bin/ls under there, for example.
So all I do is create a bootable image that is the proto area from a build.
The script is here.
What does this do?
My (non-debug) proto area seems to be around 464MB, so a 512MB ramdisk works just fine. You could start deleting (or, indeed, adding) stuff to tune the image you create. The ISO image is 166MB, ready for VirtualBox.
The important thing to realise is that illumos, of itself, does not create a complete operating system. Even core OS functionality requires additional third-party components, which will not be present in the image you create. In particular, libxml2, zlib, and openssl are missing. What this means is that anything depending on these will not function. The list of things that won't work includes SMF, which is an integral part of the normal boot and operations.
So instead of init launching SMF, I have it run a shell instead. (I actually do this via a script rather than directly from init, this allows me to put up a message, and also allows me to run other startup commands if so desired.)
This is what it looks like:
A surprising amount of stuff actually works in this environment. Certainly most of the standard unix commands that I've tried are just fine. Although it has to be appreciated that none of the normal boot processing has been done at this point, so almost nothing has been set up. (And / won't budge from being read-only which is a little confusing.)
Interestingly, there was a discussion on IRC that wandered off into realms afar, but it got me thinking about making oddball illumos images.
So then I thought - how would you build something that was pure illumos. As in illumos, the whole of illumos, and nothing but illumos.
Somewhat surprisingly, this works. For some definition of works, anyway.
The idea is pretty simple. After building illumos, you end up with the artefacts that are created by the build populating a proto area. This has the same structure as a regular system, so you can find usr/bin/ls under there, for example.
So all I do is create a bootable image that is the proto area from a build.
The script is here.
What does this do?
- Copies the proto area to a staging area, so it can be manipulated
- Modifies inittab
- Sets up grub for boot
- Copies the kernel state files from the running system (otherwise, they're blank and the kernel is clueless)
- Creates a block device and builds a ufs file system on it
- Copies the staging area to it
- Compresses the block device
- Copies it back to the platform staging area
- Creates a bootable iso
My (non-debug) proto area seems to be around 464MB, so a 512MB ramdisk works just fine. You could start deleting (or, indeed, adding) stuff to tune the image you create. The ISO image is 166MB, ready for VirtualBox.
The important thing to realise is that illumos, of itself, does not create a complete operating system. Even core OS functionality requires additional third-party components, which will not be present in the image you create. In particular, libxml2, zlib, and openssl are missing. What this means is that anything depending on these will not function. The list of things that won't work includes SMF, which is an integral part of the normal boot and operations.
So instead of init launching SMF, I have it run a shell instead. (I actually do this via a script rather than directly from init, this allows me to put up a message, and also allows me to run other startup commands if so desired.)
This is what it looks like:
A surprising amount of stuff actually works in this environment. Certainly most of the standard unix commands that I've tried are just fine. Although it has to be appreciated that none of the normal boot processing has been done at this point, so almost nothing has been set up. (And / won't budge from being read-only which is a little confusing.)
Sunday, September 20, 2015
How low can Tribblix go?
One of the things I wanted to do with Tribblix was to allow it to run in places that other illumos distros couldn't reach.
One possible target here is systems with less than gargantuan memory capacities.
(Now, I don't have any such systems. But VirtualBox allows you to adjust the amount of memory in a guest very easily, so that's what I'm doing.)
I started out by building a 32-bit only image. That is, I built a regular (32- and 64-bit combined) image, and simply deleted all the 64-bit pieces. You can do this slightly better by building custom 32-bit only packages, but it was much simpler to identify all the directories named amd64 and delete them.
(Why focus on 32-bit here? The default image has both a 32-bit and 64-bit kernel, and many libraries are shipped in both flavours too. So removing one of the 32-bit or 64-bit flavours will potentially halve the amount of space we need. It makes more sense to drop the 64-bit files - it's easier to do, and it's more likely that real systems with minimal memory are going to be 32-bit.)
The regular boot archive in Tribblix is 160M in size (the file on the ISO is gzip-compressed and ends up being about a third of that), but it's loaded into memory as a ramdisk so the full size is a hard limit on how much memory you're going to need to boot the ISO. You might be able to run off disk with less, as we'll see later. The 32-bit boot archive can be shrunk to 90M, and still has a little room to work in.
The other part of booting from media involves the /usr file system being a compressed lofi mount from a file. I've made a change in the upcoming release by removing perl from the live boot (it's only needed for intrd, which is disabled anyway if you're booting from media), which saves a bit of space, and the 32-bit version of /usr is about a third smaller than the regular combined 32/64-bit variant. Without any additional changes, it is about 171M.
So, the boot archive takes a fixed 90M, and the whole of /usr takes 171M. Let's call that 256M of basic footprint.
I know that regular Tribblix will boot and install quite happily with 1G of memory, and previous experience is that 768M is fine too.
So I started with a 512M setup. The ISO boots just fine. I tried an install to ZFS. The initial part of the install - which is a simple cpio of the system as booted from media - worked fine, if very slowly. The second part of the base install (necessary even if you don't add more software) adds a handful of packages. This is where it really started to struggle, it just about managed the first package and then ground completely to a halt.
Now, I'm sure you could tweak the system a little further to trim the size of both the boot archive and /usr, or tweak the amount of memory ZFS uses, but we're clearly close to the edge.
So then I tried exactly the same setup, installing to UFS instead of ZFS. And it installs absolutely fine, and goes like greased lightning. OK, the conclusion here is that if you want a minimal system with less than 512M of memory, then don't bother with ZFS but aim at UFS instead.
Reducing memory to 256M, the boot and install to UFS still work fine.
With 192M of memory, boot is still good, the install is starting to get a bit sluggish.
If I go down to 128M of memory, the ISO won't boot at all.
However, if I install with a bit more memory, and then reduce it later, Tribblix on UFS works just fine with 128M of memory. Especially if you disable a few services. (Such as autofs cron zones-monitoring zones power fmd. Not necessarily what you want to do in production, but this isn't supposed to be production.)
It looks like 128M is a reasonable practical lower limit. The system is using most of the 128M (it's starting to write data to swap, so there's clearly not much headroom).
Going lower also starts to hit real hard limits. While 120M is still good, 112M fails to boot at all (I get "do_bop_phys_alloc Out of memory" errors from the kernel - see the fakebop source). I'm sure I could go down a bit further, but I think the next step is to start removing drivers from the kernel, which will reduce both the installed boot archive size and the kernel's memory requirements.
I then started to look more closely at the boot archive. On my test machine, it was 81M in size. Removing all the drivers I felt safe with dropped it down to 77M. That still seems quite large.
Diving into the boot archive itself, and crawling through the source for bootadm, I then found that the boot archive was a ufs archive that's only 25% full. It turns out that the boot archive will be hsfs if the system finds /usr/bin/mkisofs, otherwise it uses ufs. And it looks like the size calculation is a bit off, leading to an archive that's massively oversized. After installing mkisofs and rebuilding the boot archive, I got back to something that was 17M, which is much better.
On testing with the new improved boot archive, boot with 96M, or even 88M, memory is just fine.
Down to 80M of memory, and I hit the next wall. The system looks as though it will boot reasonably well, but /etc/svc/volatile fills up and you run out of swap. I suspect this is before it's had any opportunity to add the swap partition, but once it's in that state it can't progress.
Overall, in answer to the question in the title, a 32-bit variant of Tribblix will install (using UFS) on a system with as little as 192M of memory, and run on as little as 96M.
One possible target here is systems with less than gargantuan memory capacities.
(Now, I don't have any such systems. But VirtualBox allows you to adjust the amount of memory in a guest very easily, so that's what I'm doing.)
I started out by building a 32-bit only image. That is, I built a regular (32- and 64-bit combined) image, and simply deleted all the 64-bit pieces. You can do this slightly better by building custom 32-bit only packages, but it was much simpler to identify all the directories named amd64 and delete them.
(Why focus on 32-bit here? The default image has both a 32-bit and 64-bit kernel, and many libraries are shipped in both flavours too. So removing one of the 32-bit or 64-bit flavours will potentially halve the amount of space we need. It makes more sense to drop the 64-bit files - it's easier to do, and it's more likely that real systems with minimal memory are going to be 32-bit.)
The regular boot archive in Tribblix is 160M in size (the file on the ISO is gzip-compressed and ends up being about a third of that), but it's loaded into memory as a ramdisk so the full size is a hard limit on how much memory you're going to need to boot the ISO. You might be able to run off disk with less, as we'll see later. The 32-bit boot archive can be shrunk to 90M, and still has a little room to work in.
The other part of booting from media involves the /usr file system being a compressed lofi mount from a file. I've made a change in the upcoming release by removing perl from the live boot (it's only needed for intrd, which is disabled anyway if you're booting from media), which saves a bit of space, and the 32-bit version of /usr is about a third smaller than the regular combined 32/64-bit variant. Without any additional changes, it is about 171M.
So, the boot archive takes a fixed 90M, and the whole of /usr takes 171M. Let's call that 256M of basic footprint.
I know that regular Tribblix will boot and install quite happily with 1G of memory, and previous experience is that 768M is fine too.
So I started with a 512M setup. The ISO boots just fine. I tried an install to ZFS. The initial part of the install - which is a simple cpio of the system as booted from media - worked fine, if very slowly. The second part of the base install (necessary even if you don't add more software) adds a handful of packages. This is where it really started to struggle, it just about managed the first package and then ground completely to a halt.
Now, I'm sure you could tweak the system a little further to trim the size of both the boot archive and /usr, or tweak the amount of memory ZFS uses, but we're clearly close to the edge.
So then I tried exactly the same setup, installing to UFS instead of ZFS. And it installs absolutely fine, and goes like greased lightning. OK, the conclusion here is that if you want a minimal system with less than 512M of memory, then don't bother with ZFS but aim at UFS instead.
Reducing memory to 256M, the boot and install to UFS still work fine.
With 192M of memory, boot is still good, the install is starting to get a bit sluggish.
If I go down to 128M of memory, the ISO won't boot at all.
However, if I install with a bit more memory, and then reduce it later, Tribblix on UFS works just fine with 128M of memory. Especially if you disable a few services. (Such as autofs cron zones-monitoring zones power fmd. Not necessarily what you want to do in production, but this isn't supposed to be production.)
It looks like 128M is a reasonable practical lower limit. The system is using most of the 128M (it's starting to write data to swap, so there's clearly not much headroom).
Going lower also starts to hit real hard limits. While 120M is still good, 112M fails to boot at all (I get "do_bop_phys_alloc Out of memory" errors from the kernel - see the fakebop source). I'm sure I could go down a bit further, but I think the next step is to start removing drivers from the kernel, which will reduce both the installed boot archive size and the kernel's memory requirements.
I then started to look more closely at the boot archive. On my test machine, it was 81M in size. Removing all the drivers I felt safe with dropped it down to 77M. That still seems quite large.
Diving into the boot archive itself, and crawling through the source for bootadm, I then found that the boot archive was a ufs archive that's only 25% full. It turns out that the boot archive will be hsfs if the system finds /usr/bin/mkisofs, otherwise it uses ufs. And it looks like the size calculation is a bit off, leading to an archive that's massively oversized. After installing mkisofs and rebuilding the boot archive, I got back to something that was 17M, which is much better.
On testing with the new improved boot archive, boot with 96M, or even 88M, memory is just fine.
Down to 80M of memory, and I hit the next wall. The system looks as though it will boot reasonably well, but /etc/svc/volatile fills up and you run out of swap. I suspect this is before it's had any opportunity to add the swap partition, but once it's in that state it can't progress.
Overall, in answer to the question in the title, a 32-bit variant of Tribblix will install (using UFS) on a system with as little as 192M of memory, and run on as little as 96M.
Sunday, September 06, 2015
Fixing SLiM
Having been using my version of SLiM as a desktop login manager for a few days, I had seen a couple of spontaneous logouts.
After minimal investigation, this was a trivial configuration error on my part. And, fortunately, easy to fix.
The slim process is managed by SMF. This ensures that it starts at boot (at the right time, I've written it so that it's dependent on the console-login service, so it launches immediately the CLI login is ready) and that it gets restarted if it exits for whatever reason.
So I had seen myself being logged out on a couple of different occasions. Once when exiting a VNC session (as another user, no less); another whilst running a configure script.
A quick look at the SMF log file, in /var/svc/log/system-slim:default.log, gives an immediate hint:
So, the fix is a standard one, which I had forgotten entirely. Just add the following snippet to the SMF manifest:
After minimal investigation, this was a trivial configuration error on my part. And, fortunately, easy to fix.
The slim process is managed by SMF. This ensures that it starts at boot (at the right time, I've written it so that it's dependent on the console-login service, so it launches immediately the CLI login is ready) and that it gets restarted if it exits for whatever reason.
So I had seen myself being logged out on a couple of different occasions. Once when exiting a VNC session (as another user, no less); another whilst running a configure script.
A quick look at the SMF log file, in /var/svc/log/system-slim:default.log, gives an immediate hint:
[ Sep 5 13:37:17 Stopping because process dumped core. ]So, a process in the slim process contract - which is all processes launched from the desktop - dumped core, SMF spotted it happening, and restarted the whole desktop session. You really don't want that, especially as as a desktop session can be comprised of essentially arbitrary applications, and random core dumps are not entirely unexpected.
So, the fix is a standard one, which I had forgotten entirely. Just add the following snippet to the SMF manifest:
and everything is much better behaved.
<property_group name='startd' type='framework'>
<!-- sub-process core dumps shouldn't restart session -->
<propval name='ignore_error' type='astring'
value='core,signal' />
</property_group>
Thursday, September 03, 2015
Tribblix Graphical Login
Up to this point, login to Tribblix has been very traditional. The system boots to a console login, you enter your username and password, and then start your graphical desktop in whatever manner you choose.
That's reasonable for old-timers such as myself, but we can do better. The question is how to do that.
OpenSolaris, and thus OpenIndiana, have used gdm, from GNOME. I don't have GNOME, and don't wish to be forever locked in dependency hell, so that's not really an option for me.
There's always xdm, but it's still very primitive. I might be retro, but I'm also in favour of style and bling.
I had a good long look at LightDM, and managed to get that ported and running a while back. (And part of that work helped get it into XStreamOS.) However, LightDM is a moving target, it's evolving off in other directions, and it's quite a complicated beast. As a result, while I did manage to get it to work, I was never happy enough to enable it.
I've gone back to SLiM, which used to be hosted at BerliOS. The current source appears to be here. It has the advantage of being very simple, with minimal dependencies.
I made a few modification and customizations, and have it working pretty well. As upstream doesn't seem terribly active, and some of my changes are pretty specific, I decided to fork the code, my repo is here.
Apart from the basic business of making it compile correctly, I've put in a working configuration file, and added an SMF manifest.
SLiM doesn't have a very good mechanism for selecting what environment you get when you log in. By default it will execute your .xinitrc (and fail horribly if you don't have one). There is a mechanism where it can look in /usr/share/xsessions for .desktop files, and you can use F1 to switch between them, but there's no way currently to filter that list, or tell it what order to show then in, or have a default. So I switched that bit off.
I already have a mechanism in Tribblix to select the desktop environment, called tribblix-session. This allows you to use the setxsession and setvncsession commands to define which session you want to run, either in regular X (via the .xinitrc file) or using VNC. So my SLiM login calls a script that hooks into and cooperates with that, and then falls back on some sensible defaults - Xfce, MATE, WindowMaker, or - if all else fails - twm.
It's been working pretty well so far. It can also do automatic login for a given user, and there are magic logins for special purposes (console, halt, and reboot, with the root password).
Now what I need is a personalized theme.
That's reasonable for old-timers such as myself, but we can do better. The question is how to do that.
OpenSolaris, and thus OpenIndiana, have used gdm, from GNOME. I don't have GNOME, and don't wish to be forever locked in dependency hell, so that's not really an option for me.
There's always xdm, but it's still very primitive. I might be retro, but I'm also in favour of style and bling.
I had a good long look at LightDM, and managed to get that ported and running a while back. (And part of that work helped get it into XStreamOS.) However, LightDM is a moving target, it's evolving off in other directions, and it's quite a complicated beast. As a result, while I did manage to get it to work, I was never happy enough to enable it.
I've gone back to SLiM, which used to be hosted at BerliOS. The current source appears to be here. It has the advantage of being very simple, with minimal dependencies.
I made a few modification and customizations, and have it working pretty well. As upstream doesn't seem terribly active, and some of my changes are pretty specific, I decided to fork the code, my repo is here.
Apart from the basic business of making it compile correctly, I've put in a working configuration file, and added an SMF manifest.
SLiM doesn't have a very good mechanism for selecting what environment you get when you log in. By default it will execute your .xinitrc (and fail horribly if you don't have one). There is a mechanism where it can look in /usr/share/xsessions for .desktop files, and you can use F1 to switch between them, but there's no way currently to filter that list, or tell it what order to show then in, or have a default. So I switched that bit off.
I already have a mechanism in Tribblix to select the desktop environment, called tribblix-session. This allows you to use the setxsession and setvncsession commands to define which session you want to run, either in regular X (via the .xinitrc file) or using VNC. So my SLiM login calls a script that hooks into and cooperates with that, and then falls back on some sensible defaults - Xfce, MATE, WindowMaker, or - if all else fails - twm.
It's been working pretty well so far. It can also do automatic login for a given user, and there are magic logins for special purposes (console, halt, and reboot, with the root password).
Now what I need is a personalized theme.
Wednesday, September 02, 2015
The 32-bit dilemma
Should illumos, and the distributions based on it - such as Tribblix - continue to support 32-bit hardware?
(Note that this is about the kernel and 32-bit hardware, I'm not aware of any good cause to start dropping 32-bit applications and libraries.)
There are many good reasons to go 64-bit. Here are a few:
So, I know I'm retro and all, but it's getting very hard to justify keeping 32-bit support.
Going to a model where we just support 64-bit hardware has other advantages:
Are there any arguments for keeping 32-bit hardware support?
My own plan for Tribblix was that once I had got to releasing version 1 then version 2 would drop 32-bit hardware support. (I don't need illumos to drop it, I can remove the support as I postprocess the illumos build and create packages.) As time goes on, I'm starting to wonder whether to drop 32-bit earlier.
(Note that this is about the kernel and 32-bit hardware, I'm not aware of any good cause to start dropping 32-bit applications and libraries.)
There are many good reasons to go 64-bit. Here are a few:
- Support for 32-bit SPARC hardware never existed (Sun dropped it with Solaris 10, before OpenSolaris)
- Most new hardware is 64-bit, new 32-bit systems are very rare
- Even "old" x86 hardware is now 64-bit
- DragonFly BSD went 64-bit
- Solaris 11 dropped 32-bit
- SmartOS is 64-bit only
- Applications - or runtimes such as GO and Java 8 - are starting to only exist in 64-bit versions
- We're maintaining, building, and shipping code that effectively nobody uses, and is therefore effectively untested
- The time_t problem (traditional 32-bit time runs out in 2038)
So, I know I'm retro and all, but it's getting very hard to justify keeping 32-bit support.
Going to a model where we just support 64-bit hardware has other advantages:
- It makes SPARC and x86 equivalent
- We can make userland 64-bit by default
- ...which makes solving the time_t problem easier
- ...and any remaining issues with large files and 64-bit inode numbers go away
- We can avoid isaexec
- At least on x86, 64-bit applications perform better
- Eliminating 32-bit drivers and kernel makes packages and install images smaller
Are there any arguments for keeping 32-bit hardware support?
- It's a possible differentiator - a feature we have that others don't. On the other hand, if the potential additional user base is zero then it makes no difference
- There is still some existence of 32-bit hardware (mainly in embedded contexts)
My own plan for Tribblix was that once I had got to releasing version 1 then version 2 would drop 32-bit hardware support. (I don't need illumos to drop it, I can remove the support as I postprocess the illumos build and create packages.) As time goes on, I'm starting to wonder whether to drop 32-bit earlier.
Saturday, August 29, 2015
Tribblix meets MATE
One of the things I've been working on in Tribblix is to ensure that there's a good choice of desktop options. This varies from traditional window managers (all the way back to the old awm), to modern lightweight desktop environments.
The primary desktop environment (because it's the one I use myself most of the time) is Xfce, but I've had Enlightenment available as well. Recently, I've added MATE as an additional option.
OK, here's the obligatory screenshot:
While it's not quite as retro as some of the other desktop options, MATE has a similar philosophy to Tribblix - maintaining a traditional environment in a modern context. As a continuation of GNOME 2, I find it to have a familiar look and feel, but I also find it to be much less demanding both at build and run time. In addition, it's quite happy with older hardware or with VNC.
Building MATE on Tribblix was very simple. The dependencies it has are fairly straightforward - there aren't that many, and most of them you would tend to have anyway as part of a modern system.
To give a few hints, I needed to add dconf, a modern intltool, itstool, iso-codes, libcanberra, zenity, and libxklavier. Having downloaded the source tarballs, I built packages in this order (this isn't necessarily the strict dependency order, it was simply the most convenient):
The other thing I added was the murrine gtk2 theme engine. I had been getting odd warnings from applications for a while mentioning murrine, but MATE was competent enough to give me a meaningful warning.
I've been pretty impressed with MATE, it's a worthy addition to the available desktop environments, with a good philosophy and a clean implementation.
The primary desktop environment (because it's the one I use myself most of the time) is Xfce, but I've had Enlightenment available as well. Recently, I've added MATE as an additional option.
OK, here's the obligatory screenshot:
While it's not quite as retro as some of the other desktop options, MATE has a similar philosophy to Tribblix - maintaining a traditional environment in a modern context. As a continuation of GNOME 2, I find it to have a familiar look and feel, but I also find it to be much less demanding both at build and run time. In addition, it's quite happy with older hardware or with VNC.
Building MATE on Tribblix was very simple. The dependencies it has are fairly straightforward - there aren't that many, and most of them you would tend to have anyway as part of a modern system.
To give a few hints, I needed to add dconf, a modern intltool, itstool, iso-codes, libcanberra, zenity, and libxklavier. Having downloaded the source tarballs, I built packages in this order (this isn't necessarily the strict dependency order, it was simply the most convenient):
- mate-desktop
- mate-icon-theme
- eom (the image viewer)
- caja (the file manager)
- atril (the document viewer, disable libsecret)
- engramap (the archive manager)
- pluma (the text editor)
- mate-menus
- mateweather (is pretty massive)
- mate-panel
- mate-session
- marco (the window manager)
- mate-backgrounds
- mate-themes (from 1.8)
- libmatekbd
- mate-settings-daemon
- mate-control-center
The other thing I added was the murrine gtk2 theme engine. I had been getting odd warnings from applications for a while mentioning murrine, but MATE was competent enough to give me a meaningful warning.
I've been pretty impressed with MATE, it's a worthy addition to the available desktop environments, with a good philosophy and a clean implementation.
Monday, August 10, 2015
Whither open source?
According to the Free Software Definition, there are 4 essential freedoms:
Access to the source code and an open-source license are necessary preconditions for software freedom, but not sufficient.
And, unfortunately, we are living in an era where it is becoming ever more difficult to exercise the freedoms listed above.
Consider freedom 0. In the past, essentially all free software ran perfectly well on essentially every hardware platform and operating system. At the present time, much of what claims to be open-source software is horrendously platform-specific - sometimes by ignorance (I don't expect every developer to be able to test on all platforms), but there's a disturbing trend of deliberately excluding non-preferred platforms.
There is increasing use of new languages and runtimes, which are often very restricted in terms of platform support. If you look at some of the languages like Node.JS, Go, and Rust, you'll see that they explicitly target the common hardware architectures (x86 and ARM), deliberately and consciously excluding other platforms. Add to that the trend for self-referential bootstrapping (where you need X to build X) and you can see other platforms frozen out entirely.
So, much of freedom 0 has been emasculated. What of freedom 1?
Yes, I might be able to look at the source code. (Although, in many cases, it is opaque and undocumented.) And I might be able to crack open an editor and type in a modification. But actually being able to use that modification is a whole different ball game.
Actually building software from source often enters you into a world of pain and frustration. Fighting your way through Dependency Hell, struggling with arcane and opaque build systems, becoming frustrated with the vagaries of the autotools (remember how the configure script works - it makes a bunch of random and unsubstantiated guesses about the state of your system, the ignores half the results, and often needs explicitly overriding, making a mockery of the "auto" part), only to discover that "works on my system" is almost a religion.
Current trends like Docker make this problem worse. Rather than having to pay lip-service to portability by having to deal with the vagaries of multiple distributions, authors can now restrict the target environment even more narrowly - "works in my docker image" is the new normal. (I've had some developers come out and say this explicitly.)
The conclusion: open-source software is becoming increasingly narrow and proprietary, denying users the freedoms they deserve.
- The freedom to run the program as you wish, for any purpose (freedom 0).
- The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
- The freedom to redistribute copies so you can help your neighbor (freedom 2).
- The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.
Access to the source code and an open-source license are necessary preconditions for software freedom, but not sufficient.
And, unfortunately, we are living in an era where it is becoming ever more difficult to exercise the freedoms listed above.
Consider freedom 0. In the past, essentially all free software ran perfectly well on essentially every hardware platform and operating system. At the present time, much of what claims to be open-source software is horrendously platform-specific - sometimes by ignorance (I don't expect every developer to be able to test on all platforms), but there's a disturbing trend of deliberately excluding non-preferred platforms.
There is increasing use of new languages and runtimes, which are often very restricted in terms of platform support. If you look at some of the languages like Node.JS, Go, and Rust, you'll see that they explicitly target the common hardware architectures (x86 and ARM), deliberately and consciously excluding other platforms. Add to that the trend for self-referential bootstrapping (where you need X to build X) and you can see other platforms frozen out entirely.
So, much of freedom 0 has been emasculated. What of freedom 1?
Yes, I might be able to look at the source code. (Although, in many cases, it is opaque and undocumented.) And I might be able to crack open an editor and type in a modification. But actually being able to use that modification is a whole different ball game.
Actually building software from source often enters you into a world of pain and frustration. Fighting your way through Dependency Hell, struggling with arcane and opaque build systems, becoming frustrated with the vagaries of the autotools (remember how the configure script works - it makes a bunch of random and unsubstantiated guesses about the state of your system, the ignores half the results, and often needs explicitly overriding, making a mockery of the "auto" part), only to discover that "works on my system" is almost a religion.
Current trends like Docker make this problem worse. Rather than having to pay lip-service to portability by having to deal with the vagaries of multiple distributions, authors can now restrict the target environment even more narrowly - "works in my docker image" is the new normal. (I've had some developers come out and say this explicitly.)
The conclusion: open-source software is becoming increasingly narrow and proprietary, denying users the freedoms they deserve.
Sunday, August 02, 2015
Blank Zones
I've been playing around with various zone configurations on Tribblix. This is going beyond the normal sparse-root, whole-root, partial-root, and various other installation types, into thinking about other ways you can actually use zones to run software.
One possibility is what I'm tentatively calling a Blank zone. That is, a zone that has nothing running. Or, more precisely, just has an init process but not the normal array of miscellaneous processes that get started up by SMF in a normal boot.
You might be tempted to use 'zoneadm ready' rather than 'zoneadm boot'. This doesn't work, as you can't get into the zone:
Why not simply disable the SMF services you don't need? This is fine if you still want SMF and most of the services, but SMF itself is quite a beast, and the minimal set of service dependencies is both large and extremely complex. In practice, you end up running most things just to keep the SMF dependencies happy.
Now, SMF is started by init using the following line (I've trimmed the redirections) from /etc/inittab
OK, so all we have to do is delete this entry, and we just get init. Right? Wrong! It's not quite that simple. If you try this then you get a boot failure:
In practice, this isn't fatal - the zone is still running, but apart from wondering why it's behaving like this it would be nice to have the zone boot without errors.
Looking at the source for init, it soon becomes clear what's happening. The init process is now intimately aware of SMF, so essentially it knows that its only job is to get startd running, and startd will do all the work. However, it's clear from the code that it's only looking for the smf id in the first field. So my solution here is to replace startd with an infinite sleep.
(As an aside, this led to illumos bug 6019, as the manpage for sleep(1) isn't correct. Using 'sleep infinite' as the manpage suggests led to other failures.)
Then, the zone boots up, and the process tree looks like this:
To get into the zone, you just need to use zlogin. Without anything running, there aren't the normal daemons (like sshd) available for you to connect to. It's somewhat disconcerting to type 'netstat -a' and get nothing back.
For permanent services, you could run them from inittab (in the traditional way), or have an external system that creates the zones and uses zlogin to start the application. Of course, this means that you're responsible for any required system configuration and for getting any prerequisite services running.
In particular, this sort of trick works better with shared-IP zones, in which the network is configured from the global zone. With an exclusive-IP zone, all the networking would need to be set up inside the zone, and there's nothing running to do that for you.
Another thought I had was to use a replacement init. The downside to this is that the name of the init process is baked into the brand definition, so I would have to create a duplicate of each brand to run it like this. Just tweaking the inittab inside a zone is far more flexible.
It would be nice to have more flexibility. At the present time, I either have just init, or the whole of SMF. There's a whole range of potentially useful configurations between these extremes.
The other thing is to come up with a better name. Blank zone. Null zone. Something else?
One possibility is what I'm tentatively calling a Blank zone. That is, a zone that has nothing running. Or, more precisely, just has an init process but not the normal array of miscellaneous processes that get started up by SMF in a normal boot.
You might be tempted to use 'zoneadm ready' rather than 'zoneadm boot'. This doesn't work, as you can't get into the zone:
zlogin: login allowed only to running zones (test1 is 'ready').So you do actually need to boot the zone.
Why not simply disable the SMF services you don't need? This is fine if you still want SMF and most of the services, but SMF itself is quite a beast, and the minimal set of service dependencies is both large and extremely complex. In practice, you end up running most things just to keep the SMF dependencies happy.
Now, SMF is started by init using the following line (I've trimmed the redirections) from /etc/inittab
smf::sysinit:/lib/svc/bin/svc.startd
OK, so all we have to do is delete this entry, and we just get init. Right? Wrong! It's not quite that simple. If you try this then you get a boot failure:
INIT: Absent svc.startd entry or bad contract template. Not starting svc.startd.
Requesting maintenance mode
In practice, this isn't fatal - the zone is still running, but apart from wondering why it's behaving like this it would be nice to have the zone boot without errors.
Looking at the source for init, it soon becomes clear what's happening. The init process is now intimately aware of SMF, so essentially it knows that its only job is to get startd running, and startd will do all the work. However, it's clear from the code that it's only looking for the smf id in the first field. So my solution here is to replace startd with an infinite sleep.
smf::sysinit:/usr/bin/sleep Inf
(As an aside, this led to illumos bug 6019, as the manpage for sleep(1) isn't correct. Using 'sleep infinite' as the manpage suggests led to other failures.)
Then, the zone boots up, and the process tree looks like this:
# ptree -z test1
10210 zsched
10338 /sbin/init
10343 /usr/bin/sleep Inf
To get into the zone, you just need to use zlogin. Without anything running, there aren't the normal daemons (like sshd) available for you to connect to. It's somewhat disconcerting to type 'netstat -a' and get nothing back.
For permanent services, you could run them from inittab (in the traditional way), or have an external system that creates the zones and uses zlogin to start the application. Of course, this means that you're responsible for any required system configuration and for getting any prerequisite services running.
In particular, this sort of trick works better with shared-IP zones, in which the network is configured from the global zone. With an exclusive-IP zone, all the networking would need to be set up inside the zone, and there's nothing running to do that for you.
Another thought I had was to use a replacement init. The downside to this is that the name of the init process is baked into the brand definition, so I would have to create a duplicate of each brand to run it like this. Just tweaking the inittab inside a zone is far more flexible.
It would be nice to have more flexibility. At the present time, I either have just init, or the whole of SMF. There's a whole range of potentially useful configurations between these extremes.
The other thing is to come up with a better name. Blank zone. Null zone. Something else?
Saturday, August 01, 2015
The lunacy of -Werror
First, a little history for those of you young enough not to have lived through perl. In the perl man page, there's a comment in the BUGS section that says:
(The -w switch enables warnings about grotty code.) Unfortunately, many developers misunderstood this. They wrote their perl script, and then added the -w switch as though it was a magic bullet that fixed all the errors in your code, without even bothering to think about looking at the output it generated or - heaven forbid - actually fixing the problems. The result was that, with a CGI script, your apache error log was full of output that nobody ever read.
The correct approach, of course, is to develop with the -w switch, fix all the warnings it reports as part of development, and then turn it off. (Genuine errors will still be reported anyway, and you won't have to sift through garbage to find them, or worry about your service going down because the disk filled up.)
Move on a decade or two, and I'm starting to see a disturbing number of software packages being shipped that have -Werror in the default compilation flags. This almost always results in the build failing.
If you think about this for a moment, it should be obvious that enabling -Werror by default is a really dumb idea. There are two basic reasons:
(Having a build target that generates a warning report that you can send back to the developer would be useful, though.)
The -w switch is not mandatory.
(The -w switch enables warnings about grotty code.) Unfortunately, many developers misunderstood this. They wrote their perl script, and then added the -w switch as though it was a magic bullet that fixed all the errors in your code, without even bothering to think about looking at the output it generated or - heaven forbid - actually fixing the problems. The result was that, with a CGI script, your apache error log was full of output that nobody ever read.
The correct approach, of course, is to develop with the -w switch, fix all the warnings it reports as part of development, and then turn it off. (Genuine errors will still be reported anyway, and you won't have to sift through garbage to find them, or worry about your service going down because the disk filled up.)
Move on a decade or two, and I'm starting to see a disturbing number of software packages being shipped that have -Werror in the default compilation flags. This almost always results in the build failing.
If you think about this for a moment, it should be obvious that enabling -Werror by default is a really dumb idea. There are two basic reasons:
- Warnings are horribly context sensitive. It's difficult enough to remove all the warnings given a single fully constrained environment. As soon as you start to vary the compiler version, the platform you're building on, or the versions of the (potentially many) prerequisites you're building against, getting accidental warnings is almost inevitable. (And you can't test against all possibilities, because some of those variations might not even exist at the point of software release.)
- The warnings are only meaningful to the original developer. The person who has downloaded the code and is trying to build it has no reason to be burdened by all the warnings, let alone be inconvenienced by unnecessary build failures.
(Having a build target that generates a warning report that you can send back to the developer would be useful, though.)
Friday, July 24, 2015
boot2docker on Tribblix
Containers are the new hype, and Docker is the Poster Child. OK, I've been running containerized workloads on Solaris with zones for over a decade, so some of the ideas behind all this are good; I'm not so sure about the implementation.
The fact that there's a lot of buzz is unmistakeable, though. So being familiar with the technology can't be a bad idea.
I'm running Tribblix, so running Docker natively is just a little tricky. (Although if you actually wanted to do that, then Triton from Joyent is how to do it.)
But there's boot2docker, which allows you to run Docker on a machine - by spinning up a copy of VirtualBox for you and getting that to actually do the work. The next thought is obvious - if you can make that work on MacOS X or Windows, why not on any other OS that also supports VirtualBox?
So, off we go. First port of call is to get VirtualBox installed on Tribblix. It's an SVR4 package, so should be easy enough. Ah, but, it has special-case handling for various Solaris releases that cause it to derail quite badly on illumos.
Turns out that Jim Klimov has a patchset to fix this. It doesn't handle Tribblix (yet), but you can take the same idea - and the same instructions - to fix it here. Unpack the SUNWvbox package from datastream to filesystem format, edit the file SUNWvbox/root/opt/VirtualBox/vboxconfig.sh, replacing the lines
# S11 without 'pkg'?? Something's wrong... bail.
errorprint "Solaris $HOST_OS_MAJORVERSION detected without executable $BIN_PKG !? I are confused."
exit 1
with
# S11 without 'pkg'?? Likely an illumos variant
HOST_OS_MINORVERSION="152"
and follow Jim's instructions for updating the pkgmap, then just pkgadd from the filesystem image.
Next, the boot2docker cli. I'm assuming you have go installed already - on Tribblix, "zap install go" will do the trick. Then, in a convenient new directory,
env GOPATH=`pwd` go get github.com/boot2docker/boot2docker-cli
That won't quite work as is. There are a couple of patches. The first is to the file src/github.com/boot2docker/boot2docker-cli/virtualbox/hostonlynet.go. Look for the CreateHostonlyNet() function, and replace
out, err := vbmOut("hostonlyif", "create")
if err != nil {
return nil, err
}
with
out, err := vbmOut("hostonlyif", "create")
if err != nil {
// default to vboxnet0
return &HostonlyNet{Name: "vboxnet0"}, nil
}
The point here is that , on a Solaris platform, you always get a hostonly network - that's what vboxnet0 is - so you don't need to create one, and in fact the create option doesn't even exist so it errors out.
The second little patch is that the arguments to SSH don't quite match the SunSSH that comes with illumos, so we need to remove one of the arguments. In the file src/github.com/boot2docker/boot2docker-cli/util.go, look for DefaultSSHArgs and delete the line containing IdentitiesOnly=yes (which is the option SunSSH doesn't recognize).
Then you need to rebuild the project.
env GOPATH=`pwd` go clean github.com/boot2docker/boot2docker-cli
env GOPATH=`pwd` go build github.com/boot2docker/boot2docker-cli
Then you should be able to play around. First, download the base VM image it'll run:
./boot2docker-cli download
Configure VirtualBox
./boot2docker-cli init
Start the VM
./boot2docker-cli up
Log into it
./boot2docker-cli ssh
Once in the VM you can run docker commands (I'm doing it this way at the moment, rather than running a docker client on the host). For example
docker run hello-world
Or,
docker run -d -P --name web nginx
./boot2docker-cli down
While this is interesting, and reasonably functional, certainly to the level of being useful for testing, a sign of the churn in the current container world is that the boot2docker cli is deprecated in favour of Docker Machine, but building that looks to be rather more involved.
The fact that there's a lot of buzz is unmistakeable, though. So being familiar with the technology can't be a bad idea.
I'm running Tribblix, so running Docker natively is just a little tricky. (Although if you actually wanted to do that, then Triton from Joyent is how to do it.)
But there's boot2docker, which allows you to run Docker on a machine - by spinning up a copy of VirtualBox for you and getting that to actually do the work. The next thought is obvious - if you can make that work on MacOS X or Windows, why not on any other OS that also supports VirtualBox?
So, off we go. First port of call is to get VirtualBox installed on Tribblix. It's an SVR4 package, so should be easy enough. Ah, but, it has special-case handling for various Solaris releases that cause it to derail quite badly on illumos.
Turns out that Jim Klimov has a patchset to fix this. It doesn't handle Tribblix (yet), but you can take the same idea - and the same instructions - to fix it here. Unpack the SUNWvbox package from datastream to filesystem format, edit the file SUNWvbox/root/opt/VirtualBox/vboxconfig.sh, replacing the lines
# S11 without 'pkg'?? Something's wrong... bail.
errorprint "Solaris $HOST_OS_MAJORVERSION detected without executable $BIN_PKG !? I are confused."
exit 1
with
# S11 without 'pkg'?? Likely an illumos variant
HOST_OS_MINORVERSION="152"
and follow Jim's instructions for updating the pkgmap, then just pkgadd from the filesystem image.
Next, the boot2docker cli. I'm assuming you have go installed already - on Tribblix, "zap install go" will do the trick. Then, in a convenient new directory,
env GOPATH=`pwd` go get github.com/boot2docker/boot2docker-cli
That won't quite work as is. There are a couple of patches. The first is to the file src/github.com/boot2docker/boot2docker-cli/virtualbox/hostonlynet.go. Look for the CreateHostonlyNet() function, and replace
out, err := vbmOut("hostonlyif", "create")
if err != nil {
return nil, err
}
with
out, err := vbmOut("hostonlyif", "create")
if err != nil {
// default to vboxnet0
return &HostonlyNet{Name: "vboxnet0"}, nil
}
The point here is that , on a Solaris platform, you always get a hostonly network - that's what vboxnet0 is - so you don't need to create one, and in fact the create option doesn't even exist so it errors out.
The second little patch is that the arguments to SSH don't quite match the SunSSH that comes with illumos, so we need to remove one of the arguments. In the file src/github.com/boot2docker/boot2docker-cli/util.go, look for DefaultSSHArgs and delete the line containing IdentitiesOnly=yes (which is the option SunSSH doesn't recognize).
Then you need to rebuild the project.
env GOPATH=`pwd` go clean github.com/boot2docker/boot2docker-cli
env GOPATH=`pwd` go build github.com/boot2docker/boot2docker-cli
Then you should be able to play around. First, download the base VM image it'll run:
./boot2docker-cli download
Configure VirtualBox
./boot2docker-cli init
Start the VM
./boot2docker-cli up
Log into it
./boot2docker-cli ssh
Once in the VM you can run docker commands (I'm doing it this way at the moment, rather than running a docker client on the host). For example
docker run hello-world
Or,
docker run -d -P --name web nginx
./boot2docker-cli down
While this is interesting, and reasonably functional, certainly to the level of being useful for testing, a sign of the churn in the current container world is that the boot2docker cli is deprecated in favour of Docker Machine, but building that looks to be rather more involved.
Wednesday, July 15, 2015
How to build a server
So, you have a project and you need a server. What to do?
This is why devops is a thing, streamlining (eradicating) processes like the above.
And this is (one reason) why developers spin up machines in the cloud. It's not that the cloud is better or cheaper (because often it isn't), it's simply to avoid dealing with dinosaurs of legacy corporate IT departments which only exist to prevent their users getting work done.
My approach to this was rather different.
User: Can I have a server?
Me: Sure. What do you want to call it?
[User, stunned at not immediately being told to get lost, thinks for a moment.]
Me: That's fine. Here you go. [Types a command to create a Solaris Zone.]
Me: Engages in a few pleasantries, to delay the user for a minute or two so that the new system will be ready and booted when they get back to their desk.
- Submit a ticket requesting the server
- Have it bounced back saying your manager needs to fill in a server build request form
- Manager submits a server build request form
- Server build manager assigns the build request to a subordinate
- Server builder creates a server build workflow in the workflow tool
- A ticket is raised with the network team to assign an IP address
- A ticket is raised with the DNS team to enter the server into DNS
- A ticket is raised with the virtual team to assign resources on the VMware infrastructure
- Take part in a 1000 message 100 participant email thread of doom arguing whether you really need 16G of memory in your server
- A ticket is raised with the storage team to allocate storage resources
- Server builder manually configures the Windows DHCP server to hand out the IP address
- Virtual Machine is built
- You're notified that the server is "ready"
- Take part in a 1000 message 100 participant email thread of doom arguing that when you asked for Ubuntu that's what you actually wanted rather then the corporate standard of RHEL5
- A ticket is raised with the Database team to install the Oracle client
- Database team raise a ticket with the unix team to do the step of the oracle install that requires root privileges
- A ticket is raised with the ops team to add the server to monitoring
- A ticket is raised with your outsourced backup provider to enable backups on the server
- Take part in a 1000 message 100 participant email thread of doom over whether the system has been placed on the correct VLAN
- Submit another ticket to get the packages you need installed
- Move server to another VLAN, redoing steps 6, 7, and 11
- Submit another ticket to the storage team because they set up the NFS exports on their filers for the old IP address
This is why devops is a thing, streamlining (eradicating) processes like the above.
And this is (one reason) why developers spin up machines in the cloud. It's not that the cloud is better or cheaper (because often it isn't), it's simply to avoid dealing with dinosaurs of legacy corporate IT departments which only exist to prevent their users getting work done.
My approach to this was rather different.
User: Can I have a server?
Me: Sure. What do you want to call it?
[User, stunned at not immediately being told to get lost, thinks for a moment.]
Me: That's fine. Here you go. [Types a command to create a Solaris Zone.]
Me: Engages in a few pleasantries, to delay the user for a minute or two so that the new system will be ready and booted when they get back to their desk.
Thursday, June 11, 2015
Badly targetted advertising
The web today is essentially one big advertising stream. Everywhere you go you're bombarded by adverts.
OK, I get that it's necessary. Sites do cost money to run, people who work on them have to get paid. It might be evil, but (in the absence of an alternative funding model) it's a necessary evil.
There's a range of implementations. Some subtle, others less so. Personally, I take note of the unsubtle and brash ones, the sort that actively interfere with what I'm trying to achieve, and mark them as companies I'm less likely to do business with. The more subtle ones I tolerate as the price for using the modern web.
What is abundantly clear, though, is how much tracking of your activities goes on. For example, I needed to do some research on email suppliers yesterday - and am being bombarded with adverts for email services today. If I go away, I get bombarded with adverts for hotels at my destination. Near Christmas I get all sorts of advertising popping up based on the presents I've just purchased.
The thing is, though, that most of these adverts are wrong and pointless. The idea that because I searched for something, or visited a website on a certain subject, might indicate that I would be interested in the same things in future, is simply plain wrong.
Essentially, if I'm doing something on the web, then I have either (a) succeeded in the task at hand (bought an item, booked a hotel), or (b) failed completely. In either case, basing subsequent advertising on past activities is counterproductive.
If I've booked a hotel, then the last thing I'm going to do next is book another hotel for the same dates at the same location. More sensible behaviour for advertisers would be to prime the system to stop advertising hotels, and then advertise activities and events (for which they even know the dates) at my destination. It's likely to be more useful for me, and more likely to get a successful response for the advertiser. Likewise, once I've bought an item, stop advertising that and instead move on to advertising accessories.
And if I've failed in my objectives, ramming more of the same down my throat is going to frustrate me and remind me of the failure.
In fact, I wonder if a better targeting strategy would be to turn things around completely, and advertise random items excluding the currently targeted items. That opens up the possibility of serendipity - triggering a response that I wasn't even aware of, rather than trying to persuade me to do something I already actively wanted to do.
OK, I get that it's necessary. Sites do cost money to run, people who work on them have to get paid. It might be evil, but (in the absence of an alternative funding model) it's a necessary evil.
There's a range of implementations. Some subtle, others less so. Personally, I take note of the unsubtle and brash ones, the sort that actively interfere with what I'm trying to achieve, and mark them as companies I'm less likely to do business with. The more subtle ones I tolerate as the price for using the modern web.
What is abundantly clear, though, is how much tracking of your activities goes on. For example, I needed to do some research on email suppliers yesterday - and am being bombarded with adverts for email services today. If I go away, I get bombarded with adverts for hotels at my destination. Near Christmas I get all sorts of advertising popping up based on the presents I've just purchased.
The thing is, though, that most of these adverts are wrong and pointless. The idea that because I searched for something, or visited a website on a certain subject, might indicate that I would be interested in the same things in future, is simply plain wrong.
Essentially, if I'm doing something on the web, then I have either (a) succeeded in the task at hand (bought an item, booked a hotel), or (b) failed completely. In either case, basing subsequent advertising on past activities is counterproductive.
If I've booked a hotel, then the last thing I'm going to do next is book another hotel for the same dates at the same location. More sensible behaviour for advertisers would be to prime the system to stop advertising hotels, and then advertise activities and events (for which they even know the dates) at my destination. It's likely to be more useful for me, and more likely to get a successful response for the advertiser. Likewise, once I've bought an item, stop advertising that and instead move on to advertising accessories.
And if I've failed in my objectives, ramming more of the same down my throat is going to frustrate me and remind me of the failure.
In fact, I wonder if a better targeting strategy would be to turn things around completely, and advertise random items excluding the currently targeted items. That opens up the possibility of serendipity - triggering a response that I wasn't even aware of, rather than trying to persuade me to do something I already actively wanted to do.
Sunday, June 07, 2015
Building LibreOffice on Tribblix
Having decent tools is necessary for an operating system to be useful, and one of the basics for desktop use is an office suite - LibreOffice being the primary candidate.
Unfortunately, there aren't prebuilt binaries for any of the Solaris or illumos distros. So I've been trying to build LibreOffice from source for a while. Finally, I have a working build on Tribblix.
This is what I did. Hopefully it will be useful to other distros. This is just a straight dump of my notes.
First, you'll need java (the jdk), and the perl Archive::Zip module. You'll need boost, and harfbuzz with the icu extensions. Plus curl, hunspell, cairo, poppler, neon.
Then you'll need to build (look on this page for links to some of this stuff):
If you don't tell it otherwise, LibreOffice will download these and try to build them itself. And generally these have problems building cleanly, which it's fairly easy to fix while building them in isolation, but would be well nigh impossible when they're buried deep inside the LibreOffice build system
For librevenge, pass --disable-werror to configure.
For libmspub, replace the call to pow() in src/lib/MSPUBMetaData.cpp with std::pow().
For libmspub, remove zlib from the installed pc file (Tribblix, and some of the other illumos other distros, don't supply a pkgconfig file for zlib).
For liborcus, run the following against all the Makefiles that the configure step generates:
gsed -i 's:-DMDDS_HASH_CONTAINER_BOOST:-pthreads -DMDDS_HASH_CONTAINER_BOOST:'
For mdds, make sure you have a PATH that has the gnu install ahead of the system install program when running make install.
For ixion, it's a bit more involved. You need some way of getting -pthreads past configure *and* make. For configure, I used:
env boost_cv_pthread_flag=-pthreads CFLAGS="-O -pthreads" CPPFLAGS="-pthreads" CXXFLAGS="-pthreads" configure ...
and for make:
gmake MDDS_CFLAGS=-pthreads
For orcus, it looks to pkgconfig to find zlib, so you'll need to prevent that:
env ZLIB_CFLAGS="-I/usr/include" ZLIB_LIBS="-lz" configure ...
For libvisio, replace the call to pow() in src/lib/VSDMetaData.cpp with std::pow().
For libvisio, remove zlib and libxml-2.0 from the installed pc file.
If you want to run a parallel make, don't use gmake 3.81. Version 4.1 is fine.
With all those installed you can move on to LibreOffice.
Unpack the main tarball.
chmod a+x bin/unpack-sources
mkdir -p external/tarballs
and then symlink or copy the other tarballs (help, translations, dictionaries) into external/tarballs (otherwise, it'll try downloading them again).
Download and run this script to patch the builtin version of glew.
Edit the following files:
And replace "LINUX" with "SOLARIS". That part of the makefiles is needed on all unix-like systems, not just Linux.
In the file
sc/source/core/tool/interpr1.cxx
replace the call to pow() on line 3160 with std::pow()
In the file
sal/qa/inc/valueequal.hxx
replace the call to pow() on line 87 with std::pow()
In the file
include/vcl/window.hxx
You'll need to #undef TRANSPARENT before it's used (otherwise, it picks up a rogue definition from the system).
And you'll need to create a compilation symlink:
mkdir -p instdir/program
ln -s libGLEW.so.1.10 instdir/program/libGLEW.so
This is the configure command I used:
env PATH=/usr/gnu/bin:$PATH \
./configure --prefix=/usr/versions/libreoffice-44 \
--with-system-hunspell \
--with-system-curl \
--with-system-libpng \
--with-system-clucene=no \
--with-system-libxml \
--with-system-jpeg=no \
--with-system-cairo \
--with-system-harfbuzz \
--with-gnu-cp=/usr/gnu/bin/cp \
--with-gnu-patch=/usr/gnu/bin/patch \
--disable-gconf \
--without-doxygen \
--with-system-openssl \
--with-system-nss \
--disable-python \
--with-system-expat \
--with-system-zlib \
--with-system-poppler \
--disable-postgresql-sdbc \
--with-system-icu \
--with-system-neon \
--disable-odk \
--disable-firebird-sdbc \
--without-junit \
--disable-gio \
--with-jdk-home=/usr/jdk/latest \
--disable-gltf \
--with-system-libwps \
--with-system-libwpg \
--with-system-libwpd \
--with-system-libmspub \
--with-system-librevenge \
--with-system-orcus \
--with-system-mdds \
--with-system-libvisio \
--with-help \
--with-vendor="Tribblix" \
--enable-release-build=yes \
--with-parallelism=8
and then to make:
env LD_LIBRARY_PATH=/usr/lib/mps:`pwd`/instdir/ure/lib:`pwd`/instdir/sdk/lib:`pwd`/instdir/program \
PATH=/usr/gnu/bin:$PATH \
/usr/gnu/bin/make -k build
(Using 'make build' is supposed to avoid the checks, many of which fail. You'll definitely need to run 'make -k' with a parallel build, because otherwise some of the test failures will stop the build before all the other parallel parts of the build have finished.)
Then create symlinks for all the .so files in /usr/lib/mps in instdir/program, and instdir/program/soffice should start.
Unfortunately, there aren't prebuilt binaries for any of the Solaris or illumos distros. So I've been trying to build LibreOffice from source for a while. Finally, I have a working build on Tribblix.
This is what I did. Hopefully it will be useful to other distros. This is just a straight dump of my notes.
First, you'll need java (the jdk), and the perl Archive::Zip module. You'll need boost, and harfbuzz with the icu extensions. Plus curl, hunspell, cairo, poppler, neon.
Then you'll need to build (look on this page for links to some of this stuff):
- cppunit-1.13.2
- librevenge-0.0.2
- libwpd-0.10.0
- libwpg-0.3.0
- libmspub-0.1.2
- libwps-0.3.1
- mdds_0.11.2
- libixion-0.7.0
- liborcus-0.7.0
- libvisio-0.1.1
If you don't tell it otherwise, LibreOffice will download these and try to build them itself. And generally these have problems building cleanly, which it's fairly easy to fix while building them in isolation, but would be well nigh impossible when they're buried deep inside the LibreOffice build system
For librevenge, pass --disable-werror to configure.
For libmspub, replace the call to pow() in src/lib/MSPUBMetaData.cpp with std::pow().
For libmspub, remove zlib from the installed pc file (Tribblix, and some of the other illumos other distros, don't supply a pkgconfig file for zlib).
For liborcus, run the following against all the Makefiles that the configure step generates:
gsed -i 's:-DMDDS_HASH_CONTAINER_BOOST:-pthreads -DMDDS_HASH_CONTAINER_BOOST:'
For mdds, make sure you have a PATH that has the gnu install ahead of the system install program when running make install.
For ixion, it's a bit more involved. You need some way of getting -pthreads past configure *and* make. For configure, I used:
env boost_cv_pthread_flag=-pthreads CFLAGS="-O -pthreads" CPPFLAGS="-pthreads" CXXFLAGS="-pthreads" configure ...
and for make:
gmake MDDS_CFLAGS=-pthreads
For orcus, it looks to pkgconfig to find zlib, so you'll need to prevent that:
env ZLIB_CFLAGS="-I/usr/include" ZLIB_LIBS="-lz" configure ...
For libvisio, replace the call to pow() in src/lib/VSDMetaData.cpp with std::pow().
For libvisio, remove zlib and libxml-2.0 from the installed pc file.
If you want to run a parallel make, don't use gmake 3.81. Version 4.1 is fine.
With all those installed you can move on to LibreOffice.
Unpack the main tarball.
chmod a+x bin/unpack-sources
mkdir -p external/tarballs
and then symlink or copy the other tarballs (help, translations, dictionaries) into external/tarballs (otherwise, it'll try downloading them again).
Download and run this script to patch the builtin version of glew.
Edit the following files:
- svx/Executable_gengal.mk
- sw/Executable_tiledrendering.mk
- vcl/Executable_ui-previewer.mk
- desktop/Library_sofficeapp.mk
- vcl/Library_vcl.mk
And replace "LINUX" with "SOLARIS". That part of the makefiles is needed on all unix-like systems, not just Linux.
In the file
sc/source/core/tool/interpr1.cxx
replace the call to pow() on line 3160 with std::pow()
In the file
sal/qa/inc/valueequal.hxx
replace the call to pow() on line 87 with std::pow()
In the file
include/vcl/window.hxx
You'll need to #undef TRANSPARENT before it's used (otherwise, it picks up a rogue definition from the system).
And you'll need to create a compilation symlink:
mkdir -p instdir/program
ln -s libGLEW.so.1.10 instdir/program/libGLEW.so
This is the configure command I used:
env PATH=/usr/gnu/bin:$PATH \
./configure --prefix=/usr/versions/libreoffice-44 \
--with-system-hunspell \
--with-system-curl \
--with-system-libpng \
--with-system-clucene=no \
--with-system-libxml \
--with-system-jpeg=no \
--with-system-cairo \
--with-system-harfbuzz \
--with-gnu-cp=/usr/gnu/bin/cp \
--with-gnu-patch=/usr/gnu/bin/patch \
--disable-gconf \
--without-doxygen \
--with-system-openssl \
--with-system-nss \
--disable-python \
--with-system-expat \
--with-system-zlib \
--with-system-poppler \
--disable-postgresql-sdbc \
--with-system-icu \
--with-system-neon \
--disable-odk \
--disable-firebird-sdbc \
--without-junit \
--disable-gio \
--with-jdk-home=/usr/jdk/latest \
--disable-gltf \
--with-system-libwps \
--with-system-libwpg \
--with-system-libwpd \
--with-system-libmspub \
--with-system-librevenge \
--with-system-orcus \
--with-system-mdds \
--with-system-libvisio \
--with-help \
--with-vendor="Tribblix" \
--enable-release-build=yes \
--with-parallelism=8
and then to make:
env LD_LIBRARY_PATH=/usr/lib/mps:`pwd`/instdir/ure/lib:`pwd`/instdir/sdk/lib:`pwd`/instdir/program \
PATH=/usr/gnu/bin:$PATH \
/usr/gnu/bin/make -k build
(Using 'make build' is supposed to avoid the checks, many of which fail. You'll definitely need to run 'make -k' with a parallel build, because otherwise some of the test failures will stop the build before all the other parallel parts of the build have finished.)
Then create symlinks for all the .so files in /usr/lib/mps in instdir/program, and instdir/program/soffice should start.
Sunday, May 31, 2015
What sort of DevOps are you?
What sort of DevOps are you? Can you even define DevOps?
Nobody
really knows what DevOps is, there are almost as many definitions as
practitioners. Part of the problem is that the name tends to get tacked onto anything to make it seem trendy. (The same way that "cloud" has been abused.)
Whilst stereotypical, I tend to separate the field into
the puritans and the pragmatists.
The puritanical vision of
DevOps is summarized by the mantra of "Infrastructure as Code". In this
world, it's all about tooling (often, although not exclusively, based around configuration
management).
From the pragmatist viewpoint, it's rather
about driving organizational and cultural change to enable people to
work together to benefit the business, instead of competing with each
other to benefit their own department or themselves. This is largely a
reaction to legacy departmental silos that simply toss tasks over the
wall to each other.
I'm firmly in the pragmatist camp.
Tooling helps, but you can use all the tools in the world badly if you
don't have the correct philosophy and culture.
I
see a lot of emphasis being placed on tooling. Partly this is because
in the vendor space, tooling is all there is - vendors frame the
discussion in terms of how tooling (in particular, their tool) can
improve your business. I don't have a problem with vendors doing this,
they have to sell stuff after all, but I regard conflating their
offerings with DevOps in the large, or even defining DevOps as a discipline, as misleading at best.
Another worrying
trend (I'm seeing an awful lot of this from recruiters, not necessarily practitioners)
is the stereotypical notion that DevOps is still about getting rid of
legacy operations and having developers carry the pager. This again
starts out in terms of a conflict between Dev and Ops and, rather than
resolving it by combining forces, simply throws one half of the team
away.
Where I do see a real problem is that smaller organizations
might start out with only developers, and then struggle to adopt
operational practices. Those of us with a background in operations need
to find a way to integrate with development-led teams and organizations.
(The same problem arises when you have a subversive development team in
a large business that's going round the back of traditional operations,
and eventually find that they need operational support.)
I was encouraged that the recent DOXLON meetup had a couple of really interesting talks about culture. Practitioners know that this is important, we really need to get the word out.
Where have all the SSDs gone?
My current and previous laptop - that's a 3-year timespan - both had an internal SSD rather than rotating rust. The difference between those and prior systems was like night and day - instant-on, rather than the prior experience of making a cup of coffee while waiting for the old HDD system to stagger into life.
My current primary desktop system is also SSD based. Power button to fully booted is a small number of seconds. Applications are essentially instant - certainly compared to startup times for things like firefox that used to be double-digit seconds before it was ready to go.
(This startup speed changes usage patterns. Who really needs suspend/resume when the system boots in the time it takes to settle comfortably in your chair?)
So I was a little surprised, while browsing in a major high street electronics retailer, to find hardly any evidence of SSDs. Every desktop system had an HDD. Almost all the laptops were HDD based. A couple of the all-in-ones had hybrid drives. SSDs were conspicuous by their absence.
I had actually noticed this trend while looking online. I've just checked the desktops on the Dell site, and there's no sign of a system with an SSD option.
Curious, I asked the shop assistant, who replied that SSDs were far too expensive.
I'm not sure I buy the cost argument. An SSD actually costs the same as an HDD - at least, the range of prices is exactly the same. So the prices will stay unchanged, but obviously the capacity will be quite a bit less. And it looks like the sales pitch is about capacity.
But even there, the capacity numbers are meaningless. It's purely bragging rights, disconnected from reality. With any of the HDD options, you're looking at hundreds of thousands of songs or pictures. Very few typical users will need anything like that much - and if you do, you're going to need to look at redundancy or backup. And with media streaming and cloud-based backup, local storage is more a liability than an asset.
So, why such limited penetration of SSDs into the home computing market?
My current primary desktop system is also SSD based. Power button to fully booted is a small number of seconds. Applications are essentially instant - certainly compared to startup times for things like firefox that used to be double-digit seconds before it was ready to go.
(This startup speed changes usage patterns. Who really needs suspend/resume when the system boots in the time it takes to settle comfortably in your chair?)
So I was a little surprised, while browsing in a major high street electronics retailer, to find hardly any evidence of SSDs. Every desktop system had an HDD. Almost all the laptops were HDD based. A couple of the all-in-ones had hybrid drives. SSDs were conspicuous by their absence.
I had actually noticed this trend while looking online. I've just checked the desktops on the Dell site, and there's no sign of a system with an SSD option.
Curious, I asked the shop assistant, who replied that SSDs were far too expensive.
I'm not sure I buy the cost argument. An SSD actually costs the same as an HDD - at least, the range of prices is exactly the same. So the prices will stay unchanged, but obviously the capacity will be quite a bit less. And it looks like the sales pitch is about capacity.
But even there, the capacity numbers are meaningless. It's purely bragging rights, disconnected from reality. With any of the HDD options, you're looking at hundreds of thousands of songs or pictures. Very few typical users will need anything like that much - and if you do, you're going to need to look at redundancy or backup. And with media streaming and cloud-based backup, local storage is more a liability than an asset.
So, why such limited penetration of SSDs into the home computing market?
Subscribe to:
Posts (Atom)