Whither open source?

According to the Free Software Definition, there are 4 essential freedoms:

• The freedom to run the program as you wish, for any purpose (freedom 0).
• The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Access to the source code and an open-source license are necessary preconditions for software freedom, but not sufficient.

And, unfortunately, we are living in an era where it is becoming ever more difficult to exercise the freedoms listed above.

Consider freedom 0. In the past, essentially all free software ran perfectly well on essentially every hardware platform and operating system. At the present time, much of what claims to be open-source software is horrendously platform-specific - sometimes by ignorance (I don't expect every developer to be able to test on all platforms), but there's a disturbing trend of deliberately excluding non-preferred platforms.

There is increasing use of new languages and runtimes, which are often very restricted in terms of platform support. If you look at some of the languages like Node.JS, Go, and Rust, you'll see that they explicitly target the common hardware architectures (x86 and ARM), deliberately and consciously excluding other platforms. Add to that the trend for self-referential bootstrapping (where you need X to build X) and you can see other platforms frozen out entirely.

So, much of freedom 0 has been emasculated. What of freedom 1?

Yes, I might be able to look at the source code. (Although, in many cases, it is opaque and undocumented.) And I might be able to crack open an editor and type in a modification. But actually being able to use that modification is a whole different ball game.

Actually building software from source often enters you into a world of pain and frustration. Fighting your way through Dependency Hell, struggling with arcane and opaque build systems, becoming frustrated with the vagaries of the autotools (remember how the configure script works - it makes a bunch of random and unsubstantiated guesses about the state of your system, the ignores half the results, and often needs explicitly overriding, making a mockery of the "auto" part), only to discover that "works on my system" is almost a religion.

Current trends like Docker make this problem worse. Rather than having to pay lip-service to portability by having to deal with the vagaries of multiple distributions, authors can now restrict the target environment even more narrowly - "works in my docker image" is the new normal. (I've had some developers come out and say this explicitly.)

The conclusion: open-source software is becoming increasingly narrow and proprietary, denying users the freedoms they deserve.

Blank Zones

I've been playing around with various zone configurations on Tribblix. This is going beyond the normal sparse-root, whole-root, partial-root, and various other installation types, into thinking about other ways you can actually use zones to run software.

One possibility is what I'm tentatively calling a Blank zone. That is, a zone that has nothing running. Or, more precisely, just has an init process but not the normal array of miscellaneous processes that get started up by SMF in a normal boot.

You might be tempted to use 'zoneadm ready' rather than 'zoneadm boot'. This doesn't work, as you can't get into the zone:

zlogin: login allowed only to running zones (test1 is 'ready').

So you do actually need to boot the zone.

Why not simply disable the SMF services you don't need? This is fine if you still want SMF and most of the services, but SMF itself is quite a beast, and the minimal set of service dependencies is both large and extremely complex. In practice, you end up running most things just to keep the SMF dependencies happy.

Now, SMF is started by init using the following line (I've trimmed the redirections) from /etc/inittab

smf::sysinit:/lib/svc/bin/svc.startd

OK, so all we have to do is delete this entry, and we just get init. Right? Wrong! It's not quite that simple. If you try this then you get a boot failure:

INIT: Absent svc.startd entry or bad contract template.  Not starting svc.startd.
Requesting maintenance mode

In practice, this isn't fatal - the zone is still running, but apart from wondering why it's behaving like this it would be nice to have the zone boot without errors.

Looking at the source for init, it soon becomes clear what's happening. The init process is now intimately aware of SMF, so essentially it knows that its only job is to get startd running, and startd will do all the work. However, it's clear from the code that it's only looking for the smf id in the first field. So my solution here is to replace startd with an infinite sleep.

smf::sysinit:/usr/bin/sleep Inf

(As an aside, this led to illumos bug 6019, as the manpage for sleep(1) isn't correct. Using 'sleep infinite' as the manpage suggests led to other failures.)

Then, the zone boots up, and the process tree looks like this:

# ptree -z test1
10210 zsched
  10338 /sbin/init
    10343 /usr/bin/sleep Inf

To get into the zone, you just need to use zlogin. Without anything running, there aren't the normal daemons (like sshd) available for you to connect to. It's somewhat disconcerting to type 'netstat -a' and get nothing back.

For permanent services, you could run them from inittab (in the traditional way), or have an external system that creates the zones and uses zlogin to start the application. Of course, this means that you're responsible for any required system configuration and for getting any prerequisite services running.

In particular, this sort of trick works better with shared-IP zones, in which the network is configured from the global zone. With an exclusive-IP zone, all the networking would need to be set up inside the zone, and there's nothing running to do that for you.

Another thought I had was to use a replacement init. The downside to this is that the name of the init process is baked into the brand definition, so I would have to create a duplicate of each brand to run it like this. Just tweaking the inittab inside a zone is far more flexible.

It would be nice to have more flexibility. At the present time, I either have just init, or the whole of SMF. There's a whole range of potentially useful configurations between these extremes.

The other thing is to come up with a better name. Blank zone. Null zone. Something else?

The lunacy of -Werror

First, a little history for those of you young enough not to have lived through perl. In the perl man page, there's a comment in the BUGS section that says:

The -w switch is not mandatory.

(The -w switch enables warnings about grotty code.) Unfortunately, many developers misunderstood this. They wrote their perl script, and then added the -w switch as though it was a magic bullet that fixed all the errors in your code, without even bothering to think about looking at the output it generated or - heaven forbid - actually fixing the problems. The result was that, with a CGI script, your apache error log was full of output that nobody ever read.

The correct approach, of course, is to develop with the -w switch, fix all the warnings it reports as part of development, and then turn it off. (Genuine errors will still be reported anyway, and you won't have to sift through garbage to find them, or worry about your service going down because the disk filled up.)

Move on a decade or two, and I'm starting to see a disturbing number of software packages being shipped that have -Werror in the default compilation flags. This almost always results in the build failing.

If you think about this for a moment, it should be obvious that enabling -Werror by default is a really dumb idea. There are two basic reasons:

1. Warnings are horribly context sensitive. It's difficult enough to remove all the warnings given a single fully constrained environment. As soon as you start to vary the compiler version, the platform you're building on, or the versions of the (potentially many) prerequisites you're building against, getting accidental warnings is almost inevitable. (And you can't test against all possibilities, because some of those variations might not even exist at the point of software release.)
2. The warnings are only meaningful to the original developer. The person who has downloaded the code and is trying to build it has no reason to be burdened by all the warnings, let alone be inconvenienced by unnecessary build failures.

To be clear, I'm not saying - at all - that the original developer shouldn't be using -Werror and fixing all the warnings (and you might want to enable it for your CI builds to be sure you catch regressions), but distributing code with it enabled is simply being rude to your users.

(Having a build target that generates a warning report that you can send back to the developer would be useful, though.)

boot2docker on Tribblix

Containers are the new hype, and Docker is the Poster Child. OK, I've been running containerized workloads on Solaris with zones for over a decade, so some of the ideas behind all this are good; I'm not so sure about the implementation.

The fact that there's a lot of buzz is unmistakeable, though. So being familiar with the technology can't be a bad idea.

I'm running Tribblix, so running Docker natively is just a little tricky. (Although if you actually wanted to do that, then Triton from Joyent is how to do it.)

But there's boot2docker, which allows you to run Docker on a machine - by spinning up a copy of VirtualBox for you and getting that to actually do the work. The next thought is obvious - if you can make that work on MacOS X or Windows, why not on any other OS that also supports VirtualBox?

So, off we go. First port of call is to get VirtualBox installed on Tribblix. It's an SVR4 package, so should be easy enough. Ah, but, it has special-case handling for various Solaris releases that cause it to derail quite badly on illumos.

Turns out that Jim Klimov has a patchset to fix this. It doesn't handle Tribblix (yet), but you can take the same idea - and the same instructions - to fix it here. Unpack the SUNWvbox package from datastream to filesystem format, edit the file SUNWvbox/root/opt/VirtualBox/vboxconfig.sh, replacing the lines

             # S11 without 'pkg'?? Something's wrong... bail.
             errorprint "Solaris $HOST_OS_MAJORVERSION detected without executable $BIN_PKG !? I are confused."
             exit 1

with

         # S11 without 'pkg'?? Likely an illumos variant
         HOST_OS_MINORVERSION="152"

and follow Jim's instructions for updating the pkgmap, then just pkgadd from the filesystem image.

Next, the boot2docker cli. I'm assuming you have go installed already - on Tribblix, "zap install go" will do the trick. Then, in a convenient new directory,

env GOPATH=`pwd` go get github.com/boot2docker/boot2docker-cli

That won't quite work as is. There are a couple of patches. The first is to the file src/github.com/boot2docker/boot2docker-cli/virtualbox/hostonlynet.go. Look for the CreateHostonlyNet() function, and replace

    out, err := vbmOut("hostonlyif", "create")
    if err != nil {
        return nil, err
    }

with

    out, err := vbmOut("hostonlyif", "create")
    if err != nil {
               // default to vboxnet0
        return &HostonlyNet{Name: "vboxnet0"}, nil
    }

The point here is that , on a Solaris platform, you always get a hostonly network - that's what vboxnet0 is - so you don't need to create one, and in fact the create option doesn't even exist so it errors out.

The second little patch is that the arguments to SSH don't quite match the SunSSH that comes with illumos, so we need to remove one of the arguments. In the file src/github.com/boot2docker/boot2docker-cli/util.go, look for DefaultSSHArgs and delete the line containing IdentitiesOnly=yes (which is the option SunSSH doesn't recognize).

Then you need to rebuild the project.

env GOPATH=`pwd` go clean github.com/boot2docker/boot2docker-cli
env GOPATH=`pwd` go build github.com/boot2docker/boot2docker-cli

Then you should be able to play around. First, download the base VM image it'll run:

./boot2docker-cli download

Configure VirtualBox

./boot2docker-cli init

Start the VM

./boot2docker-cli up

Log into it

./boot2docker-cli ssh

Once in the VM you can run docker commands (I'm doing it this way at the moment, rather than running a docker client on the host). For example

docker run hello-world

Or,

docker run -d -P --name web nginx

Shut the VM down

./boot2docker-cli down

While this is interesting, and reasonably functional, certainly to the level of being useful for testing, a sign of the churn in the current container world is that the boot2docker cli is deprecated in favour of Docker Machine, but building that looks to be rather more involved.

How to build a server

So, you have a project and you need a server. What to do?

1. Submit a ticket requesting the server
2. Have it bounced back saying your manager needs to fill in a server build request form
3. Manager submits a server build request form
4. Server build manager assigns the build request to a subordinate
5. Server builder creates a server build workflow in the workflow tool
6. A ticket is raised with the network team to assign an IP address
7. A ticket is raised with the DNS team to enter the server into DNS
8. A ticket is raised with the virtual team to assign resources on the VMware infrastructure
9. Take part in a 1000 message 100 participant email thread of doom arguing whether you really need 16G of memory in your server
10. A ticket is raised with the storage team to allocate storage resources
11. Server builder manually configures the Windows DHCP server to hand out the IP address
12. Virtual Machine is built
13. You're notified that the server is "ready"
14. Take part in a 1000 message 100 participant email thread of doom arguing that when you asked for Ubuntu that's what you actually wanted rather then the corporate standard of RHEL5
15. A ticket is raised with the Database team to install the Oracle client
16. Database team raise a ticket with the unix team to do the step of the oracle install that requires root privileges
17. A ticket is raised with the ops team to add the server to monitoring
18. A ticket is raised with your outsourced backup provider to enable backups on the server
19. Take part in a 1000 message 100 participant email thread of doom over whether the system has been placed on the correct VLAN
20. Submit another ticket to get the packages you need installed
21. Move server to another VLAN, redoing steps 6, 7, and 11
22. Submit another ticket to the storage team because they set up the NFS exports on their filers for the old IP address

There's actually a few more steps in many cases, but I think you get the idea.

This is why devops is a thing, streamlining (eradicating) processes like the above.

And this is (one reason) why developers spin up machines in the cloud. It's not that the cloud is better or cheaper (because often it isn't), it's simply to avoid dealing with dinosaurs of legacy corporate IT departments which only exist to prevent their users getting work done.

My approach to this was rather different.

User: Can I have a server?

Me: Sure. What do you want to call it?

[User, stunned at not immediately being told to get lost, thinks for a moment.]

Me: That's fine. Here you go. [Types a command to create a Solaris Zone.]

Me: Engages in a few pleasantries, to delay the user for a minute or two so that the new system will be ready and booted when they get back to their desk.

Badly targetted advertising

The web today is essentially one big advertising stream. Everywhere you go you're bombarded by adverts.

OK, I get that it's necessary. Sites do cost money to run, people who work on them have to get paid. It might be evil, but (in the absence of an alternative funding model) it's a necessary evil.

There's a range of implementations. Some subtle, others less so. Personally, I take note of the unsubtle and brash ones, the sort that actively interfere with what I'm trying to achieve, and mark them as companies I'm less likely to do business with. The more subtle ones I tolerate as the price for using the modern web.

What is abundantly clear, though, is how much tracking of your activities goes on. For example, I needed to do some research on email suppliers yesterday - and am being bombarded with adverts for email services today. If I go away, I get bombarded with adverts for hotels at my destination. Near Christmas I get all sorts of advertising popping up based on the presents I've just purchased.

The thing is, though, that most of these adverts are wrong and pointless. The idea that because I searched for something, or visited a website on a certain subject, might indicate that I would be interested in the same things in future, is simply plain wrong.

Essentially, if I'm doing something on the web, then I have either (a) succeeded in the task at hand (bought an item, booked a hotel), or (b) failed completely. In either case, basing subsequent advertising on past activities is counterproductive.

If I've booked a hotel, then the last thing I'm going to do next is book another hotel for the same dates at the same location. More sensible behaviour for advertisers would be to prime the system to stop advertising hotels, and then advertise activities and events (for which they even know the dates) at my destination. It's likely to be more useful for me, and more likely to get a successful response for the advertiser. Likewise, once I've bought an item, stop advertising that and instead move on to advertising accessories.

And if I've failed in my objectives, ramming more of the same down my throat is going to frustrate me and remind me of the failure.

In fact, I wonder if a better targeting strategy would be to turn things around completely, and advertise random items excluding the currently targeted items. That opens up the possibility of serendipity - triggering a response that I wasn't even aware of, rather than trying to persuade me to do something I already actively wanted to do.

Building LibreOffice on Tribblix

Having decent tools is necessary for an operating system to be useful, and one of the basics for desktop use is an office suite - LibreOffice being the primary candidate.

Unfortunately, there aren't prebuilt binaries for any of the Solaris or illumos distros. So I've been trying to build LibreOffice from source for a while. Finally, I have a working build on Tribblix.

This is what I did. Hopefully it will be useful to other distros. This is just a straight dump of my notes.

First, you'll need java (the jdk), and the perl Archive::Zip module. You'll need boost, and harfbuzz with the icu extensions. Plus curl, hunspell, cairo, poppler, neon.

Then you'll need to build (look on this page for links to some of this stuff):

• cppunit-1.13.2
• librevenge-0.0.2
• libwpd-0.10.0
• libwpg-0.3.0
• libmspub-0.1.2
• libwps-0.3.1
• mdds_0.11.2
• libixion-0.7.0
• liborcus-0.7.0
• libvisio-0.1.1

If you don't tell it otherwise, LibreOffice will download these and try to build them itself. And generally these have problems building cleanly, which it's fairly easy to fix while building them in isolation, but would be well nigh impossible when they're buried deep inside the LibreOffice build system

For librevenge, pass --disable-werror to configure.

For libmspub, replace the call to pow() in src/lib/MSPUBMetaData.cpp with std::pow().

For libmspub, remove zlib from the installed pc file (Tribblix, and some of the other illumos other distros, don't supply a pkgconfig file for zlib).

For liborcus, run the following against all the Makefiles that the configure step generates:

gsed -i 's:-DMDDS_HASH_CONTAINER_BOOST:-pthreads -DMDDS_HASH_CONTAINER_BOOST:'

For mdds, make sure you have a PATH that has the gnu install ahead of the system install program when running make install.
For ixion, it's a bit more involved. You need some way of getting -pthreads past configure *and* make. For configure, I used:

env boost_cv_pthread_flag=-pthreads CFLAGS="-O -pthreads" CPPFLAGS="-pthreads" CXXFLAGS="-pthreads" configure ...

and for make:

gmake MDDS_CFLAGS=-pthreads

For orcus, it looks to pkgconfig to find zlib, so you'll need to prevent that:

 env ZLIB_CFLAGS="-I/usr/include" ZLIB_LIBS="-lz" configure ...

For libvisio, replace the call to pow() in src/lib/VSDMetaData.cpp with std::pow().

For libvisio, remove zlib and libxml-2.0 from the installed pc file.

If you want to run a parallel make, don't use gmake 3.81. Version 4.1 is fine.

With all those installed you can move on to LibreOffice.

Unpack the main tarball.

chmod a+x bin/unpack-sources
mkdir -p external/tarballs

and then symlink or copy the other tarballs (help, translations, dictionaries) into external/tarballs (otherwise, it'll try downloading them again).

Download and run this script to patch the builtin version of glew.

Edit the following files:

• svx/Executable_gengal.mk
• sw/Executable_tiledrendering.mk
• vcl/Executable_ui-previewer.mk
• desktop/Library_sofficeapp.mk
• vcl/Library_vcl.mk

And replace "LINUX" with "SOLARIS". That part of the makefiles is needed on all unix-like systems, not just Linux.

In the file

sc/source/core/tool/interpr1.cxx

replace the call to pow() on line 3160 with std::pow()

In the file

sal/qa/inc/valueequal.hxx

replace the call to pow() on line 87 with std::pow()

In the file

include/vcl/window.hxx

You'll need to #undef TRANSPARENT before it's used (otherwise, it picks up a rogue definition from the system).

And you'll need to create a compilation symlink:

mkdir -p  instdir/program
ln -s libGLEW.so.1.10 instdir/program/libGLEW.so

This is the configure command I used:

env PATH=/usr/gnu/bin:$PATH \
./configure --prefix=/usr/versions/libreoffice-44 \
--with-system-hunspell \
--with-system-curl \
--with-system-libpng \
--with-system-clucene=no \
--with-system-libxml \
--with-system-jpeg=no \
--with-system-cairo \
--with-system-harfbuzz \
--with-gnu-cp=/usr/gnu/bin/cp \
--with-gnu-patch=/usr/gnu/bin/patch \
--disable-gconf \
--without-doxygen \
--with-system-openssl \
--with-system-nss \
--disable-python \
--with-system-expat \
--with-system-zlib \
--with-system-poppler \
--disable-postgresql-sdbc \
--with-system-icu \
--with-system-neon \
--disable-odk \
--disable-firebird-sdbc \
--without-junit \
--disable-gio \
--with-jdk-home=/usr/jdk/latest \
--disable-gltf \
--with-system-libwps \
--with-system-libwpg \
--with-system-libwpd \
--with-system-libmspub \
--with-system-librevenge \
--with-system-orcus \
--with-system-mdds \
--with-system-libvisio \
--with-help \
--with-vendor="Tribblix" \
--enable-release-build=yes \
--with-parallelism=8

and then to make:

env LD_LIBRARY_PATH=/usr/lib/mps:`pwd`/instdir/ure/lib:`pwd`/instdir/sdk/lib:`pwd`/instdir/program \
PATH=/usr/gnu/bin:$PATH \
/usr/gnu/bin/make -k build

(Using 'make build' is supposed to avoid the checks, many of which fail. You'll definitely need to run 'make -k' with a parallel build, because otherwise some of the test failures will stop the build before all the other parallel parts of the build have finished.)

Then create symlinks for all the .so files in /usr/lib/mps in instdir/program, and instdir/program/soffice should start.

What sort of DevOps are you?

What sort of DevOps are you? Can you even define DevOps?

 

Nobody really knows what DevOps is, there are almost as many definitions as practitioners. Part of the problem is that the name tends to get tacked onto anything to make it seem trendy. (The same way that "cloud" has been abused.)

 

Whilst stereotypical, I tend to separate the field into the puritans and the pragmatists.

 

The puritanical vision of DevOps is summarized by the mantra of "Infrastructure as Code". In this world, it's all about tooling (often, although not exclusively, based around configuration management).

 

From the pragmatist viewpoint, it's rather about driving organizational and cultural change to enable people to work together to benefit the business, instead of competing with each other to benefit their own department or themselves. This is largely a reaction to legacy departmental silos that simply toss tasks over the wall to each other.

 

I'm firmly in the pragmatist camp. Tooling helps, but you can use all the tools in the world badly if you don't have the correct philosophy and culture.

 

I see a lot of emphasis being placed on tooling. Partly this is because in the vendor space, tooling is all there is - vendors frame the discussion in terms of how tooling (in particular, their tool) can improve your business. I don't have a problem with vendors doing this, they have to sell stuff after all, but I regard conflating their offerings with DevOps in the large, or even defining DevOps as a discipline, as misleading at best.

 

Another worrying trend (I'm seeing an awful lot of this from recruiters, not necessarily practitioners) is the stereotypical notion that DevOps is still about getting rid of legacy operations and having developers carry the pager. This again starts out in terms of a conflict between Dev and Ops and, rather than resolving it by combining forces, simply throws one half of the team away.

 

Where I do see a real problem is that smaller organizations might start out with only developers, and then struggle to adopt operational practices. Those of us with a background in operations need to find a way to integrate with development-led teams and organizations. (The same problem arises when you have a subversive development team in a large business that's going round the back of traditional operations, and eventually find that they need operational support.)

 

I was encouraged that the recent DOXLON meetup had a couple of really interesting talks about culture. Practitioners know that this is important, we really need to get the word out.

Where have all the SSDs gone?

My current and previous laptop - that's a 3-year timespan - both had an internal SSD rather than rotating rust. The difference between those and prior systems was like night and day - instant-on, rather than the prior experience of making a cup of coffee while waiting for the old HDD system to stagger into life.

My current primary desktop system is also SSD based. Power button to fully booted is a small number of seconds. Applications are essentially instant - certainly compared to startup times for things like firefox that used to be double-digit seconds before it was ready to go.

(This startup speed changes usage patterns. Who really needs suspend/resume when the system boots in the time it takes to settle comfortably in your chair?)

So I was a little surprised, while browsing in a major high street electronics retailer, to find hardly any evidence of SSDs. Every desktop system had an HDD. Almost all the laptops were HDD based. A couple of the all-in-ones had hybrid drives. SSDs were conspicuous by their absence.

I had actually noticed this trend while looking online. I've just checked the desktops on the Dell site, and there's no sign of a system with an SSD option.

Curious, I asked the shop assistant, who replied that SSDs were far too expensive.

I'm not sure I buy the cost argument. An SSD actually costs the same as an HDD - at least, the range of prices is exactly the same. So the prices will stay unchanged, but obviously the capacity will be quite a bit less. And it looks like the sales pitch is about capacity.

But even there, the capacity numbers are meaningless. It's purely bragging rights, disconnected from reality. With any of the HDD options, you're looking at hundreds of thousands of songs or pictures. Very few typical users will need anything like that much - and if you do, you're going to need to look at redundancy or backup. And with media streaming and cloud-based backup, local storage is more a liability than an asset.

So, why such limited penetration of SSDs into the home computing market?

How illumos sets the keyboard type

It was recently pointed out that, while the Tribblix live image prompts you for the keyboard type, the installer doesn't carry that choice through to the installed system.

Which is right. I hadn't written any code to do that, and hadn't even thought of it. (And as I personally use a US unix keyboard then the default is just fine for me, so hadn't noticed the omission.)

So I set out to discover how to fix this. And it's a twisty little maze.

The prompt when the live image boots comes from the kbd command, called as 'kbd -s'. It does the prompt and sets the keyboard type - there's nothing external involved.

So to save that, we have to query the system. To do this, run kbd with the -t and -l arguments

# kbd -t
USB keyboard

# kbd -l
type=6
layout=33 (0x21)
delay(ms)=500
rate(ms)=40

OK, in the -l output type=6 means a USB keyboard, so that matches up. These are defined in <kbd.h>

#define KB_KLUNK        0x00            /* Micro Switch 103SD32-2 */

#define KB_VT100        0x01            /* Keytronics VT100 compatible */

#define KB_SUN2         0x02            /* Sun-2 custom keyboard */

#define KB_VT220        0x81            /* Emulation VT220 */

#define KB_VT220I       0x82            /* International VT220 Emulation */

#define KB_SUN3         3               /* Type 3 Sun keyboard */

#define KB_SUN4         4               /* Type 4 Sun keyboard */

#define KB_USB          6               /* USB keyboard */

#define KB_PC           101             /* Type 101 AT keyboard */

#define KB_ASCII        0x0F            /* Ascii terminal masquerading as kbd */

That handles the type, and basically everything today is a type 6.

Next, how is the keyboard layout matched. That's the 33 in the output. The layouts are listed in the file

/usr/share/lib/keytables/type_6/kbd_layouts

Which are a key-value map of name to number. So what we have is:

US-English=33

And if you check the source for the kbd command, 33 is the default.

Note that the numbers that kbd -s generates to prompt the user with have absolutely nothing to do with the actual type - the prompt just makes up an incrementing sequence of numbers.

So, how is this then loaded into a new system? Well, that's the keymap service, which has a method script that then calls

/usr/lib/set_keyboard_layout

(yes, it's a twisty maze). That script gets the layout by calling eeprom like so:

/usr/sbin/eeprom keyboard-layout

Now, usually you'll see:

keyboard-layout=Unknown

which is fair enough, I haven't set it.

On x86, eeprom is emulated, using the file

/boot/solaris/bootenv.rc

So, to copy the keyboard layout from the live instance to the newly

installed system, I need to:

1. Get the layout from kbd -l

2. Parse /usr/share/lib/keytables/type_6/kbd_layouts to get the name that corresponds to that number.

3. Poke that back into eeprom by inserting an entry into bootenv.rc

Oof. This is arcane.

Tribblix scorecard

I was just tidying up some of the documentation and scripts I used to create Tribblix, including pushing many of the components up to repositories on github.  One of the files I found was a quick sketch of my initial aims for the distro. I'm going to list those below, with a commentary as to how well I've done.

It must be possible to install a zone without requiring external resources

Success. On Tribblix, you don't need a repo to install a whole or sparse root zone. You can also install a partial-root zone that has a subset of the global zone's packages. (Clearly, adding software to a zone that isn't installed in the global zone will require an external source of packages, although it's possible to pre-cache them in the global zone.)

It must be possible to minimize a system, removing all extraneous software including perl, python, java.

Almost successful. There's no need in Tribblix for any of perl, python, or java. There are still pieces of illumos that can drag in perl in particular, but there is work to eliminate those as well. (One corollorary to this aim is that you can freely and arbitrarily replace any of perl, python, or java by completely different versions without constraint.)

It should be possible to upgrade a system from the live CD

In theory, this could be made to work trivially. The live CD contains both a minimalist installed image and additional packages. During installation, the minimalist image is simply copied to the destination, and additional packages added separately. As a space optimization, I don't put the packages in the minimalist image on the iso, as they would never be used during normal installation.

It should be possible to use any filesystem of the user's choice (although zfs is preferred)

Success. Although the default file system at install is zfs, the live CD comes with a ufs install script (which does work, although it doesn't get uch testing) which should be extensible to other file systems. In addition, I've built systems running with an nfs root file system.

It must be possible to select which versions of utilities are to be installed; and to have multiple versions installed simultaneously. It should be possible to define one of those installed versions as the default.

Partially successful. The way this is implemented is that certain utilities are installed under /usr/versions, and it's possible to have different versions co-exist. I've tried various levels of granularity, so it's a work in progress. For example, OpenJDK has a different package for each update (so you can have 7u71 and 7u75 installed together), whereas for python I just have 2.7 (for all values of 2.7.x) and 3.4. There are symlinks in the regular locations so they're in the regular search path, which can be modified to refer to a different version if the administrator so desires, but there isn't a built-in mechanism such as mediators - by default, the most recently installed version wins.

It must be possible to install by functionality rather than requiring users to understand packages. (Probably implemented as package groups or clusters.)

Success. I define overlays of useful functionality to hide packages, and the zap utility, the installer, and the zone tools are largely based around overlays as the fundamental unit of installation.

It should be possible to use small system configurations. Requiring over 1G of memory just to boot isn't acceptable.

Success. Tribblix will install and run in 512M (with limitations - making heavy use of java or firefox will be painful). I have booted the installer in slightly less, and run in 256M (it's pretty easy to try this in an emulator such as VirtualBox), but the way the installer works, by loading a full image archive into memory, will limit truly small configurations, as the root archive itself is almost 200M.

It should be possible to customize what's installed from the live CD (to a limited extent, not arbitrarily)

Success. You can define the installed software profile by choosing which overlays should be installed.

Overall, all of those initial aims have been met, or could easily be met by making trivial adjustments. I think that's a pretty good scorecard overall.

In addition to the general aims for Tribblix, I wrote down a list of milestones against which to measure progress. The milestones were more about implementation details rather than general aims (things like "migrate from gcc3 to gcc4", "build illumos from scratch", "become self-hosting", "create an upgrade mechanism", and "make a sparc version", or "have LibreOffice working"). That's where the "milestone" nomenclature in the Tribblix releases comes from, although I never specified in which order I would attack the milestones, it just makes for a convenient "yes, I got that working" point at which I might put out a new iso for download.

In terms of progress against those milestones, about the only one left to do that's structural is the upgrade capability. It's almost there, but needs more polish. Much of the rest is adding applications. So it's at this point that I can really start to think about producing something that I can call 1.0.

Setting up Logical Domains, part 2

In part 1, I talked about the server side of Logical Domains. This time, I'll cover how I set up a guest domain.

First, I have created a zfs pool called storage on the host, and I'm going to present a zvol (or maybe several) to the guests.

I'm going to create a little domain called ldom1.

ldm add-domain ldom1
ldm set-core 1 ldom1
ldm add-memory 4G ldom1
ldm add-vnet vnet1 primary-vsw0 ldom1

Now create and add a virtual disk. Create a dataset for the ldom and a 24GB volume inside it, add it to the storage service and set it as the boot device.

zfs create storage/ldom1
zfs create -V 24gb storage/ldom1/disk0
ldm add-vdsdev /dev/zvol/dsk/storage/ldom1/disk0 ldom1_disk0@primary-vds0
ldm add-vdisk disk0 ldom1_disk0@primary-vds0 ldom1
ldm set-var auto-boot\?=true ldom1
ldm set-var boot-device=disk0 ldom1

Then bind resources, list, and start:

ldm bind-domain ldom1
ldm list-domain ldom1
ldm start-domain ldom1

You can connect to the console by looking at the number under CONS in the list-domain output:

# ldm list-domain ldom1
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
ldom1            bound      ------  5000    8     4G
 
# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Connecting to console "ldom1" in group "ldom1" ....
Press ~? for control options ..


T5140, No Keyboard
Copyright (c) 1998, 2014, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.33.6.e, 4096 MB memory available, Serial #83586105.
Ethernet address 0:14:4f:fb:6c:39, Host ID: 84fb6c39.



Boot device: disk0  File and args:
Bad magic number in disk label
ERROR: /virtual-devices@100/channel-devices@200/disk@0: Can't open disk label package

ERROR: boot-read fail

Evaluating:

Can't open boot device

{0} ok

Attempt to boot it off the net and it says:

Requesting Internet Address for 0:14:4f:f9:87:f9

which doesn't match the MAC address in the console output. So if you want to jumpstart the box, and that's easy enough, you need to do a 'boot net' first to get the actual MAC address of the LDOM to add it to your jumpstart server.

To add an iso image to boot from:

ldm add-vdsdev options=ro /var/tmp/img.iso iso@primary-vds0
ldm add-vdisk iso iso@primary-vds0 ldom1

At the ok prompt, you can issue the 'show-disks' command to see what disk devices are present. To boot off the iso:

boot /virtual-devices@100/channel-devices@200/disk@1:f

And it should work. This is how I've been testing the Tribblix images for SPARC, by the way.

Setting up Logical Domains, part 1

I've recently been playing with Logical Domains (aka LDOMs, aka Oracle VM Server for SPARC). For those unfamiliar with the technology, it's a virtualization framework built into the hardware of pretty well all current SPARC systems, more akin to VMware than Solaris zones.

For more information, see here, here, or here.

First, why use it? Especially when Solaris has zones. The answer is that it addresses a different set of problems. Individual LDOMs are more independent and much more isolated than zones. You can partition resources more cleanly, and different LDOMs don't have to be at the same patch level (to my mind, it's not that you can have a different level of patches in each LDOM that matters, but that you can do maintenance of each LDOM to different schedules that matters). One key advantage I find is that the virtual switch you set up with LDOMs is much better at dealing with complex network configuration (I have hosts scattered across maybe dozens of VLANs, trying to fake that up on Solaris 10 is a bit of a bind). And some applications don't really get on with zones - I would build new systems around zones, but ill-understood and poorly documented legacy systems might be easier to drop inside an LDOM.

That dealt with, here's how I setup up one of my machines (a T5140, as practice for live deployment on a bunch of replacement T4-1 systems) as an LDOM host. I'll cover setting up the guest side in a second post.

Make sure the firmware is current - these are the minimum revs:

T2 - 7.4.5
T3 - 8.3
T4 - 8.4.2c

Then install the LDOM software.

cd /var/tmp
unzip p17291713_31_SOLARIS64.zip
cd OVM_Server_SPARC-3_1/Install
./install-ldm

You'll get asked if you want to launch the configuration assistant after installation. I chose n, and you can run ldmconfig at any later time. (If you want - it's best not to.)

Now need to apply the LDOM patch

svcadm disable -s ldmd
patchadd 150817-02
svcadm enable ldmd

Verify things are working as expected:

# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-c--  SP      128   32544M   0.1%  16m

You should see the primary domain.

The next step is to establish default services and configure the control domain.

The necessary services are:

Virtual console
Virtual disk server
Virtual switch service

ldm add-vds primary-vds0 primary
ldm add-vcc port-range=5000-5100 primary-vcc0 primary
ldm add-vsw net-dev=nxge0 primary-vsw0 primary

verify with

ldm list-services primary

And we need to limit the control domain to a limited set of resources. The way I do this (this is just a personal view) is, on a system with N cores, to define N units each with 1 core with 1/N of the total memory. Assign one of those units to the primary domain, and then build the guest domains with 1 or more of those units (sizing as necessary - note that you can resize them on the fly so you don't have to get it perfect first time around). You can get very detailed and start allocating down to individual threads and specific amounts of memory, but it's so much better to just keep it simple.

For a T2/T2+/T3 you need to futz with crypto MAUs. This is unnecessary on later systems.

To show:

ldm list -o crypto primary

To add

ldm set-mau 1 primary

To set the CPU resources of the control domain

ldm set-core 1 primary

(I want to just set 1 core. Allocation best done by cores, not threads.)

Start the reconfig

ldm start-reconf primary

Fix the memory

ldm set-memory 4G primary

Save the config to the SP

ldm add-config initial

verify the config has been saved

ldm list-config

Reboot to activate

shutdown -y -g0 -i6

OK, so that creates a 1-core (8-thread) 4G control domain. And that all seems to work.

Next steps are to configure networking and enable terminal services. From the console (as you're reconfiguring the primary network):

ifconfig vsw0 plumb
ifconfig nxge0 down unplumb
ifconfig nxge0 inet6 down unplumb
ifconfig vsw0 inet 172.18.1.128 netmask 255.255.255.0 broadcast 172.18.1.255 up
ifconfig vsw0 inet6 plumb up
mv /etc/hostname.nxge0 /etc/hostname.vsw0
mv /etc/hostname6.nxge0 /etc/hostname6.vsw0

For a T4, replace nxge with igb.

At this point, you have a machine with minimal resources assigned to the primary domain, which looks for all the world like a regular Solaris box, ready to create guest domains using the remaining resources.

Tribblix progress

I recently put out a Milestone 12 image for Tribblix.

It updates illumos, built natively on Tribblix. There's been a bit of discussion recently about whether illumos needs actual releases, as opposed to being continuously updated. It doesn't have releases, so when I come to make a Tribblix release I simply check out the current gate, build, and package it. After all, it's supposed to be ready to ship at any time.

Note that I don't maintain a fork of illumos-gate, I build it essentially as-is. This is the same for all the components I build for Tribblix - I keep true to unmodified upstream as much as possible.

The one change I have made is to SVR4 packaging. I've removed the dependency on openssl and wanboot (bug #5188), which is a good thing. It means that you can't use signed SVR4 packages, but I've never encountered one. Nor can pkgadd now directly retrieve a package via http, but the implementation via wanboot was spectacularly dire, and you're much better off using curl or wget, which allows proper repository management (as zap does). Packaging is a little quicker now, but this change also makes it much easier to update openssl in future (it's difficult to update something your packaging system is linked against).

Tribblix is now firmly committed to gcc4 (as opposed to the old gcc3 in OpenSolaris). I've rebuilt gcc to fix visibility support. If you've ever seen 'warning: visibility attribute not supported in this configuration' then you'll have stumbled across this. Basically, you need to ensure objdump is found during the gcc build - either by making sure it's in the path or by setting OBJDUMP to point to it.

I've added a new style of zones - alternate root zones. These are sparse root zones, but instead of inheriting from the global zone you can use an alternate installed image. More on that later.

There's the usual slew of updates to various packages, including the obviously sensitive bash and openssl.

There's an interesting fix to python. I put software that might come in multiple versions underneath /usr/versions and use symlinks so that applications can be found in the normal locations. Originally, /usr/bin/python was a symlink that went to ../versions/python-x.y.x/bin/python. This works fine most of the time. However, if you called it as /bin/python it couldn't find its modules, so the symlink has to be ../../usr/versions/python-x.y.x/bin/python which makes things work as desired.

The package catalogs now contain package sizes and checksums, allowing verification of downloaded packages. I need to update zap to actually use this data, and to retry or resume failed or incomplete downloads. (It's a shame that curl doesn't automatically resume incomplete downloads the way that wget does.)

At a future milestone, upgrades will be supported (regular package updates have worked for a while, I'm talking about a whole distro upgrade here). It's possible to upgrade by hand already, but it requires a few extra workarounds (such as forcing postremove scripts to always exit 0) to make it work properly. I've got most of the preparatory work in place now. Upgrading zones looks a whole lot more complicated, though (and I haven't really seen it done well elsewhere).

Now, off to work on the next update.

Building illumos-gate on Tribblix

Until recently, I've used an OpenIndiana system to build the illumos packages that go into Tribblix. Clearly this is less than ideal - it would be nice to be able to build all of Tribblix on Tribblix.

This has always been a temporary expedient. So, here's how to build illumos-gate on Tribblix.

(Being able to do so is also good in that it increases the number of platforms on which a vanilla illumos-gate can be built.)

First, download and install Tribblix (version 0m10 or later). I recommend installing the kitchen-sink.

Then, if you're running 0m10, apply some necessary updates. As root:

    zap refresh-overlays
    zap refresh-catalog
    zap update-overlay develop
    zap uninstall TRIBdev-object-file
    zap install TRIBdev-object-file

This won't be necessary in future releases, but I found some packaging issues which interfered with the illumos build (although other software doesn't bother), including some symlinks so that various utilities are where illumos-gate expects.

I run the build in a zone. It requires a non-standard environment, and using a zone means that I don't have to corrupt the global zone, and I can repeatably guarantee that I get a correct build environment.

Then, install a build zone. This will be a whole-root zone in which we copy the develop overlay from the global zone, and add the illumos-build overlay into the zone. (It will download the packages for the illumos-build overlay the first time you do this, but will cache them so if you repeat this later - and I tend to create build zones more or less at will - it won't have to). You need to specify a zone name and give it an IP address.

    zap create-zone -t whole \
        -z il-build -i 172.18.1.206 \
        -o develop -O illumos-build

This will automatically boot the zone, you just have to wait until SMF has finished initialising.

Configure the zone so it can resolve names from DNS:

    cp /etc/resolv.conf /export/zones/il-build/root/etc/
    cp /etc/nsswitch.dns /export/zones/il-build/root/etc/nsswitch.conf

Go into the zone

  zlogin il-build

In the zone, create a user to do the build, and a couple of hacky fixes

    rm /usr/bin/cpp
    cd /usr/bin ; ln -s ../gnu/bin/xgettext gxgettext

(The first is a bug in my gcc, the latter is a Makefile bug.)

If you want to build with SMB printing

    zap install TRIBcups

Now, as the user, the build largely follows the normal instructions: you can use git to clone illumos-gate, unpack the closed bins, copy illumos.sh and nightly.sh, and edit illumos.sh to customize the build.

There are a few things you need to do to get a successful build. The first is to add the following to illumos.sh

export SUPPRESSPKGDEP=true

this is necessary as the IPS dependency step uses the installed image; as Tribblix uses SVR4 packaging, there isn't one. You can still create the IPS repo (and I do, as that's what I then turn into SVR4 packages), but the dependency step needs to be suppressed.

If you want to build with CUPS, then you'll need to have installed cups, and you'll need to patch smb. Alternatively, avoid pulling in CUPS by adding this to illumos.sh:

export ENABLE_SMB_PRINTING='#'

As Tribblix uses newer glib, the API has changed slightly and hal uses the old API. There is a proper fix, but you can simply:

gsed -i '/g_type_init/d' usr/src/cmd/hal/hald/hald.c

Note that this means that you won't be able to run the hal components on a system with a downrev glib.

Then you should be able to run a build:

time ./nightly.sh illumos.sh

The build should be clean (I see ELF runtime attribute warnings, all coming from glib and ffi, but those don't actually matter, and I'm not sure illumos should be complaining about errors in its external dependencies anyway).

Software verification of SVR4 packages with pkgchk

On Solaris (and Tribblix) you can use the pkgchk command to verify that the contents of a software package are correctly installed.

The simplest invocation is to give pkgchk the name of a package:

pkgchk SUNWcsl

I would expect SUNWcsl to normally validate cleanly. Whereas something like SUNWcsr will tend to produce lots of output as it contains lots of configuration files that get modified. (Use the -n flag to suppress most of the noise.

If you want to check individual files, then you can use

pkgchk -p /usr/bin/ls

or (and I implemented this as part of the OpenSolaris project) you can feed a list of files on stdin:

find /usr/bin -mtime -150 | pkgchk -i -

However, it turns out that there's a a snag with the basic usage of pkgchk to analyze a package, in that it will trust the contents file - both for the list of files in the package, and for their attributes.

Modifying the list of files can be a result of using installf and removef. For example, I delete some of the junk out of /usr/ucb (such as /usr/ucb/cc so as to be sure no poor unfortunate user can ever run it), and use removef to clean up the contents file. A side-effect of this is that pkgchk won't normally be able to detect that those files are missing.

Modifying file attributes can be the result of a second package installing the same pathname with different attributes. Having multiple packages deliver a directory is common, but you can also have multiple packages own a file. Whichever package was installed last gets to choose which attributes are correct, and the normal pkgchck is blind to any changes as a result.

There's a trick to get round this. From Solaris 10, the original package metadata (and unmodified copies of editable files) are kept. Each package has a directory in /var/sadm/pkg, and in each of those you'll find a save directory. This is used when installing zones, so you get a pristine copy. However, you can also use the pkgmap file to verify a package:

pkgchk -m /var/sadm/pkg/SUNWscpu/save/pspool/SUNWscpu/pkgmap

and this form of usage will detect files that have been removed or modified by tools that are smart enough to update the contents file.

(Because those save files are used by zones, you'll find they don't exist in a zone because they wouldn't be needed there. So this trick only works in a global zone, or you need to manually copy the pkgmap file.)

Partial root zones

In Tribblix, I support sparse-root and whole-root zones, which work largely the same way as in Solaris 10.

The implementation of zone creation is rather different. The original Solaris implementation extended packaging - so the packaging system, and every package, had to be zone-aware. This is clearly unsustainable. (Unfortunately, the same mistake was made when IPS was introduced.)

Apart from creating work, this approach limits flexibility - in order to innovate with zones, for example by adding new types, you have to extend the packaging system, and then modify every package in existence.

The approach taken by Tribblix is rather different. Instead of baking zone architecture into packaging, packaging is kept dumb and the zone creation scripts understand how packages are put together.

In particular, the decision as to whether a given file is present in a zone (and how it ends up there) is not based on package attributes, but is a simple pathname filter. For example, files under /kernel never end up in a zone. Files under /usr might be copied (for a whole-root zone) or loopback mounted (for a sparse-root zone). If it's under /var or /etc, you get a fresh copy. And so on. But the decision is based on pathname.

It's not just the files within packages that get copied. The package metadata is also copied; the contents file is simply filtered by pathname - and that's how the list of files to copy is generated. This filtering takes place during zone creation, and is all done by the zone scripts - the packaging tools aren't invoked (one reason why it's so quick). The scripts, if you want to look, are at /usr/lib/brand/*/pkgcreatezone.

In the traditional model, the list of installed packages in the zone is (initially) identical to that in the global zone. For a sparse-root zone, you're pretty much stuck with that. For a whole-root zone, you can add and remove packages later.

I've been working on some alternative models for zones in Tribblix that add more flexibility to zone creation. These will appear in upcoming releases, but I wanted to talk about the technology.

The first of these is what you might call a partial-root zone. This is similar to a whole-root zone in the sense that you get an independent copy, rather than being loopback mounted. And, it's using the same TRIBwhole brand. The difference is that you can specify a subset of the overlays present in the global zone to be installed in the zone. For example, you would use the following install invocation:

zoneadm -z myzone install -o developer

and only the developer overlay (and the overlays it depends on) will be installed in the zone.

This is still a copy - the installed files in the global zone are the source of the files that end up in the zone, so there's still no package installation, no need for repository access, and it's pretty quick.

This is still a filter, but you're now filtering both on pathname and package name.

As for package metadata, for partial-root zones, references to the packages that don't end up being used are removed.

That's the subset variant. The next obvious extension is to be able to specify additional packages (or, preferably, overlays) to be installed at zone creation time. That does require an additional source of packages - either a repository or a local cache - which is why I treat it as a logically distinct operation.

Time to get coding.

Cloud analogies: Food As A Service

There's a recurring analogy of Cloud as utility, such as electrical power. I'm not convinced by this, and regard a comparison of the Cloud with the restaurant trade as more interesting. Read on...

Few IT departments build their own hardware, in the same way that few people grow their own food or keep their own livestock. Most buy from a supplier, in the same way that most buy food from a supermarket.

You could avoid cooking by eating out for every meal. Food as a Service, in current IT parlance.

The Cloud shares other properties with a restaurant. It operates on demand. It's self service, in the sense that anyone can walk in and order - you don't have to be a chef. There's a fixed menu of dishes, and portion sizes are fixed. It deals with wide fluctuations of usage throughout the day. For basic dishes, it can be more expensive than cooking at home. It's elastic, and scales, whereas most people would struggle if 100 visitors suddenly dropped by for dinner.

There's a wide choice of restaurants. And a wide variety of pricing models to match - Prix Fixe, a la carte, all you can eat.

Based on this analogy, the current infatuation with moving everything to the cloud would be the same as telling everybody that they shouldn't cook at home, but should always order in or eat out. You no longer need a kitchen, white goods, or utensils, nor do you need to retain any culinary skills.

Sure, some people do eat primarily at a basic burger bar. Some eat out all the time. Some have abandoned the kitchen. Is it appropriate for everyone?

Many people go out to eat not necessarily to avoid preparing their own food, but to eat dishes they cannot prepare at home, to try something new, or for special occasions.

In other words, while you can eat out for every meal, Food as a Service really comes into its own when it delivers capabilities beyond that of your own kitchen. Whether that be in the expertise of its staff, the tools in its kitchens, or the special ingredients that it can source, a restaurant can take your tastebuds places that your own kitchen can't.

As for the lunacy that is Private Cloud, that's really like setting up your own industrial kitchen and hiring your own chefs to run it.

Slimming down logstash

Following on from my previous post on logstash, it rapidly becomes clear that the elasticsearch indices grow rather large.

After a very quick look, it was obvious that some of the fields I was keeping were redundant or unnecessary.

For example, why keep the pathname of the log file itself? It doesn't change over time, and you can work out the name of the file easily (if you ever wanted it, and I can't see why you ever would - if you wanted to identify a source, that ought to be some other piece of data you create).

Also, why keep the full log message? You've parsed it, broken it up, and stored the individual fields you're interested in. So why keep the whole thing, a duplicate of the information you're already storing?

With that in mind, I used a mutate clause to remove the file name and the original log entry, like so:

  mutate {
     remove_field => "path"
     remove_field => "message"
  }

After this simple change, the daily elasticsearch indices on the first system I tried this on shrank from 4.5GB to 1.6GB - almost a factor of 3. Definitely worthwhile, and there are benefits in terms of network traffic, search performance, elasticsearch memory utilization, and capacity for future growth as well.

Zone logs and logstash

Today I was playing with logstash, with the plan to produce a real-time scrolling view of our web traffic.

It's easy enough. Run a logstash shipper on each node, feed everything into redis, get logstash to pull from redis into elasticsearch, then run the logstash front-end and use Kibana to create a dashboard.

Then the desire for efficiency strikes. We're running Solaris zones, and there are a lot of them. Each logstash instance takes a fair chunk of memory, so it seems like a waste to run one in each zone.

So what I wanted to do was run a single copy of logstash in the global zone, and get it to read all the zone logs, yet present the data just as though it had been run in the zone.

The first step was to define which logs to read. The file input can take wildcards, leading to a simple pattern:

input {
  file {
    type => "apache"
    path => "/storage/*/opt/proquest/*/apache/logs/access_log"
  }
}

There's a ZFS pool storage, each zone has a zfs file system named after the zone. So the name of the zone is the directory under /storage. So I can pick out the name of the zone and put it into a variable called zonename like so:

  grok {
    type => "apache"
    match => ["path","/storage/%{USERNAME:zonename}/%{GREEDYDATA}"]
  }

(If it looks odd to use the USERNAME pattern, the naming rules for our zones happen to be the same as for user names, so I use an existing pattern rather than define a new one.)

I then want the host entry associated with this log to be that of the zone, rather than the default of the global zone. So I mutate the host entry:

  mutate {
    type => "apache"
    replace => [ "host","%{zonename}.our.company.name" ]
  }

And that's pretty much it. It's very simple, but most of the documentation I could find was incorrect in the sense that it applied to old versions of logstash.

There were a couple of extra pieces of information that I then found it useful to add. The simplest was to duplicate the original host entry into a servername, so I can aggregate all the traffic associated with a physical host. The second was to pick out the website name from the zone name (in this case, the zone name is the short name of the website, with a suffix appended to distinguish the individual zones).

  grok {
    type => "apache"
    match => ["zonename","%{WORD:sitename}-%{GREEDYDATA}"]
  }

Then sitename contains the short name of the site, again allowing me to aggregate the statistics from all the zones that serve that site.