Sunday, May 20, 2012

Vendor Stack vs build your own

Operating System distributions are getting ever more bloated, including more and more packages. While this reduces the need for the end user to build their own software, does it actually eliminate the need for systems administrators to manage the software on their systems?

I would argue that in many cases having software you rely on as part of the operating system is actually a hindrance rather than a help.

For example, much of my work involves building web servers. These include Java, Apache, Tomcat, MySQL and the like. And, when we deploy systems, we explicitly use our own private copies of each component in the stack.

This is a deliberate choice. And there are several reasons behind it.

For one, it ensures that we have exactly the build time options and, (in the case of apache) the modules we need. Often we require slightly different choices than the defaults.

Keeping everything separate insulates us from vendor changes - we're completely unaffected by a vendor applying a harmful patch, or from "upgrading" to a newer version of the components.

A corollary to this is that we can patch and update the OS on our servers with much more freedom, as we don't have to worry about the effect on our application stack at all. It goes the other way - we can update the components in our stack without having to touch the OS.

It also means that we can move applications between systems with different patch levels, able to go to both newer and older systems easily - and indeed, between different operating systems and chip architectures.

As we use Solaris zones extensively, this also allows us to have different zones with the components at different revision levels.

With all this, we simply don't need a vendor to supply the various components of the stack. If the OS needs them for something else then fine, we just don't want to get involved. In some cases (databases are the prime example) we go to some effort to make sure they don't get installed at all, because some poor user using the wrong version is likely to get hurt.

All this makes me wonder why operating system vendors bother with maintaining central copies of software that are no use to us. Indeed, many application stacks on unix systems come with their own private copies of the components they need, for exactly the reasons I outlined above. (I've lost count of the number of times something from Sun installed it's own private copy of Java.)

(While the above considers one particular aspect of servers, it's equally true of desktops. Perhaps even more so, as many operating system releases are primarily defined by how incompatible their user interface is to previous releases.)


Anonymous said...

Interesting, because when I was a system admin, I felt much the same way. I remember being upset when Solaris decided to include perl, because it was going to conflict with whatever version we had on our systems at the time.

And yet, if an OS doesn't include some of these things (a webserver, language runtime, whatever) it gets pummeled in reviews. It seems reviewers are mainly interested in how quickly can they get up and running and how convenient it is for them. I wonder if OS development has been optimizing for the wrong things here.

Fazal Majid said...

Yes. I have my own build scripts (built around GNU Make) to generate the 400+ dependencies in /usr/local, that I have been accumulating over the last 12 years. I started this just to get a repeatable build process for PHP back when all modules had to be statically linked in.

The problem is some vendor packages like the obsolete version of Python that ships with Solaris can't be removed as they are dependencies for pkg. Similarly you can't always override /usr/lib with /usr/local/lib using crle as system commands like ssh (depends on OpenSSL) start randomly crashing. Our very own version of Windows' DLL hell, I suppose.