Wednesday, June 07, 2006

That's Reliable?

I was just looking at a Yankee Group report on server reliability, as reported by Yahoo!.

Now it's nice to hear that they think Solaris is winning on reliability. I knew that :-)

However, going beyond the headline they find:
  • 3-5 failures per server per year
  • 10-19.5 hours of downtime per server per year
Ouch!

Of course, that's server uptime, not service uptime. With a decent architecture you would have some backup so that the service would be available even if a server failed. And servers do fail, no matter how good they are, or need maintenance work.

But whatever, I don't regard 99.8% availabilty as anything like good. In fact, it's terrible.

My own experience is that Solaris is pretty damn reliable. Much better than the figures quoted, at any rate. And Windows servers themselves don't seem to be too bad (although they do seem vulnerable to major corruption events which, while rare, involve significant outage), although PC networks overall seem very fragile. Linux I've found to be less robust, with older versions simply wedging and hanging regularly (something that I believe has been dramatically improved), but I suspect a lot of Linux problems are due to people believing the myth that it's free and will run on any old piece of junk hardware, and so they use junk hadware and don't manage it properly - with predictable consequences.

The other aspect of system reliability is applications and, quite frankly, application reliability is often simply not up to scratch.

Friday, May 26, 2006

Lapping it up

My intention to learn some new tricks has been going quite well.

I've been working with PHP on a little project of mine. It's a system monitoring application - data about the health of systems is inserted into a database and reports displayed on a web page.

It was written using JSP, and that works extremely well up to a point. But it doesn't really need any specific technology - there's nothing in JSP that is especially suited to the task, which is fairly simple. Rewriting it in PHP therefore looked fairly easy, and having a real project is a good way to learn something new.

And it worked very well. PHP is very easy to develop in, and I was able to implement some new features as well.

So how would I compare PHP and JSP? Each has advantages and disadvantages:

PHP is quicker to get going in, so putting together simple prototypes and building on them is slightly easier.

I found it easier to make silly mistakes in PHP - the language is a bit more forgiving, so that some typos that would have generated a failure in JSP get through and you end up with something that doesn't quite work right. You need to impose a bit more discipline with PHP.

PHP doesn't need a huge java process to run a servlet engine. This has always bothered me - the overhead of a servlet container, just to dynamically create a few web pages, is considerable.

The really big advantage of PHP, though, is that it's very easy to create images on the fly. The GD extension is pretty well standard, and works (although not capable of horizontal dashed line - doh!). I've tried various ways of creating graphs using JSP and, while it's possible, it's neither easy nor pleasant. There's clearly for an opening here for a dead-easy JSP image generator to be created (or, if it exists, to come out from wherever it's been hiding).

The downside to PHP was the length of time it took to put the stack together. Tomcat is a zip file you unzip, and you're basically done. PHP makes the common mistake of using autotools and is a right pain in the neck to configure and build correctly as a result. (Note that its not PHP's fault; it's autoconf.)

So that project is almost done, and while I wouldn't claim to be fluent in PHP, at least I'm capable of asking directions in the language. Now on to ruby on rails when I get some more free time...

OpenSolaris contributions

As Jim has just noted, some of my contributions to OpenSolaris recently got putback.

This is great. It's great that Sun - as an organization - gives external contributors like myself the opportunity to put changes into Solaris. It's great that individuals within Sun (thanks Dave!) take on the time and effort required to integrate the changes into the codebase.

It's taken a while to get going. I was starting to work on fixes during the pilot stage of the OpenSolaris project, over a year ago. The fixes just putback aren't those (although I am looking at getting that work integrated too). This set of fixes to the install consolidation was actually due to something I was trying to do at work, and pkgchk wasn't cooperating. In the old days, I could have spent days fighting with support trying to persuade them that (a) it was a problem, and (b) that it might be worth fixing. Instead of which I can now get in there and fix the thing properly.

I have some other fixes planned for the Solaris package tools as well, with several possible performance improvements having been identified.

Actually getting the fixes into OpenSolaris isn't hard (the Sun sponsor does most of the legwork). It would do no harm to start off simple, just to run through the process (I didn't, of course, and it was the first external contribution to the install consolidation as well, so it was a learning experience for us).

Monday, May 15, 2006

Old dog, new tricks

I've decided it's time to learn a few new tricks.

The first trick I'm trying to learn is PHP. I'm used to java and JSP pages (although these modern new-fangled frameworks do nothing for me), but there seem to lots of web-based applications out there using PHP, and if I'm going to install and maintain some of them, I ought to understand how the language works.

The second trick is Ruby on Rails. The promise of simplicity is very appealing to me. That agrees with my systems philosophy - KISS.

(More generally, the more completely you can understand a system, the better the chance you have of making it work properly.)

After a few trails and tribulations, I've now got both software stacks up and running. Off to try a few examples...

Flimsy fans?

Anyone having trouble with the main fans ("rear fan 1") in a W2100z?

We've got 3 W2100z boxes at work, from two different batches. Even the oldest one is only a year old.

Now all 3 have had the main fan die, needing hardware replacement. The system is dead while waiting for a new fan to show up, which is somewhat annoying.

(No, it's not the old BIOS problem.)

Still, my old SunBlade 2000 had its disk die today, so sparc boxes aren't immune. This means I'm temporarily without a system running a nevada build - I shall have to go root around in the dumpster tomorrow to see if there are any spare FCAL drives left.

dumb

Every so often I have it reinforced just how stupid using autoconf can be.

I maintain my own software stack in /usr/local - applications that don't come with the operating system, or different version of ones that do.

I was just updating gzip. So the ./configure script, when invoked with --help, says:

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc. You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.


OK, so I do ./configure, make, make install.

What the heck?!?!?!

It's installed it in /usr, overwriting the system copies!

Looking at the configure output in more detail, it did say:

checking for prefix by checking for gzip... /usr/bin/gzip


I know I can't blame autoconf itself for this one, because the tool has been misused, but using a tool and then violating both its common conventions (default prefix of /usr/local) and its own self-documenting behaviour, is really bad.

And one last thing. It installs itself as zcat, not gzcat. On Solaris, zcat isn't even the same beast.

Thursday, May 11, 2006

MyISAM vs InnoDB

I've been using MySQL for the database componenet of a number of projects over the years.

Usually, I've used the MyISAM storage engine. It's fast (that's the reason for using MySQL in the first place), and generally reliable.

MyISAM isn't robust against system failure though. System crashes, reboots, and power cuts tend not to be handled very well. (Yeah, I know, they shouldn't happen in the first place, but this is the real world.)

I don't need the transactional capabilities of InnoDB (from the functional point of view even MyISAM is overkill for most of what I do), but something more robust would help.

So I thought I would do a quick check of the impact of using InnoDB on the system. This isn't a benchmark, it's completely unscientific, and all that, but it told me what I needed to know.

Running InnoDB, one minute's worth of disk activity looks like:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 7.0 0.8 34.1 0.0 0.1 1.6 12.4 1 5 d0

whereas with MyISAM it looks like:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 2.3 0.8 3.0 0.0 0.0 3.5 14.5 1 1 d0


OK, so InnoDB generates 3 times as many writes, and 10 times as much data transfer, as MyISAM. And the mysqld process consumes correspondingly more cpu time (although it's still very small - much less than 1% of a processor), so the system load average is a bit lower with MyISAM too.

I don't think this rules out InnoDB, although it does indicate that there is a significant cost to changing, and to scale up by a factor 10 (which I'm going to need to do, and then some) would likely have problems if I was using InnoDB. If I go down that route, I need to do more optimisation of the system design and the database client interactions.

Wednesday, May 10, 2006

Sinking ship?

Is this the end for SGI?

It's been a long painful slide, but inexorably downwards, and I can't really see any way back (short of a government funded rescue package).

I remember sitting through a sales presentation from SGI some years ago. We were told of their grand plans to throw away their well-respected workstation technology and become a shifter of (not quite standard or compatible with anybody elses) PCs running Windows NT; to throw away IRIX and adopt Linux; and to throw away their own RISC chips and go down with Itanium. (Itanium wasn't even shipping at the time.)

We regarded this as doubly suicidal. Not only were the products entirely unattractive to us (as existing customers with investments in their hardware and software platforms, incompatibility is a big turn-off), but we could see the plan failing and were thus reluctant to invest in products from a company that - in a single presentation - had gone from a front-runner to certainly doomed.

Sunday, May 07, 2006

Outdated libraries

Every so often, the ugly interaction of Solaris and the undisciplined open software world completely breaks a project I'm working on.

Latest case in point - libxml2. Solaris ships a horrifically antiquated version (2.6.10, to be exact) with Solaris 10 (and nevada, currently). Snag is, PHP requires 2.6.11 or later and won't build.

There are a couple of bugs open: 6362386 and 6193876. Not to mention 6378879.

It's been this way for 16 months. Surely time enough for something to have happened?

I realise that, because certain core components of Solaris itself rely upon libxml2, it cannot be upgraded without due care. If that is the case, then those components should use a private, compatible, copy, and allow the publically available version to be kept reasonably current.

Unfortunately the download site appears to be down at the moment, so I'll have to grab the source and build my own up-to-date copy tomorrow before I can make progress.

It's not as if libxml2 is the only external component that is somewhat antiquated. In fact, quite a lot of it is getting sufficiently old as to be useless, and if Solaris is going to be used as a platform for other open-source applications then some serious updating is going to have to be done.

Great Open Source

Looked fairly innocuous to start with, but this blog on ZDnet points to a list of the top 50 open source projects. Check it out - there are some real nuggets, and starting points for further exploration.

Friday, May 05, 2006

Solaris to/from Linux?

So Matty gave some reasons why people might consider switching from Solaris to Linux.

This is all a matter of opinion, of course. Despite computers being binary systems, everything in IT is shades of grey (often very dirty shades of grey). So here's my take on Matty's points, in the same order:

1. Integrated LAMP stack. Actually, all my Solaris boxes could have a full SAMP stack installed, but I would never use it. I wouldn't (and didn't) use it on Linux either. In both cases I would much rather install the application stack separately. It's very much safer that way - not only can I be sure that all the components are at specified known levels, and are the same across platforms, but I can be sure that my OS vendor isn't going to screw me around. I always had to wipe all evidence of apache, mysql, and friends from a RedHat system in order to get operational stability; unfortunately it appears that Sun are heading down the same misguided path by bundling more services in Solaris.

2. At my previous place of employment, you could always tell when the next version of Fedora had been released - all the group of developers were surly and miserable for a week because there desktop had been randomly rearranged and they had to relearn it. I reckon each of them lost a week's productivity as a result. Sun aren't much better - open source software doesn't care for operational stability, and every time Sun update Gnome or JDS in Solaris nothing works. And I'm now convinced we're going backwards. (As an aside, I've had many more problems with desktop apps not working right on Fedora than Solaris, but that may have been because the Solaris versions didn't try to push the bleeding edge so hard.)

3. The JES stack deserves a good kicking. My experience of it has been woeful. So I agree with that part. The other side of the coin is regular updates. As I said in point 1, I don't want my OS provider to randomly modify the working of key applications. So I wouldn't use either of them.

4. My experience of ISVs is that they hate supporting Linux. And I can't blame them - the qualification effort is horrendous. I asked one the other week about Solaris x86 support, and their answer was "we keep getting asked about that, but it's not on our roadmap". Anothe one went "phew!" when we said we were going to deploy on Solaris - he was then confident it would work out of the box. But, yes, there does seem to be more visibility for Linux - I suspect vendors think that's what customers want, when most of the time any alignment being customer requirements and vendor offerings is random at best. At least with Solaris, if it works it works, rather than having to stick to one particular release of one particular Linux distribution with a specific set of patches.

5. Not so; managing applications and patches on Solaris is an absolute doddle. Provided you ignore completely Sun's perpetually failed attempts to improve the process - patchpro, smpatch, prodreg, update connection are all worthy only of the dustbin. Fortunately it's trivial to write your own tools, or use pkg-get or pca which are massively better than anything Sun have come up with so far.

6. Why not upgrade? Modulo bugs in occasional releases (and these are the sorts of bugs that mean that not every build qualifies as a regular Express release) regular upgrade or live upgrade work fine. There's no need to futz with bfu unless you really want to.

7. At least with Solaris you have the option of zones! Are zones a universal panacea? No. But they are enormously useful for a whole range of operations where you need to consolidate or isolate services. I can believe smpatch update could take days, but then I wouldn't do it like that - I wouldn't use smpatch, for starters, and I wouldn't create the zones and then apply the patches. For this sort of thing our mode of operation was to simply migrate services onto a zone on a different box, and then rebuild machines, if the patch overhead was too great. (Based on the philosophy of never patching or rebooting a live service.) The fact is that there's no getting away from the fact that by creating 25 zones, you've increased the update cost 25 times. It is certainly true, though, that there's room for improvement in the patch tools. They are significantly slower than can be accounted for by the actual work they're doing.

8. It's a tricky balancing act. And I thin Sun have done pretty well with OpenSolaris. There was massive concern that the open source process would destroy Solaris' core strengths, and it doesn't seem to have done so yet. My hope was that it would address some of Solaris' weaknesses, and I don't think it's actually done that yet, but the level of engagement is there, and the signs are promising. To be honest, I'm not sure I would want to use an OS that would allow outsiders to put code back to the kernel source tree - does Red Hat allow me to modify their kernel source?

One thing that worries me about end-user putbacks is that the whole thing has to be vetted and managed, and that has to be done by Sun (because their position is that if it goes into OpenSolaris then it goes into Solaris - there is only one codebase), so it costs Sun more to have a community member fix something than to have one of their own engineers do it. Given the financials of the past few years, this doesn't strike me as an optimal situation.

There are some good points here - it's not as if Solaris hasn't got any weaknesses. The real killer is the availability of commercial software. If you're only offered it under Linux then you're going to have to put up with increased costs in terms of licensing, support, hardware, and staffing (or just pretend they don't exist), or choose another product. But there are many places where one person's ideal is anathema to someone else, and we just have to accept that there is no one solution to every problem.

Thursday, May 04, 2006

Well, that didn't work...

I notice that Eric made a decent comment about my recent rant on the decline of the desktop.

Now, I've tried xfce a couple of times in the past, but it has similarities to CDE that have always put me off. But I've never tried fluxbox (although I did try blackbox back in the day).

So I thought it deserved a try. Downloaded and installed it (which was fine apart from the usual problems).

I started it up, and everything seemed unusually sluggish. That's odd, this is supposed to be light and fast. I then made the mistake of clicking on the background and selecting 'About'. This sent the whole thing into a tailspin. 4G of memory usage later and my machine is like stuck in treacle.

Worth a try, but I think I have to write that one off.

I'm leaning back towards windowmaker. (Anyone know which black hole the windowmaker website has disappeared down at the moment?)

Wednesday, May 03, 2006

tuxpaint

One of my girls came home from school yesterday raving about Tux Paint.

It looks pretty good stuff to me. (The fact that one of my girls is mad about penguins might also make it popular.)

They're using it on Windows at School, but it's cross-platform, so I built it for Solaris too.

I had to go through the prerequistes:
In all cases I did a

(setenv CC cc ; setenv CXX CC ; setenv F77 f77 ; ./configure )
gmake
gmake install

to build, using the free Sun Compiler, as libtool couldn't cope with gcc for some reason.

Then to build tuxpaint

gmake CC=gcc


This wasn't quite enough. I needed to add an explicit

-lpng -lsocket

to the link list, and then it fails because it needs strcasestr().

OK, so I grabbed a shim copy of strcasestr from OpenSolaris (for example, this one), compiled it, and linked that in.

And, because it used a simple makefile, fixing up the problems was trivial.

Oh, and I've filed a couple of bugs. One against tuxpaint so that it handles a missing strcasestr (or doesn't require it at all); and an RFE that Solaris include this function (as opposed to the several private copies dotted about in the source), as this isn't the first time I've seen applications want it.

Saturday, April 29, 2006

Wither the desktop?

It's a personal thing. My desktop is a virtual space in which I work, play, and socialize. As such, I want it to work my way.

This covers several aspects: the way it looks (colour scheme, imagery); the way it behaves (in response to my requests); what it does (which programs are available).

In unix-like environments, the desktop was originally a number of independent windows looked after by a window manager. This has evolved into the (more or less) tightly integrated desktop environments such as GNOME and KDE that we have today.

In years gone by...

Some of the early window managers were fairly basic. I remember wm and uwm, but I really liked the old Ardent window manager. It was friendly and very customizable.

Time went on and the original generation of window managers bit the dust. It seemed that twm was the new standard. I never used twm itself much, as I was starting to run into problems caused by having too many windows open. So I ended up using tvtwm
instead.

Customizing tvtwm was fairly easy (assuming you're happy to edit configuration files) and the degree of customization available was quite extensive. You had complete control over all the mouse button events, for example. I defined all my own menus. And you could define your colour scheme extremely precisely - I had the window decorations for each type of window in different colours, so I could more easily spot a particular application on screen.

Then along came desktop environments - a window manager with an associated set of libraries, and a (more or less) complete toolset. Open Look and CDE were the well known ones, although I'm sure there were others.

However, I found one thing in common with both Open Look and CDE. I hated them both. Utterly and completely. They both forced me into unnatural and counterintuitive ways of working, and I can't stand either of them for more than 5 minutes.

We're now into the brave new world of Gnome and KDE. Again, the aim is a complete and all encompassing desktop environment with a set of libraries and a complete set of applications.

Unlike Open Look and CDE, I've found I can tolerate both KDE and Gnome. Sure, they're slow, and both have irritating features, but both are good enough that I don't start swearing at the screen after a couple of minutes.

Have we really made progress in the last decade? I'm not sure we have. Some of the applications we have now certainly have more functionality than was available 10 years ago, but I don't see that they are necessarily better suited to today's problems than the applications of 10 years ago were to the problems of the time.

Put it another way. Ten years ago the applications and environments I had available met my needs of the time more than adequately. That's no longer true. And it's not as if my requirements have actually changed all that much - certainly less than the overall computing landscape has.

I'm not optimistic about the future. I was less than impressed with the Gnome-based JDS that will shortly go into OpenSolaris. I see very little of interest, and a lot of regressions.

What I also see is a lack of variety, a lack of excitement, no spark. Coupled with an increasing inability to do the basics, and the desktop is withering away.

When it comes down to it, I'm not actually asking for very much. I want to be able to set the focus policy, define the actions to be taken on mouse clicks, define what menus appear, and under what circumstances, and with what contents, and define the shape, colour, and imagery of decorations. And then use the applications that I want.

The desktop environments appear to have become less customizable, not more. Consider the available themes. There are thousands for WindowMaker (and some of them are quite decent). How many Gnome themes are there? Now, OK, a WindowMaker theme doesn't really do very much, but it does what you want.

It's not as if the desktop frameworks have provided a solid foundation on which to build better applications, either. Most of the standard applications that ship are pretty poor. I would much rather have a dedicated window manager and ally that with best of breed applications than have a bunch of applications that happen to be built using the same toolkit.

In summary, I feel that desktop development has headed off down a cul-de-sac, and we need to get back on the main road to make real progress.

Thursday, April 27, 2006

More distractions...

As if I didn't have enough distractions, I've been playing with Google SketchUp.

(And yes, this means on the Windows box...)

I've used various CAD and 3-D programs over the years (and the majority of consumer ones felt like they were just some third-rate CAD lookalike), and they were pretty hard work. SketchUp is brilliant - so easy to build a model.

Wednesday, April 26, 2006

Die, configure, die!!!

So I'm building a new version of Eterm and, as usual, the ./configure script is getting in the way.

First it couldn't find libast. That was legitimate - so I install libast and expect all to be well.

So now it constructs a compiler command line that gcc chokes on.

That's fine, I tell it to use cc instead and it can't find libast. The reason is simple - it's putting the -last ahead of the -L switch to tell it where the library is.

So I fix that and the compile just blows up. Looking more closely:

cc -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/i clude
-I/usr/local/i clude -I/usr/ope wi /i clude -O -c
actions.c -KPIC -DPIC -o .libs/actions.lo

Look closely: it's whacked all the n's out of the include directories.

So I randomly shuffle my PATH (presumably it's found a sed it doesn't like, but isn't that the sort of thing that autoconf is supposed to solve?) and eventually get it to work.

Won't compile. The code simply isn't liked by the Sun compiler.

Ho hum. Back to gcc - let's try a different version. OK, so I get it to configure and make, but it still fails, this time with an Undefined symbol error.

The upshot of all this is that autoconf has just wasted 10 minutes of my time and completely failed to produce a successful result at the end of it all.

Sadly, this is becoming the norm.

Tuesday, April 25, 2006

Retro gaming

There aren't enough hours in the day.

I'm trying to wrap up a bunch of contributions to OpenSolaris, and have managed to get some of them submitted, but I find myself getting distracted.

I just got Pro Pinball: Timeshock for PSOne off eBay (and that has to be the best Pinball simulation ever) so I'm spending a little time on that.

I just got the PS2 Atari Anthology and it's a blast. I remember most of the games. The great thing about having it on the PS2 is that it doesn't cost a fortune to play! The downside is that the controls don't always translate to a console gamepad all that well so some of the games can be quite tricky to play.

And then I've been playing some of the old ZX Spectrum games on World of Spectrum, using the Java emulator.

Gotta go, there's a hi-score to beat...

Sunday, April 16, 2006

Graphical DTrace

So the Chime Visualization Tool for DTrace is now available.

I'm itching to try this out, and to see how you can build bridges with kstat (by means of jkstat or something different).

Unfortunately there appears to be a network problem with my test machine. Or more specifically there's a problem somewhere in the path between where I'm sat and where it's at. I've managed to determine that the machine itself is fine, but doing any work on it is impossible. It being a holiday I don't expect it to be fixed for a day or two.

Wednesday, April 12, 2006

Extra Software

No operating system - no matter how good - comes with a complete set of every piece of software you're likely to want. There are always cases where an end-user needs to add additional software to meet specific needs. Corporate servers need business software; developers may want to live on the bleeding edge.

I used to maintain a lot of stuff myself. But this has become harder and harder over time, as build systems become more complex and less intelligent, and dependencies becore more entwined and harder to resolve.

One place I now use to keep stuff up to date - and to try software out easily without having to invest the effort in building the whole dependency tree from scratch myself - is Blastwave.

It's very use. Install pkg-get and go install. It grabs the software you want and installs it and any dependencies you need. All simple and painless.

Of course, what it also shows you is how bad this dependency tracking gets. I wanted to try out a couple of pieces of software this afternoon - dia and enlightenment as it happens - and off it went, installing package after package. It's a lot easier than doing it myself, but complexity is on the rise and I'm not sure how close we are to total meltdown.

Install investigations

There's been a reaonable amount of discussion recently regarding Solaris installation speed. Indeed, there's even some idea of what the problem is.

However, closer investigation reveals that rewriting the contents file isn't the limiting factor in a Solaris install. It's not even in the top 3, which are:
  • The time taken to uncompress the data (they're bzip2 compressed)
  • The time taken write Solaris to the disk
  • The SMF manifest import on first boot

Currently, the contents file rewrite is having a race with the pkg tools overhead for 4th place.

Why this disconnect between the obvious problem - and it is a problem - of rewriting this large file many times and its relatively minor importance to Solaris install times? After all, for a slightly trimmed full install Solaris itself accounts for 3.5G of writes, and rewriting the contents file is twice that.

The point is, though, that rewriting the contents file involves a small number of large writes, which are quick. Solaris itself is many small files, so it generates something like 100 times the number of I/O operations.

Not only that, but it's possible to tweak the package installation order to minimize the amount of rewritten data. Simply install all the small packages early and leave the big packages that bloat the contents file until last. Doing so could - in principle - reduce the contents file rewrites by an order of magnitude.

This affects zone installs as well. For zone install, the uncompress cost doesn't exist at all. And for a sparse root zone, there's no Solaris to write to disk - it's loopback mounted from the global zone. So the contents file is much more important as a limiting factor for zone creation performance. However, I've managed to halve the contents file rewrites by tweaking the package installation order. I've not got that much control over the installation order, as it seems to depend on both dependency calculations and the order that opendir() goes through /var/sadm/pkg, but even then a gain of a factor 2 was fairly easy.

This isn't to say that the management of the contents file isn't an interesting and important subject that can lead to some benefits, but the relative importance of it in install performance can easily be substantially overstated. There's other low-hanging fruit to have a go at!

Monday, April 10, 2006

./configure is evil

For years I've used emacs as my editor.I don't want to get into religious wars here - if you want to use vi, then that's fine too. I happen to like emacs because I started out with EDT and TPU under VMS, and when I moved off the VAX I had to find an alternative - and I was able to make emacs play ball pretty easily.

I don't use an IDE, or indeed any custom authoring tools for that matter. I write web pages in emacs, as plain HTML.

Si I was glad to find out that I'm not entirely alone. Bill Rushmore just wrote about Emacs as a Java IDE. Spurred on by this, I thought about upgrading the version of emacs that I use.

I've been stuck on GNU emacs, version 19, for a very long time. The reason I haven't upgraded to later version is that they're too darn slow. (The same goes for XEmacs.) Which is one reason for not using an IDE - an editor has to start faster than I can start typing.

I now have a dual Opteron desktop, so it should be possible to get the latest version to start up fast enough, right?

I don't know, the darn thing won't build.

So I download emacs, unpack it on my Solaris box, see what configure options I have, and type:

./configure --prefix=/usr/local/versions/emacs-21.4 --without-gcc
--with-xpm --with-jpeg --with-tiff --with-gif --with-png

Why without-gcc? I need this to go real fast, so I want to use the latest studio compiler and crank it up.

It fails. No xpm, png, in fact no nothing.

configure:2694: /usr/ccs/lib/cpp conftest.c >/dev/null 2>conftest.out
"/usr/include/sys/isa_defs.h", line 500: undefined control

Well, there's a fundamental problem. The configure script is trying to be too clever by half, and is calling cpp directly. Don't do that. Run it the same way the compiler does, and things will work much better.

So I tell it to use "cc -E". This gets it past some of the cpp stuff, but there are two new problems that crop up:

configure:5430: cc -o conftest -I/usr/openwin/include - g - O
-I/usr/openwin/include -L/usr/openwin/lib conftest.c -lpng -lz -lm
- lX11 - lkvm - lelf - lsocket - lnsl - lkstat 1>&5
ld: fatal: file g: open failed: No such file or directory
ld: fatal: file O: open failed: No such file or directory
ld: fatal: file lX11: open failed: No such file or directory
ld: fatal: file lkvm: open failed: No such file or directory
ld: fatal: file lelf: open failed: No such file or directory
ld: fatal: file lsocket: open failed: No such file or directory
ld: fatal: file lnsl: open failed: No such file or directory
ld: fatal: file lkstat: open failed: No such file or directory

There's a major thing wrong here. Why has it randomly decided to put spaces in? (And why does it think it needs libelf, libkvm, and libkstat just to see if it can find a png function? I'll let it off some of the other libraries, as although it doesn't need to specify them all there were times in the past when you had to.)

That's not all:

"junk.c", line 586: invalid input token: 8.elc
...

So it looks like it's trying to use the C preprocessor in different ways.

Foiled again.

Really, I can't see why anyone would think that getting a configure script to make a bunch of random guesses (and often get them wrong) is anything other than a stupid idea. There was a time when unix variants differed dramatically and you needed a way to tell them apart, but that time has long gone. As it is, we've now got to the point where the amount of software that compiles and works is heading towards zero, and the worst thing is that - unlike in the old days when it would have taken a couple of seconds to fix the Makefile to make it work - actually correcting the error when it goofs up like this is almost impossible.

Wednesday, March 22, 2006

poking around with dtrace

Having released jfsstat, I've been using it on my test system just to see what would come up.

I left it running looking at file accesses - lookup, access, pathconf, readlink - and started to notice some odd patterns.

The first was that /proc was getting a very steady 25 lookups and 12.5 accesses per second. OK, that's easy - I had top running (refreshing every 5 seconds) and this system has 60 processes on it, so every refresh top is doing 1 access and 2 lookup per process.

The next was that / (and this machine just has the single filesystem) was seeing about 600 lookup and 10 access requests every 10 seconds.

The third part of the pattern was that roughly every minute there was a veritable storm of operations on the / filesystem.

Now, one of the longer term aims of my writing tools like jfsstat and jkstat is that you see some strange activity, and you can click on the activity there and then and drill down deeper - probably with dtrace. I'm not there yet, but I thought I would practice my dtrace skills and do it by hand.

OK, so what I'm after is ufs lookups. What can dtrace give me?

% dtrace -l | grep ufs| grep lookup
536 vtrace ufs ufs_lookup TR_UFS_LOOKUP_END
537 vtrace ufs ufs_lookup TR_UFS_LOOKUP_START
17836 fbt ufs ufs_lookup entry
17837 fbt ufs ufs_lookup return

Right. Let's have a look at what's generating those lookup requests:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

java 612

So, that 10 second repeating burst of 600 lookups is java. In fact, I know this to be tomcat.

Now I run the script again to catch the massive burst of activity that happens every minute:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

sched 24
pwd 28
sh 75
fping 91
server.sh 112
uname 150
mysqld 162
ping.sh 238
nscd 491
cron 498
init 988
load.sh 1014
rup 1539
java 1836
perl 2120
awk 2394
mysql 63666

I've hit the tomcat activity burst 3 times, but the once a minute is coming from something launched by cron - a system monitoring script that runs fping and rup and pokes the results back into a mysql database. But what on earth is mysql doing making 63666 lookup requests?

(The first question I asked was - is this one instance of mysql or many? If I aggregate on pid as well as execname then I see that I'm running a lot of copies of mysql, each of which generates 786 lookup requests on the filesystem.)

Next question: what are the pathnames that are being used in the lookup request? To get this, I need to understand the ufs_lookup call itself in a little more detail. So the source tells us that the 4th argument is a struct pathname, and the string I'm after is the pn_buf member. So let's see what pathnames are being looked up. First the little tomcat burst:

dtrace -n 'fbt:ufs:ufs_lookup:entry{@[stringof(args[3]->pn_buf)]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

/opt/XSload/tomcat/conf/Catalina/localhost 6
/opt/XSload/tomcat/webapps/ROOT/META-INF/context.xml 6
/opt/XSload/tomcat/webapps/jsp-examples/META-INF/context.xml 6
...
/opt/XSload/tomcat/webapps/xsload/WEB-INF 18
/opt/XSload/tomcat/webapps/xsload.war 20
/opt/XSload/tomcat/conf/Catalina/localhost/host-manager.xml 35
/opt/XSload/tomcat/conf/Catalina/localhost/manager.xml 35
/opt/XSload/tomcat/webapps/balancer/META-INF/context.xml 35
/opt/XSload/tomcat/conf/context.xml 90

Pretty clear, really - every 10 seconds tomcat goes round checking to see if you've gone and modified anything.

Now for the mysql burst. What are the pathnames here? There's quite a lot of output, so I've trimmed it a bit:

...
/usr/ccs/lib/libc.so.1 432
/usr/ccs/lib/libcrypt_i.so.1 432
/usr/ccs/lib/libcurses.so.1 432
/usr/ccs/lib/libgen.so.1 432
/usr/ccs/lib/libm.so.1 432
/usr/ccs/lib/libnsl.so.1 432
/usr/ccs/lib/librt.so.1 432
/usr/ccs/lib/libsocket.so.1 432
/usr/ccs/lib/libthread.so.1 432
/usr/ccs/lib/libw.so.1 432
/usr/ccs/lib/libz.so.1 432
/lib/ld.so.1 504
/opt/SUNWspro/lib/rw7/libCrun.so.1 540
/opt/SUNWspro/lib/rw7/libCstd.so.1 540
/opt/SUNWspro/lib/rw7/libc.so.1 540
/opt/SUNWspro/lib/rw7/libcrypt_i.so.1 540
/opt/SUNWspro/lib/rw7/libcurses.so.1 540
/opt/SUNWspro/lib/rw7/libgen.so.1 540
/opt/SUNWspro/lib/rw7/libm.so.1 540
/opt/SUNWspro/lib/rw7/libnsl.so.1 540
/opt/SUNWspro/lib/rw7/librt.so.1 540
/opt/SUNWspro/lib/rw7/libsocket.so.1 540
/opt/SUNWspro/lib/rw7/libthread.so.1 540
/opt/SUNWspro/lib/rw7/libw.so.1 540
/opt/SUNWspro/lib/rw7/libz.so.1 540
/opt/SUNWspro/lib/v8/libCrun.so.1 540
/opt/SUNWspro/lib/v8/libCstd.so.1 540
/usr/local/mysql/data/my.cnf 540
...
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis2/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis2/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9/libCstd_isa.so.1 756
...
/opt/SUNWspro/lib/libthread.so.1 864
/opt/SUNWspro/lib/libw.so.1 864
/opt/SUNWspro/lib/libz.so.1 864
/lib/libm.so.2 888
/usr/lib/libc.so.1 972
/usr/lib/libcrypt_i.so.1 972
...
/opt/SUNWspro/lib/v8/libgen.so.1 1080
/opt/SUNWspro/lib/v8/libm.so.1 1080
/opt/SUNWspro/lib/v8/libnsl.so.1 1080
/opt/SUNWspro/lib/v8/librt.so.1 1080
/opt/SUNWspro/lib/v8/libsocket.so.1 1080
/opt/SUNWspro/lib/v8/libthread.so.1 1080
...
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/../../libCrun.so.1 2160
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/libCstd_isa.so.1 2592
/platform/SUNW,Sun-Blade-1000/lib/../../sun4u-us3/lib/libc_psr.so.1 3360
/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1 4480

What on earth?

So, most of these lookup operations are the mysql binary looking for shared libraries when it starts. And in some less than obvious places too! So why is this? I go and have a look at the binary:

% dump -Lv mysql
[INDEX] Tag Value
[1] NEEDED libcurses.so.1
[2] NEEDED libz.so.1
[3] NEEDED librt.so.1
[4] NEEDED libcrypt_i.so.1
[5] NEEDED libgen.so.1
[6] NEEDED libsocket.so.1
[7] NEEDED libnsl.so.1
[8] NEEDED libm.so.1
[9] NEEDED libCstd.so.1
[10] NEEDED libCrun.so.1
[11] NEEDED libw.so.1
[12] NEEDED libthread.so.1
[13] NEEDED libc.so.1
[16] RUNPATH /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib
[17] RPATH /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib

The list of libraries matches what we see being looked for, so that makes sense. The problem is that the compiled in library search path contains places that it shouldn't (and some repeated), so it needlessly searches those locations (and multiple times at that) when mysql starts up.

So, problem solved. And, as I said earlier, the idea is that you be able to see this anomalous activity in one of the jkstat tools and click on it to drill down into the system to see what's going on, so that all the dtrace is done automatically on the fly for you.

It won't be that easy, of course. I'm relying on the fbt provider, so that I need to pretty well write the dtrace script by hand for each function I wish to investigate. (There isn't even a consistent naming or calling scheme - you can't just replace ufs by procfs and expect it to work.) But fortunately we have the OpenSolaris source to look at to see what's actually going on underneath the covers.

Tuesday, March 21, 2006

jfsstat

I've just updated jkstat.

The notable new feature in this version is the jfsstat utility:



This uses the new kstats introduced in Nevada recently - the kstats used by fsstat. So you need to be running Nevada build 35 or newer to see these. (ON downloads - you want SXCR build 35 or later.)

The other change that's part of this is that I've started to use JTable to display things. I don't know why I've avoided this in the past - I guess that JTable seemed rather complex, but once I had got the hang of it it turns out to be very easy. I'm also using TableSorter from the Java Swing tutorial, which gives me the ability to sort the columns for free.

Enjoy!

Friday, March 17, 2006

Home Networked

I'm writing this blog entry using my W2100z workstation upstairs in the spare room.

We've spent the last couple of weekends decorating Amanda's room, and I took the opportunity to run some cat-5 cable in the upstairs rooms. The next step was to run cable down the stairs to connect my little ethernet switch to the broadband router, and enable dhcp. Hey presto! It all works.

The girls each have a SunBlade 150. Well, I've got them, and it means I don't have to worry about them getting virus infestations. So a surfing we can go! (And it means that the main computer is free instead of being taken up by someone trying to do their homework half the evening.)

I used an all-in-one kit I got from Maplins. 50m of cable, connectors, boxes, faceplates, tools, and 4 patch leads. A great buy and a real bargain.

Wednesday, March 08, 2006

T2000 problem

I just tried to buy a pair of Sun's T2000 machines, and then discovered (very late) that the system specification doesn't match my requirements.

It's not the system itself that's the problem, but the peripheral connectivity - or rather lack of such.

In particular, the plan was to hook up a couple of 3120 SCSI arrays. Not everything wants a fancy raid array, and FC arrays (and associated HBAs) are pretty pricy anyway. In this case, I'm looking at raw spindles for database access, but also if you look at ZFS it wants plain drives - raid hardware just gets in the way. So the plan was to have 2 SCSI channels and mirror them.

This won't work in a T2000. There's only 1 free PCI-X slot (the internal SAS controller takes the other one), and the only supported SCSI HBA is single channel anyway. So at the present time you simply can't have more than a single SCSI chain on a T2000.

To say that this is annoying is an understatement. It also limits the usefulness of the T2000 for several other projects I have in the pipeline.

Now, I could use FC storage, because the T2000 does have PCI-E slots, and there are PCI-E fibre HBAs, so it would work, but you're looking at a 50% or so increase in cost, which I regard as unacceptable. (The cost differential is particularly bad in small configurations - as you push up it becomes much less pronounced.)

Ho hum. Time to construct a plan B.

Friday, March 03, 2006

OpenSolaris Installation and Packaging

It's here! Dave Miner made the announcement that the Installation and Packaging community, and the SVR4 packaging project are now up and running.

This is important stuff. Being able to install and manage software is a critical component of an operating system, and is something I do a lot of.

And the process - particularly the performance (or lack thereof) of the tools - could stand some improvement. Yes, they work, and they're solid, but with a bit of extra work they could be made better.

Wednesday, March 01, 2006

fsstat in latest OpenSolaris

I've updated my test machine to the latest OpenSolaris build.

I used the bfu archives this time. Normally I wait for it to make it into Solaris Express (SXCR) and jumpstart, but this release has two feature I wanted to investigate, and I want to try them out as soon as I can.

The two features in question are the Java binding to DTrace, and the new fsstat utility and associated kstat support. More on DTrace once I get jkstat updated to integrate with it, but I wanted to mention a little more about fsstat.

Now, I'm just a user, but I've played with fsstat a little and looked at the code, so I've got a little bit of understanding of what it's doing.

On its own, fsstat will give filesystem activity for the various filesystem types:

new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
1.22K 606 63 1.99M 569 10.2M 37.8K 993K 502M 216K 152M ufs
0 0 0 7.73K 0 14.8K 722 8.65K 3.18M 290 6.91K proc
0 0 0 1 0 10 0 0 0 0 0 nfs
0 0 0 0 0 0 0 0 0 0 0 zfs
0 0 0 87.1K 0 0 0 0 0 0 0 lofs
14.6K 2.95K 11.5K 30.8K 37 43.5K 26 32.9K 32.9M 45.9K 34.5M tmpfs
0 0 0 3.05K 0 0 0 32 5.25K 0 0 mntfs
0 0 0 0 0 0 0 0 0 0 0 nfs3
0 0 0 0 0 0 0 0 0 0 0 nfs4
0 0 0 6 0 0 0 0 0 0 0 autofs


If you give it a filesystem as an argument, it will show that:

# fsstat /tmp /var/run /etc/svc/volatile
new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
251 233 2 1.10K 21 719 26 29 34.9K 60 36.7K /tmp
20 0 0 44 8 16.8K 0 0 0 6 67 /var/run
14.5K 2.72K 11.6K 29.8K 8 26.4K 0 32.9K 32.9M 46.0K 34.5M /etc/svc/volatile

and it takes the normal interval and count arguments.

These numbers are all backed by kstats. For example:

# kstat unix:0:vopstats_ufs
module: unix instance: 0
name: vopstats_ufs class: misc
crtime 76.4978722
naccess 559104
naddmap 1280864
nclose 514400
ncmp 5179654
ncreate 1222
ndelmap 1276280
ndispose 1788458
ndump 0
ndumpctl 0
nfid 0
nfrlock 6953
nfsync 1641
ngetattr 1550291
ngetpage 5904478
ngetsecattr 2882
ninactive 30404
nioctl 114402
nlink 18
nlookup 10861279
nmap 1024824
nmkdir 17
nopen 479509
npageio 14
npathconf 70555
npoll 477742
nputpage 39301
nread 1031796
nreaddir 38829
nreadlink 785043
nrealvp 65352
nremove 602
nrename 29
nrmdir 4
nrwlock 1292387
nrwunlock 1293855
nseek 213405
nsetattr 531
nsetfl 179084
nsetsecattr 26
nshrlock 0
nspace 12
nsymlink 16
nvnevent 0
nwrite 223174
read_bytes 530866166
readdir_bytes 18340192
snaptime 9588.8833902
write_bytes 159609739

Now, that's a lot of data about what's going on.

The kstat names aren't spectacularly obvious. They all start with unix:0:vopstats_ with the unique identifier tagged onto the end. This can either be a filesystem type (ufs, lofs, nfs, zfs are some examples) or the id of a filesystem mountpoint. So in /etc/mnttab you might see

/dev/dsk/c0t1d0s0 / ufs rw,intr,...,dev=1d80008

(I've trimmed this for brevity). So the kstat you need to query for statistics on the root filesystem in this case is called unix:0:vopstats_1d80008.

And that's pretty much it. Now to get some more tools to make sense of all this new information!

Sunday, February 26, 2006

Vanished...

After a nice weekend away I'm wondering where my machine at work has got to.

It was fine when I left it on Thursday, but sometime Friday afternoon it vanished without trace. Can't log in. Can't ping it. No nothing.

I can get into my other machine (that I use for OpenSolaris testing) just fine. That's connected into the same switch, powered off the same extension lead. So it's not a network or power problem.

Oh well, I guess I'll find out what hapened to it when I get in to the office tomorrow.

Thursday, February 23, 2006

Affordable Sun Gear

Over on OpenSolaris.org, I jumped into a discussion regarding the availability - or more accurately lack of - a cheap sparc desktop machine.

At the present time, Sun sell a number of desktop sparc machines: SunBlade 150, SunBlade 1500, SunBlade 2500, and the new Ultra 45.

Frankly, why they're still selling the 150 is beyond me - it wasn't exactly quick when it was introduced in 2002 and it's a painful experience trying to use one with modern bloated software. It's horrifically expensive, and the available configurations aren't up to much. £2000 for a crippled antique? No thankyou...

The SunBlade 1500 isn't really that bad a machine. It's still over £2000, but it's not that much more than the 150 and is 2-3 times better. Still, £2000 just to get off the ground? Heavens...

I'll skip the 2500, as I think the Ultra 45 is similarly specified, but with a little more expansion and future-proofing. The starting price isn't that much more than a 1500 either, but rapdily rises.

All these boxes are out of my price range as an individual, and I couldn't really justify an employer buying them either.

The more powerful boxes are better value, but you have to pay for the privilege.

As far as I'm concerned, a reasonable entry-level box would be something like the SB1500, at about £1000 or so. Doesn't have to be fancy, but XVR-100 graphics and 512M memory minimum.

There's a similar story when it comes to sparc servers. Sun still sell the V100 and V120. OK, so there's a good market, and these machines have their uses. And they don't have much competition - there aren't many other machines of that low a spec out there.

At least with servers you definitely get into value-for-money territory as you move up the range. Certainly any of the T1 (Niagara) boxes are awesome. But there isn't much value in the sub £5000 space. Again, something like a V100 but with a modern US-IIIi processor at just over the £1000 mark would be handy.

The whole sparc low-end lineup looks incredibly stale. Even the Ultra 45 isn't much more than a SB2500 rehash (a good rehash, but still a rehash).

As might be expected, there's more in the Opteron world down at the low end. You can't really quarrel with the X2100 and Ultra 20.

But even here, Sun make it hard to get something decent. The problem (and this afflicts the whole of the range) is the lack of configuration choice. They seem to make the basic assumption that there's a cheap and nasty option that has the minimum of everything, then have another option in which most things are upgraded, and maybe another option in which everything is maxed out. That's not what I want. I don't want to have to pay for a fancy graphics card just to get a faster cpu, or the other way round. And sometimes I might actually want the base system with a 250G drive. Most PC vendors I look at allow you to select the various components of the system independently, so that I can put together a system that's balanced to meet my needs, but Sun won't let me do that.

Java DTrace

Stephen Lau announced the latest nightly OpenSolaris delivery. Looking through the changelog, I noticed:

Issues Resolved:
PSARC case 2006/054 : DTrace JNI Binding
BUG/RFE:6384263PSARC 2006/054 DTrace JNI Binding

Yay! Getting to DTrace from Java!

Monday, February 13, 2006

Evil JES Installer

I'm a glutton for punishment. Must be. I can't think of any other reason why I put myself through this.

I'm testing out the Java Enterprise System. Version 2005Q4 comes in the DVD kit with Solaris 10 Update 1, so I thought I would try that out, following these sample instructions.

The installer starts off OK, but then it complains that J2SE is obsolete. Say what? This is a brand new S10U1 install, and has a newer JDK than is supplied on the JES media. I selected manual upgrade (promising that I would upgrade it myself) in the hope that it wouldn't do anything stupid.

For what it's worth, Solaris 10 Update 1 ships with J2SE 5.0_06, while the JES media contains the older (and insecure) 5.0_04. There are two major issues here already:
  • JES ought to have its java version in sync with the version of the OS it's shipped with
  • It ought to detect a newer version and accept it as good


So I plug through the screens. (There's another one where it complains about the versions of JATO, JAXP, and JAF being out of date. Why? This is the latest all-singing all-dancing version of Solaris, hot off the press. Why aren't those components up to date?)

So I get to the end and tell it to go install. And what does it do? It downgrades the system Java to the old insecure version!

Aaaaarrrrrrgggggghhhhhh!!!!!!

This is plain bad behaviour, compounding its previous errors with a heinous crime.

Friday, February 10, 2006

JKstat updated

I've updated JKstat - my Java JNI interface to Solaris kstats.

This version adds jiostat, a graphical version of iostat. This is just a basic hack at the problem - I want to be able to select and hide statistics, sort by output, and connect associated statistics (such as a disk with its partitions, or a metadevice with its components). However, the first implementation had pretty awful performance, which is why version 0.09 got skipped.

Thursday, February 09, 2006

Why do arrays have even numbers of disks?

Like it says: Why do arrays have even numbers of disks?

Most hardware disk arrays - certainly the ones that Sun sell - currently have an even number of drives in them. The StorEdge 3x00 series have 12, while the 6130 has 14.

The problem I have is that if you take away one drive to act as a hot spare, you're then left with 11 or 13. Not only is this an odd number, it's also prime.

So, what sort of sensible grouping of the drives can you come up with? I often punt and simply create a huge raid-5 volume spanning all the drives I've got left, which is simple. But there are cases when I really want to configure 2 identical sets of disks - either to mirror or to give to 2 hosts. To make this work, I have one drive left over (so I use it as a second hot spare).

Wouldn't it be neat to add an extra drive?

Roundup

Rounding up a few loose ends:

The OpenSolaris Visual Panels project has now started. This really interesting stuff, especially from my viewpoint of developing SolView and JKstat. I agree with JC Van Nieuwenhoven's comment - that managing Solaris with a GUI is a pain - and it's good to see efforts underway to fix this.

(One thing I would say, though, is that good GUIs aren't just for novices. While I might know the 16 arcane commands to configure something, would I rather have a good gui and press 1 button? If it was a good gui, yes!)

Following on from my application profiling to see if a T2000 would be a good thing, I found this article that explains a little bit more about what sorts of behaviour might throw off the statistics. Based on this, I think my machine is spending a lot of time in memcpy.

I also notice that the Ultra 45 workstation lists a 146G SAS drive as an option. I can't see this on the published price lists for the X4x00 or the T2000, but I hope it's on its way as it would help address one of the problem areas I've been having for a while.

Tuesday, February 07, 2006

Should I get a T2000?

So I'm looking at the new Sun T2000 boxes, and I tried the test program to see if my workload is suitable.

Now, this is a web server. That's all it does. And it's running coldfusion (ie JRun, as in Java), and Oracle, so first thoughts are that it should match pretty well. So I give pfp a whirl.

# /var/tmp/pfp -p 10
We observed 407247665 instructions separated
in 11.17% floating point and 88.83% others.

This workload is not recommended for UltraSPARC T1 systems.

That's not good!

OK, so that was an isolated incident. But this machine tends to stick at about the 1.5% grey area. This is typical:

# /var/tmp/pfp -p 10
We observed 1960035483 instructions separated
in 1.61% floating point and 98.39% others.

This workload is a potential fit for UltraSPARC T1 systems
and need to be tested.

Now, what I don't know is whether there's something odd about this machine, or Oracle, or Coldfusion, or the CMS sitting atop it, or the versions (oldish), or something about the fact that this is an old V880 running an old version of Solaris that pfp can't handle properly. But in any case, the T2000 doesn't look like a given.

I also tried looking at one of the machines I built myself recently, with an Apache/Tomcat/Postgres combo:

# /var/tmp/pfp 60
We observed 2132762256 instructions separated
in 0.05% floating point and 99.95% others.

This workload is recommended for UltraSPARC T1 systems.

That's what I expected. (And I get the same sort of thing on one of my Domino boxes.)

So I'm still unclear as to whether a T2000 would be a good bet for the old webserver.

Sunday, February 05, 2006

Solaris Info Viewer

Following a discussion on the sysadmin-discuss list, I've put up SolView, a utility I put together a year or so back and then forgot about.

The idea is to have a single window that gets you to the important information about a Solaris system quickly and easily. It hasn't been extensively tested, and relies on Solaris 10 at the present time. Comments etc, especially suggestions for new capabilities or information it could display, are welcome.

Friday, February 03, 2006

Suspend/Resume

After my recent confession, and looking at start up time, I tried suspend/resume to see if that could get the system going any quicker.

Now, I had earlier problems with suspend/resume taking forever, so I tried again after updating to S10U1.

And, OK, so the resume is quicker than it was. But it's still about 3 minutes - just the same as a cold boot. There has to be something wrong here - resume should be quicker, as it just has to bring things back into memory and set them running again without having to go through the thinking step of how they got there.

Thursday, February 02, 2006

System start up time

To follow on from why I use windows, I just did some (unscientific) start-up timings on my SunBlade 1500 running Solaris 10 Update 1.

Time in seconds from pressing the on button:

  1. Solaris license terms: 41s
  2. Console Login prompt: 104s
  3. Desktop Login prompt: 133s
  4. JDS start: 165s
  5. JDS ready: 190s
  6. Terminal ready to type: 199s


There's a 30 second hole in the middle where I'm typing in my username and password (and writing down the numbers) where 10 seconds would be more reasonable, so the full length of time to actual login is almost exactly 3 minutes. (And then another 20 seconds to open Mozilla.)

(I've broken the graphical startup into 2 phases, dtlogin start and JDS start, with the human typing in the middle. The whole graphical login process takes about a minute of the 3, with the general boot taking the other 2 minutes. OK, so it's possible to make the gui startup much quicker, but there's still the whole hardware phase and kernel boot to get past.)

What's the machine been doing all this time? A quick look at iostat immediately afterwards:

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
74.5 3.5 824.7 37.0 0.3 0.8 4.3 9.7 13 46 c0t0d0

Or in raw terms from kstat:

reads 10986
writes 482
nread 124590592
nwritten 5022208

So during boot I read 125M of data off the disk, in 11,000 reads. And this actually only covers about 2/3 of the boot - it isn't until about a minute into the boot that the kstats are created. If you allow for the login delay while I'm typing username and password and the few seconds it takes to actually run the above commands, and you can see that the disk is actually well over 50% busy during the boot. Based on the disk activity, boot times can't improve by better than a factor of 2 unless the disk access pattern changes (larger reads than the average 11k seen here would help).

The cpu statistics can also be obtained:

cpu_nsec_idle 102634677402
cpu_nsec_kernel 44861276123
cpu_nsec_user 25778566225

Remember that these numbers don't cover the first 1/3 of the boot. (And they don't add up, either, as I make that 170s which is more than the 140s between crtime and snaptime.) But looking at it, the processor is less than 50% busy.

I'm not sure how to look at these numbers and convert them into the sort of boot-time improvements that might be made, but taking the 50% resource utilization at face value indicates that the portion of the boot covered by the kstat collection could be sped up by a factor 2, which takes the overall cold start from 3 minutes to 2. That wouldn't be bad, would it?

Monday, January 30, 2006

Why I use Windows

I have a confession to make.

My main home computer runs Windows XP. Now, I'm a Solaris guy, so you would expect me to have banished Windows, but no, most of the times I use a computer at home it's the Windows box that gets the nod.

There are reasons for this renegade behaviour. The two main ones are Windows ability to do suspend-to-RAM and fast user switching. Essentially, if I just want to check my email quick, then either the Windows machine is on already and I just have to switch user, or I hit the power button. In both cases it's about the time it takes me to get the seat located comfortably before I'm online and working.

Solaris can't match this. If I'm working at home then the Sun box comes on, as the superior behaviour and environment are worth waiting 10 minutes for. The boot time is awful; the JDS startup time is awful; suspend resume is only on sparc and is very much hit and miss. If we're after wider adoption, then this is a huge area we have to address.

newboot 2 - nvidia failure

OK, so having had great success upgrading my home W2100z to Solaris 10 Update 1, I tried it on my work machine.

Now, I wasn't expecting this to be entirely trouble free. My work machine gets all sorts of abuse, with various test versions of anything that might be lying about - so before it started I cleaned up an old zfs beta release, deleted all the different sorts of backup software I had been trying, and tried to clean up and back out all the hacks and kludges.

In the end, none of the problems I was anticipating surfaced. But when it came back up - no graphics. Looking at the /dev/nvidia* and /dev/fb* entries, they looked very suspect.

Turns out this is a known problem with the NVIDIA drivers and the S10U1 upgrade. And it was essentially what I had surmised by looking at the device entries, so a quick deinstall, reinstall, clear up /etc/path_to_inst and a couple of reconfiguration reboots later and everything's back in business again.

Phew!

Saturday, January 28, 2006

newboot

Got my Solaris 10 1/06 DVD kit yesterday. So popped the x86 DVD into my W2100z (running Solaris 10 FCS) and let it get on with it.

Upgraded without a hitch. Everything looks good!

The boxed set includes quite a lot of stuff, so I also updated the freeware, and added the Studio 11 compilers. I'm leaving the Java Enterprise System for another time - I'm not going to risk playing with it on my main personal machine!

One interesting thing - I've got a decent monitor (a Sun 21 inch CRT) on this machine, and it had been running at 1280x1024. I hadn't really had time to investigate tweaking Xorg to get it to run at higher resolution. After upgrading, it came up on its own at 2048x1536. This was a little over the top, so I dropped it back to 1600x1200, which is the max recommended anyway, and runs at 85Hz rather than 75Hz, and makes the text a little more readable.

Everything still works, all I need now is a little more time to get back into development work.

Thursday, January 26, 2006

Buried in good stuff

The good stuff keeps on coming.

OpenSolaris now has even more communities - particularly for sysadmins and appliances - to follow. As if approachability, networking, observability, zfs, and zones weren't enough. And that's only half the communities I'm interested in!

What I really need is a time machine so I'm able to keep track of it all.

Tuesday, January 17, 2006

Enemy action?

According to Auric Goldfinger:

Once is happenstance. Twice is coincidence. Three times is enemy action.

One: panic[cpu1]/thread=fffffe80fd47aea0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8001773c70 addr=0 occurred in module "genunix" due to a NULL pointer dereference

Two: panic[cpu0]/thread=fffffe80f83a5de0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe80010e8bd0 addr=0 occurred in module "unix" due to a NULL pointer dereference

Three: panic[cpu1]/thread=fffffe8000f56c80:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8000f56450 addr=0 occurred in module "conskbd" due to a NULL pointer dereference

That's 3 failures on my desktop machine in just over a month. For those interested, the stack traces are:


stack pointer for thread fffffe80fd47aea0: fffffe8001773cb0
fffffe8001773d70 0xffffffff8a965940()
fffffe8001773da0 port_remove_done_event+0x4b()
fffffe8001773e10 port_associate_fd+0x2b8()
fffffe8001773ec0 portfs+0x303()
fffffe8001773ed0 portfs32+0x24()
fffffe8001773f20 sys_syscall32+0xd9()


stack pointer for thread fffffe80f83a5de0: fffffe80010e8c10
fffffe80010e8d10 tcp_close+0xff()
fffffe80010e8d50 qdetach+0x84()
fffffe80010e8dc0 strclose+0x3e4()
fffffe80010e8e00 socktpi_close+0x12b()
fffffe80010e8e30 fop_close+0x2a()
fffffe80010e8e60 closef+0x62()
fffffe80010e8ec0 closeandsetf+0x249()
fffffe80010e8ed0 close+0xb()
fffffe80010e8f20 sys_syscall32+0xd9()


stack pointer for thread fffffe8000f56c80: fffffe8000f56330
fffffe8000f56c80 5()


Huh?

Wednesday, January 11, 2006

This can't be serious...

I've often wondered why Solaris patches take so long to apply.

I'm still wondering, after updating the Java patch (118668, for the technically minded). OK, so it's a big patch, but I've got a high-spec dual Opteron W2100z so it should be done in the blink of an eye.

Or maybe not. I decided to back out the old revision (this is one of those foibles I have - I tend to apply patches regularly on test boxes, so they get every revision, so I have a habit of backing out old revisions to keep things clean). The backout took 10 minutes, adding the new version 5. That's 15 minutes on one of the fastest machines around. Ouch!

One thing I did notice is that the patch backout was writing at an average of 30Mbytes/s for most of that 10 minutes. Overall, I reckon that I had about 15Gbytes of disk writes. Why on earth?????

Clearly it's up to something very clever here.

(For comparison, simply installing that version of java - which is about 135M - generated about 200Mbytes of disk writes and took about 20 seconds. And some of that is accepting the license, unpacking and verifying the archive, and building the jars. Clearly there's some scope for improvement.)

Software Stacks

I liked the idea that Ben Rockwood came up with - SIDEkick. Essentially, a complete software stack, in this case for a php-powered postgres-backed web server, in a single file.

This is along the lines of my simplicity rant the other day. What I want is things set up ready to go.

I do this myself, for most of the projects I get involved in. For a project, I build up a software stack that contains all the components necessary, with an install script that does the work - and not only installs the stack but can also put project-specific customizations in place, and install data if required. It's largely self-documenting as well, as the install script contains all the tweaks and kludges I needed to get the thing to work.

The full stack actually has quite a bundle of components. The primary ones are:

I still use Apache 1.3.X, due to problems I've had with 2.0.X and 2.2.0 not working right. Of course, not all projects require the full set.

As an example, I've used this software stack to enable easy deployment of DSpace, in addition to basic web servers and complex soap application servers.

Which raises the valid question - why build the stack myself?

There are several reasons. Being self-contained is a pretty good reason, all on its own - having everything in a single bundle with all dependencies means you don't have to worry about how to integrate it with some other component that may or may not be installed, may or may not be the right version, and may or may not be configured compatibly. It just works. You make yourself independent of other suppliers, who may change things underneath you without your knowledge. And it fully documents and supplies your requirements so that someone else can come along later and not only understand what you did but has all the bits already at hand to reproduce it.

What I would like to see is more lightweight software stacks developed - in a modular fashion so that I can just click a link and get a single file that gives me a well-defined area of functionality. Big frameworks are all very well, but there's an awful lot of complexity due to the generality that comes with a full framework, and that gets in the way of actually doing work.

This also means that I don't really want - at all - bits of the stack bundled with the operating system. These are just a pain and get in the way. I've nothing against OS suppliers having the full software stack available, but please compartmentalize it so I can safely ignore it.

Monday, January 09, 2006

Licensing Complexity

Following on the heels of my rant about keeping software simple, is another one:

Why are licensing schemes so horrendous?

A while ago, I was looking at backup solutions. I've just started to get costings together. What a nightmare!

Really, how hard can this be? But then there are different tiers of servers, prices are different for different platforms, some of it's done by volume (how many terabytes can you afford?), tape libraries by capacity in several dozen steps. There are different base products, with no explanation of the differences.

(This sort of thing screams out for an online order form where you just tick the boxes and it puts together an order for you. How do I know whether I've actually chosen the right options from 6 densely packed pages?)

Friday, January 06, 2006

Updated NVIDIA drivers [again]

Dang, after only just noticing the latest Solaris NVIDIA driver release, they've gone and updated the drivers again.

Start out Simple

One of the failings of modern IT infrastructure is that it's far too complex. Individual components are complex, and have complex interdependencies with other complex components. And some of us have to make sense of this mess every day.

Sometimes, complexity is unavoidable. That's OK. If it's a complex problem, then I expect some level of complexity in the solution. It would be better if the solution were simple, but we can't always have what we want.

Sometimes, new technologies come along that radically simplify the way that things are done. ZFS is one recent example - it takes away whole layers of complexity. But, as a rule, things get more complex over time as layer upon layer of cruft is added.

While you may need something big and complicated to solve the big and complicated version of a problem, does that mean that you need to be equally big and complicated to solve the little version of that problem? It seems that, all too often, you do need the big complicated version - with all its attendant hassles - to solve the little problems. Or, at least, that that's what we end up using.

I've always been opposed to this approach. I've always been in favour of starting off small and simple. I want to get something working, without delving deep into an impenetrable morass of configuration and tuning. Then, having got that to work (and, more to the point, having understood it), I can build on that foundation.

I've spent some time playing with the Java Enterprise System. And you know what? It's way too complex and hard to get into. I'm sure it can do wonderful things, but before it can do wonderful things I want it to do something. Anything, really - just to give me the sense of accomplishment that keeps me going to the next stage. I really can't see JES getting that much of the market, simply because most admins and organisations simple don't have the time and energy to invest in making it work at all.

Part of this is ease-of-use, but it's slightly different. It's really about the ease of getting started. And that's what complex technologies need to supply: an easy way in, to allow potential users to get started.

Tuesday, December 20, 2005

Updated NVIDIA drivers

For those not constantly checking, NVIDIA have released updated Solaris drivers.

(OK, so it was a few days ago, but I had missed it.)

LOSUG 2

Went along to the LOSUG meeting last night. Good to meet up with everyone again.

Mulled wine; nibbles; good talks; even the occasional mince pie finally made an appearance.

The lightning talks part worked pretty well.

Even managed a quick pint before having to leg it for the train home.

Monday, December 12, 2005

Domino Backup

One of the problems I'm working on at the moment is online backup of a Lotus Domino server running on Solaris.

Nothing too complicated, right? Just whip out your favourite backup solution , install the domino module, and you're good to go. Right? Wrong!

I've tried Legato Networker, which I've used for regular backups without any problems for the best part of a decade. Works on a trivial test, fails completely on the real thing. I've tried Backup Express from Syncsort (used by our PC systems) and haven't yet managed to persuade it to recognise that I've got a domino server installed.

I stumbled across BakBone, who make something called NetVault. I had never heard of it, but first impressions from the web site were good, and I was able to get an eval copy off their download site straight away. Installed pretty easily, and it wasn't too hard to work out how to drive it, so it's currently doing a test backup. (Performance isn't too bad, especially considering I've set it up to save to a disk based virtual library on the same disk array that the Domino server lives on.)

The real test, of course, is to wipe the Domino server out completely and see what happens if you restore it. More on that phase as it happens.

Tuesday, December 06, 2005

[ID 335743 kern.notice] BAD TRAP:

Bother!

As you may recall, I've been playing with apache httpd 2.2.0.

I was also looking at Derek Crudgington's comparison of Apache and Sun Webserver. So I decided to just test out Apache 1.3.34 against 2.2.0, and also going through to tomcat which was generating dynamic pages from mysql.

To cut a long story short, testing Apache 2.2.0 paniced my machine. I wasn't able to do too much damage with 1.3.34, but under stress 2.2.0 became rather sluggish, and then it and the whole machine became completely unresponsive.

Of course, it's not apache's fault. It shouldn't be capable of taking the box out. This is definitely something in Solaris that's gone awry.

A quick search of sunsolve didn't show a match, but for the enthusiast here's the (trimmed) message:


Dec 6 11:32:28 ratbert genunix: fffffe8001773b80 unix:die+da (fffffe8001773c20, 1fb955d3a)
Dec 6 11:32:28 ratbert genunix: fffffe8001773c60 unix:trap+5ea ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773c70 unix:cmntrap+11b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773d70 genunix:list_remove+b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773da0 genunix:port_remove_done_event+4b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773e10 portfs:port_associate_fd+2b8 ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773ec0 portfs:portfs+303 ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773ed0 portfs:portfs32+24 ()

Simple presentations

One of the things almost everyone has to do sooner or later is make presentations. And while some people can just stand up for a few minutes and hold an audience's attention, most of us need some sort of visual aids.

Personally, good old fashioned overhead foils work fine. But we're in the 21st century, and it's almost impossible to find an overhead projector.

So the general solution appears to be powerpoint or something similar. I've been using StarOffice Impress for a few years now. It sort of works, but it's a very clunky way of doing things.

Yesterday I came across Eric Meyer's S5, a simple but phenomenally powerful slide show system. Using a combination of XHTML and CSS you can easily and quickly put a simple presentation together. If you can edit basic html, you can put together a presentation. (And using superior content creation tools like emacs or vi, at that.)

As a very trivial example, I've put together a presentation on JKstat.

Monday, December 05, 2005

New Apache

Just recently, Apache httpd 2.2.0 was released.

Now, I've been a bit traditional here. I'm still using apache 1.3.34, almost always with mod_jk to talk to tomcat, and occasionally with mod_ssl for https support.

Why not upgrade? Well, there are two reasons really. One was that it wasn't at all obvious that 2.0.x was in fact an upgrade. It always felt like a retrograde step and if anything I would describe 1.3.x to 2.0.x as downgrading. The second is that actually getting 2.0.x installed was a right pain. They mistakenly switched to using autoconf, so it's much harder to get the installation and configuration right. (If it installs at all. Many a time I would find autoconf just goofing out on one of its random guesses and failing to do anything at all.)

So, is 2.2.0 any better? Well, it still uses autoconf - and it's still a very bad move - but it is possible with enough effort to circumvent most of its mistakes (although not all). But it includes ajp support to talk to tomcat, including load balancing. Which, coupled with native ssl support, should reduce the complexity of installation - if it works.

So far, running under Solaris 10 works fine for http. I haven't exhaustively tested the ajp support to see how well it handles load balancing and failover, but basically it works and looks good.

I had fun and games with getting https to work though. Essentially, the combination of httpd 2.2.0, gcc and the Solaris 10 openssl libraries didn't work. (Using curl I could get sslv2 to work but not sslv3, and neither mozilla nor firefox would have anything to d with my server.) Compiling up the latest openssl myself (which is what I do for 1.3.x anyway) with gcc works just fine. So it's either a gcc vs. cc incompatibility, or a version problem (Sun are supplying quite an old version), or some other strange incompatibility. It would be nice if I could rely on the openssl bits that come with Solaris, as openssl itself is a reasonable size and takes quite a while to build, but it looks as if I still have to do it myself.