Monday, January 30, 2006

Why I use Windows

I have a confession to make.

My main home computer runs Windows XP. Now, I'm a Solaris guy, so you would expect me to have banished Windows, but no, most of the times I use a computer at home it's the Windows box that gets the nod.

There are reasons for this renegade behaviour. The two main ones are Windows ability to do suspend-to-RAM and fast user switching. Essentially, if I just want to check my email quick, then either the Windows machine is on already and I just have to switch user, or I hit the power button. In both cases it's about the time it takes me to get the seat located comfortably before I'm online and working.

Solaris can't match this. If I'm working at home then the Sun box comes on, as the superior behaviour and environment are worth waiting 10 minutes for. The boot time is awful; the JDS startup time is awful; suspend resume is only on sparc and is very much hit and miss. If we're after wider adoption, then this is a huge area we have to address.

newboot 2 - nvidia failure

OK, so having had great success upgrading my home W2100z to Solaris 10 Update 1, I tried it on my work machine.

Now, I wasn't expecting this to be entirely trouble free. My work machine gets all sorts of abuse, with various test versions of anything that might be lying about - so before it started I cleaned up an old zfs beta release, deleted all the different sorts of backup software I had been trying, and tried to clean up and back out all the hacks and kludges.

In the end, none of the problems I was anticipating surfaced. But when it came back up - no graphics. Looking at the /dev/nvidia* and /dev/fb* entries, they looked very suspect.

Turns out this is a known problem with the NVIDIA drivers and the S10U1 upgrade. And it was essentially what I had surmised by looking at the device entries, so a quick deinstall, reinstall, clear up /etc/path_to_inst and a couple of reconfiguration reboots later and everything's back in business again.

Phew!

Saturday, January 28, 2006

newboot

Got my Solaris 10 1/06 DVD kit yesterday. So popped the x86 DVD into my W2100z (running Solaris 10 FCS) and let it get on with it.

Upgraded without a hitch. Everything looks good!

The boxed set includes quite a lot of stuff, so I also updated the freeware, and added the Studio 11 compilers. I'm leaving the Java Enterprise System for another time - I'm not going to risk playing with it on my main personal machine!

One interesting thing - I've got a decent monitor (a Sun 21 inch CRT) on this machine, and it had been running at 1280x1024. I hadn't really had time to investigate tweaking Xorg to get it to run at higher resolution. After upgrading, it came up on its own at 2048x1536. This was a little over the top, so I dropped it back to 1600x1200, which is the max recommended anyway, and runs at 85Hz rather than 75Hz, and makes the text a little more readable.

Everything still works, all I need now is a little more time to get back into development work.

Thursday, January 26, 2006

Buried in good stuff

The good stuff keeps on coming.

OpenSolaris now has even more communities - particularly for sysadmins and appliances - to follow. As if approachability, networking, observability, zfs, and zones weren't enough. And that's only half the communities I'm interested in!

What I really need is a time machine so I'm able to keep track of it all.

Tuesday, January 17, 2006

Enemy action?

According to Auric Goldfinger:

Once is happenstance. Twice is coincidence. Three times is enemy action.

One: panic[cpu1]/thread=fffffe80fd47aea0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8001773c70 addr=0 occurred in module "genunix" due to a NULL pointer dereference

Two: panic[cpu0]/thread=fffffe80f83a5de0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe80010e8bd0 addr=0 occurred in module "unix" due to a NULL pointer dereference

Three: panic[cpu1]/thread=fffffe8000f56c80:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8000f56450 addr=0 occurred in module "conskbd" due to a NULL pointer dereference

That's 3 failures on my desktop machine in just over a month. For those interested, the stack traces are:


stack pointer for thread fffffe80fd47aea0: fffffe8001773cb0
fffffe8001773d70 0xffffffff8a965940()
fffffe8001773da0 port_remove_done_event+0x4b()
fffffe8001773e10 port_associate_fd+0x2b8()
fffffe8001773ec0 portfs+0x303()
fffffe8001773ed0 portfs32+0x24()
fffffe8001773f20 sys_syscall32+0xd9()


stack pointer for thread fffffe80f83a5de0: fffffe80010e8c10
fffffe80010e8d10 tcp_close+0xff()
fffffe80010e8d50 qdetach+0x84()
fffffe80010e8dc0 strclose+0x3e4()
fffffe80010e8e00 socktpi_close+0x12b()
fffffe80010e8e30 fop_close+0x2a()
fffffe80010e8e60 closef+0x62()
fffffe80010e8ec0 closeandsetf+0x249()
fffffe80010e8ed0 close+0xb()
fffffe80010e8f20 sys_syscall32+0xd9()


stack pointer for thread fffffe8000f56c80: fffffe8000f56330
fffffe8000f56c80 5()


Huh?

Wednesday, January 11, 2006

This can't be serious...

I've often wondered why Solaris patches take so long to apply.

I'm still wondering, after updating the Java patch (118668, for the technically minded). OK, so it's a big patch, but I've got a high-spec dual Opteron W2100z so it should be done in the blink of an eye.

Or maybe not. I decided to back out the old revision (this is one of those foibles I have - I tend to apply patches regularly on test boxes, so they get every revision, so I have a habit of backing out old revisions to keep things clean). The backout took 10 minutes, adding the new version 5. That's 15 minutes on one of the fastest machines around. Ouch!

One thing I did notice is that the patch backout was writing at an average of 30Mbytes/s for most of that 10 minutes. Overall, I reckon that I had about 15Gbytes of disk writes. Why on earth?????

Clearly it's up to something very clever here.

(For comparison, simply installing that version of java - which is about 135M - generated about 200Mbytes of disk writes and took about 20 seconds. And some of that is accepting the license, unpacking and verifying the archive, and building the jars. Clearly there's some scope for improvement.)

Software Stacks

I liked the idea that Ben Rockwood came up with - SIDEkick. Essentially, a complete software stack, in this case for a php-powered postgres-backed web server, in a single file.

This is along the lines of my simplicity rant the other day. What I want is things set up ready to go.

I do this myself, for most of the projects I get involved in. For a project, I build up a software stack that contains all the components necessary, with an install script that does the work - and not only installs the stack but can also put project-specific customizations in place, and install data if required. It's largely self-documenting as well, as the install script contains all the tweaks and kludges I needed to get the thing to work.

The full stack actually has quite a bundle of components. The primary ones are:

I still use Apache 1.3.X, due to problems I've had with 2.0.X and 2.2.0 not working right. Of course, not all projects require the full set.

As an example, I've used this software stack to enable easy deployment of DSpace, in addition to basic web servers and complex soap application servers.

Which raises the valid question - why build the stack myself?

There are several reasons. Being self-contained is a pretty good reason, all on its own - having everything in a single bundle with all dependencies means you don't have to worry about how to integrate it with some other component that may or may not be installed, may or may not be the right version, and may or may not be configured compatibly. It just works. You make yourself independent of other suppliers, who may change things underneath you without your knowledge. And it fully documents and supplies your requirements so that someone else can come along later and not only understand what you did but has all the bits already at hand to reproduce it.

What I would like to see is more lightweight software stacks developed - in a modular fashion so that I can just click a link and get a single file that gives me a well-defined area of functionality. Big frameworks are all very well, but there's an awful lot of complexity due to the generality that comes with a full framework, and that gets in the way of actually doing work.

This also means that I don't really want - at all - bits of the stack bundled with the operating system. These are just a pain and get in the way. I've nothing against OS suppliers having the full software stack available, but please compartmentalize it so I can safely ignore it.

Monday, January 09, 2006

Licensing Complexity

Following on the heels of my rant about keeping software simple, is another one:

Why are licensing schemes so horrendous?

A while ago, I was looking at backup solutions. I've just started to get costings together. What a nightmare!

Really, how hard can this be? But then there are different tiers of servers, prices are different for different platforms, some of it's done by volume (how many terabytes can you afford?), tape libraries by capacity in several dozen steps. There are different base products, with no explanation of the differences.

(This sort of thing screams out for an online order form where you just tick the boxes and it puts together an order for you. How do I know whether I've actually chosen the right options from 6 densely packed pages?)

Friday, January 06, 2006

Updated NVIDIA drivers [again]

Dang, after only just noticing the latest Solaris NVIDIA driver release, they've gone and updated the drivers again.

Start out Simple

One of the failings of modern IT infrastructure is that it's far too complex. Individual components are complex, and have complex interdependencies with other complex components. And some of us have to make sense of this mess every day.

Sometimes, complexity is unavoidable. That's OK. If it's a complex problem, then I expect some level of complexity in the solution. It would be better if the solution were simple, but we can't always have what we want.

Sometimes, new technologies come along that radically simplify the way that things are done. ZFS is one recent example - it takes away whole layers of complexity. But, as a rule, things get more complex over time as layer upon layer of cruft is added.

While you may need something big and complicated to solve the big and complicated version of a problem, does that mean that you need to be equally big and complicated to solve the little version of that problem? It seems that, all too often, you do need the big complicated version - with all its attendant hassles - to solve the little problems. Or, at least, that that's what we end up using.

I've always been opposed to this approach. I've always been in favour of starting off small and simple. I want to get something working, without delving deep into an impenetrable morass of configuration and tuning. Then, having got that to work (and, more to the point, having understood it), I can build on that foundation.

I've spent some time playing with the Java Enterprise System. And you know what? It's way too complex and hard to get into. I'm sure it can do wonderful things, but before it can do wonderful things I want it to do something. Anything, really - just to give me the sense of accomplishment that keeps me going to the next stage. I really can't see JES getting that much of the market, simply because most admins and organisations simple don't have the time and energy to invest in making it work at all.

Part of this is ease-of-use, but it's slightly different. It's really about the ease of getting started. And that's what complex technologies need to supply: an easy way in, to allow potential users to get started.

Tuesday, December 20, 2005

Updated NVIDIA drivers

For those not constantly checking, NVIDIA have released updated Solaris drivers.

(OK, so it was a few days ago, but I had missed it.)

LOSUG 2

Went along to the LOSUG meeting last night. Good to meet up with everyone again.

Mulled wine; nibbles; good talks; even the occasional mince pie finally made an appearance.

The lightning talks part worked pretty well.

Even managed a quick pint before having to leg it for the train home.

Monday, December 12, 2005

Domino Backup

One of the problems I'm working on at the moment is online backup of a Lotus Domino server running on Solaris.

Nothing too complicated, right? Just whip out your favourite backup solution , install the domino module, and you're good to go. Right? Wrong!

I've tried Legato Networker, which I've used for regular backups without any problems for the best part of a decade. Works on a trivial test, fails completely on the real thing. I've tried Backup Express from Syncsort (used by our PC systems) and haven't yet managed to persuade it to recognise that I've got a domino server installed.

I stumbled across BakBone, who make something called NetVault. I had never heard of it, but first impressions from the web site were good, and I was able to get an eval copy off their download site straight away. Installed pretty easily, and it wasn't too hard to work out how to drive it, so it's currently doing a test backup. (Performance isn't too bad, especially considering I've set it up to save to a disk based virtual library on the same disk array that the Domino server lives on.)

The real test, of course, is to wipe the Domino server out completely and see what happens if you restore it. More on that phase as it happens.

Tuesday, December 06, 2005

[ID 335743 kern.notice] BAD TRAP:

Bother!

As you may recall, I've been playing with apache httpd 2.2.0.

I was also looking at Derek Crudgington's comparison of Apache and Sun Webserver. So I decided to just test out Apache 1.3.34 against 2.2.0, and also going through to tomcat which was generating dynamic pages from mysql.

To cut a long story short, testing Apache 2.2.0 paniced my machine. I wasn't able to do too much damage with 1.3.34, but under stress 2.2.0 became rather sluggish, and then it and the whole machine became completely unresponsive.

Of course, it's not apache's fault. It shouldn't be capable of taking the box out. This is definitely something in Solaris that's gone awry.

A quick search of sunsolve didn't show a match, but for the enthusiast here's the (trimmed) message:


Dec 6 11:32:28 ratbert genunix: fffffe8001773b80 unix:die+da (fffffe8001773c20, 1fb955d3a)
Dec 6 11:32:28 ratbert genunix: fffffe8001773c60 unix:trap+5ea ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773c70 unix:cmntrap+11b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773d70 genunix:list_remove+b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773da0 genunix:port_remove_done_event+4b ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773e10 portfs:port_associate_fd+2b8 ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773ec0 portfs:portfs+303 ()
Dec 6 11:32:28 ratbert genunix: fffffe8001773ed0 portfs:portfs32+24 ()

Simple presentations

One of the things almost everyone has to do sooner or later is make presentations. And while some people can just stand up for a few minutes and hold an audience's attention, most of us need some sort of visual aids.

Personally, good old fashioned overhead foils work fine. But we're in the 21st century, and it's almost impossible to find an overhead projector.

So the general solution appears to be powerpoint or something similar. I've been using StarOffice Impress for a few years now. It sort of works, but it's a very clunky way of doing things.

Yesterday I came across Eric Meyer's S5, a simple but phenomenally powerful slide show system. Using a combination of XHTML and CSS you can easily and quickly put a simple presentation together. If you can edit basic html, you can put together a presentation. (And using superior content creation tools like emacs or vi, at that.)

As a very trivial example, I've put together a presentation on JKstat.

Monday, December 05, 2005

New Apache

Just recently, Apache httpd 2.2.0 was released.

Now, I've been a bit traditional here. I'm still using apache 1.3.34, almost always with mod_jk to talk to tomcat, and occasionally with mod_ssl for https support.

Why not upgrade? Well, there are two reasons really. One was that it wasn't at all obvious that 2.0.x was in fact an upgrade. It always felt like a retrograde step and if anything I would describe 1.3.x to 2.0.x as downgrading. The second is that actually getting 2.0.x installed was a right pain. They mistakenly switched to using autoconf, so it's much harder to get the installation and configuration right. (If it installs at all. Many a time I would find autoconf just goofing out on one of its random guesses and failing to do anything at all.)

So, is 2.2.0 any better? Well, it still uses autoconf - and it's still a very bad move - but it is possible with enough effort to circumvent most of its mistakes (although not all). But it includes ajp support to talk to tomcat, including load balancing. Which, coupled with native ssl support, should reduce the complexity of installation - if it works.

So far, running under Solaris 10 works fine for http. I haven't exhaustively tested the ajp support to see how well it handles load balancing and failover, but basically it works and looks good.

I had fun and games with getting https to work though. Essentially, the combination of httpd 2.2.0, gcc and the Solaris 10 openssl libraries didn't work. (Using curl I could get sslv2 to work but not sslv3, and neither mozilla nor firefox would have anything to d with my server.) Compiling up the latest openssl myself (which is what I do for 1.3.x anyway) with gcc works just fine. So it's either a gcc vs. cc incompatibility, or a version problem (Sun are supplying quite an old version), or some other strange incompatibility. It would be nice if I could rely on the openssl bits that come with Solaris, as openssl itself is a reasonable size and takes quite a while to build, but it looks as if I still have to do it myself.

Wednesday, November 30, 2005

Awash with freebies...

Seems like Xmas has come early.

Of course, Sun go on at length about the Java Enterprise System. Now, this is interesting in parts, but JES is a complicated beast and likely to be of interest primarily to - well - Enterprises.

What I like, though, is the promise of free stuff a bit further down. There has been a good emphasis on developers recently - Studio 11 and Creator, for example. But what's also now promised is free versions of Tarantella and SunRay, which are likely to be of interest to a far wider range of customers.

And as I read it, the nebulous N1, including Sun Management Center, is included in the deal too.

Tuesday, November 29, 2005

suspend/resume at a crawl

I've got a Sun Blade 1500 at home (one of the old red ones). Works great.

Apart from suspend/resume, that is.

I have no idea why, but both suspend and resume take an absurd amount of time. The suspend isn't too bad (slower than it should be), but resume is in the 5-10 minute range. To use an Americanism, this sucks.

(It's doubly odd because I've tried this on a Blade 150, and that's much, much quicker.)

Get it right first time!

Many years ago I wrote a simple system and network monitoring tool. It's been developed on and off over the years, but has now reached a major impasse.

Basically, I designed it wrong 10 years ago. I started out with a 2 state system. If the status is 0, then it's fine. If the status is 1, it's broken and needs fixing. Sounds reasonable, right?

Then I realized I needed to add another state, so I defined it so that if the status is 2, there's a warning condition. And all worked well for a few years.

The problem with this scheme is that the severity of the problem isn't a linear function of the status. So I end up playing all sorts of games trying to analyze the status codes trying to work out just how bad the situation really is. It would be much easier if I could simply retrieve the maximum status out of the database - no fiddling required! And I can order problems simply by sorting on the status.

Thinking about this a bit more, this is the obvious thing to do. So obvious, in fact, that I was a dullard for not thinking about this at the start. (But, when I started writing this particular monitoring tool, I wasn't thinking about what version 3 would look like 10 years down the line. And I started out by using the return code from scripts as the status, which is where 0 and 1 came from.)

Of course, I now have to consider what the best scheme might be. Do I simply have 0 for good, 1, for warning, 2 for dead? I think the 0 for good is fine. But should I do something like 255 for dead, 128 for warning, leaving me some room to add finer levels of granularity in the future?

Decisions, decisions...

Saturday, November 26, 2005

Another JKstat update

I've updated JKstat to version 0.08.

It's getting better. The accessory widgets have been cleaned up and a couple of new ones added (distribution of packet sizes on bge interfaces, and dma transfer rate on ifb graphics cards). Rates are now accurately computed based on the actual snaptime, rather than approximately based on the intended refresh interval. A couple of internal changes streamline the whole system. And I've fixed it so that actually enumerating the kstats doesn't blindly read all the data, which improves performance.

With these changes, I'm much happier that it's closing in on its design goals. I was tempted to bump the version up to 0.1, but that would probably be premature based on the number of bugs that I introduced and fixed recently.

My next idea is to build a graphical iostat. Why is this of value? Well, pictures tell you a lot - the eye is very good at interpreting graphical data. You can dynamically hide uninteresting data, or expand areas of interest for a finer view (for example, you could dynamically expand a disk's I/O to show partition data). You can show historical rates, and generally have multiple views of the same data. You can use the gui to show additional context-sensitive data beyond the basic I/O data. And you could, in the future, link to other areas of functionality - such as dtrace to show what was causing all that I/O in the first place.