When to bury the pager

If anyone's been following me on twitter recently you may have noticed a few fraught messages about SANs and pagers.

We have an on-call rota. Being a relatively small department, this actually means that we cover the entire department - so it's possible that I might get a call to sort out a Windows problem, or that one of the Windows guys might get to sort out one of my Sun servers. But it's not usually too stressful.

This last week has been a bit of a nightmare and the problem has been so bad and so apparently intractable that I've simply buried the pager, turned off notification of email and texts on the phone, and relied on someone phoning me if anything new came up. Otherwise I would get woken up several hundred times a night for no good purpose.

Of course, today being the final day of my stint (yay!) I finally work out what's causing it.

What we've been having is the SAN storage on one of our boxes going offline. Erratically, unpredictably, and pretty often. Started last Friday, continuing on and off since.

This isn't the first time. We've seen some isolated panics, and updated drivers. They fix the panic, for sure, but now it just stays broken when it sees a problem. The system vendor, the storage vendor, and the HBA vendor got involved.

We've tried a number of fixes. Replaced the HBA. Made no difference. Put another HBA in a different slot. Made no difference. Tried running one port on each HBA rather than 2 on one. Made no difference. We're seeing problems down all paths to the storage (pretty much equally).

Last night (OK, early this morning) I noticed that the block addresses that were reporting errors weren't entirely random. There were a set of blocks that were being reported again and again. And the errors come in groups, but each group contained one of the common blocks (presumably the other were just random addresses that happened to be being accessed during the error state).

I've had conversations with some users who've been having trouble getting one of their applications to run to completion with all the problems we've had. And they're getting fraught because they have deadlines to meet.

And then I start putting two and two together. Can I find out exactly when they were running their application? OK, so they started last Friday (just about when the problem started). And we know that the system was fine for a while after a reboot, and going back it turns out that either a plain reboot, or a reboot for hardware replacement, kills whatever they're doing, and it may be later in the evening or the next morning before they start work again.

So, it's an absolutely massive coincidence - an almost perfect correlation - that we have problems that kill the entire system for hours an hour after they start their applications up, and the problems finish within seconds of their application completing a task.

So, it looks very much like there's something in their data that's killing either the SAN, the HBA, or the driver. Some random pattern of bits that causes something involved to just freak out. (I don't really thing it's a storage hardware error. It could be, but there are so many layers of abstraction and virtualisation in the way that a regular bad block would get mangled long before it gets to my server.) And it's only the one dataset that's causing grief - we have lots of other applications, and lots of servers, and none of them are seeing significant problems.

So, we can fix the problem - just don't run that thing!

And then I realize that I've seen this before. Now that's on a completely different model of server running a different version of solaris running a different filesystem on different storage. But it's files (different files) but from the same project. Creepy.

Thank heaven for sparse files!

We use ImageMagick to do a lot of image processing. I'm not sure what it's up to, but some processing needs to create temporary working files that can be quite large (in /var/tmp by default, I've moved them with TMPDIR because that filled up).

However, I now see this:

/bin/ls -l /storage/tmp
total 203037932
-rw-------   1 user grp       169845176062560 Sep  4 11:18 magick-XXT0aaXE
-rw-------   1 user grp       222497224416 Sep  4 13:24 magick-XXU0aaXE
-rw-------   1 user grp       11499827272024 Sep  4 13:11 magick-XXbFaiKF
-rw-------   1 user grp       15064771904 Sep  4 13:24 magick-XXcFaiKF
-rw-------   1 user grp       18904557170048 Sep  4 10:51 magick-XXtlaGCE
-rw-------   1 user grp       24764978480 Sep  4 13:24 magick-XXulaGCE

or, in more readable units a few seconds later:

/bin/ls -lhs /storage/tmp
total 203038194
33272257 -rw-------   1 user grp          154T Sep  4 11:18 magick-XXT0aaXE
34295031 -rw-------   1 user grp          207G Sep  4 13:24 magick-XXU0aaXE
29432967 -rw-------   1 user grp           10T Sep  4 13:11 magick-XXbFaiKF
9271301 -rw-------   1 user grp           14G Sep  4 13:24 magick-XXcFaiKF
48382483 -rw-------   1 user grp           17T Sep  4 10:51 magick-XXtlaGCE
48384155 -rw-------   1 user grp           23G Sep  4 13:24 magick-XXulaGCE

Ouch. That's on an internal 146G drive.

What on earth is it doing with a 154 terabyte file?

JKstat meets the ZFS ARC

Recently, Ben Rockwood posted a useful script to display ZFS cache statistics.

Now, all it's doing is grabbing kstats, so it wasn't much of a stretch to put together a new version of JKstat that has a new demo to display the ZFS cache statistics. Try

jkstat arcstat

.
(Requires OpenSolaris, Solaris Nevada, or Solaris 10 8/07 or later to actually have the kstats to display.)

This release of JKstat is a bit rough, as there are a few other things I was working on that aren't neatly finished off yet, but I thought it worth putting out just for the arcstat demo - any comments and suggestions for improvement would be gratefully appreciated!

So, download JKstat now - and here's a little snapshot of the new demo:

Computers - unpredictable creatures

Computers are unpredictable beasts. You would think they would be more deterministic, but reality is otherwise.

I have a server with a tape drive. We've used it for about a year, most days. Then suddenly we start getting errors. At first we thought it was a bad tape, but then multiple tapes started giving us grief. Easy enough, just use a different drive. I finally got around to debugging it last week. Swapped the drives over - still errors. Turned out to be a bad cable. That's a new one - I've not seen a SCSI cable fail like that before. (Usually they fail straight away or when you change something, not after working stably and untouched for the best part of a year.)

Yesterday I set up SNMP on some machines for monitoring purposes. Pointed the monitoring system at them, and a couple of minutes later a couple stop responding. That wasn't part of the plan. So I go to the LOM interface, and they're powered off. Call the datacenter, they haven't done anything. I have seen strange things, but snmp (running unprivileged, I might add) powering a machine off when queried? So I tell them to power themselves back on. One comes up fine, the other boots but no ZFS filesystems or zones. I try format. No SAN disks. And then:

# fcinfo hba-port
No Adapters Found.

Yikes! It had a couple of fiber-channel HBAs in it a few minutes ago.

I still don't know what happened, but some electrical gremlins had gotten into the works. So the machines had obviously shut themselves off due to lack of power. And I'm guessing that the PSUs were capable of supplying just enough power to boot the machine, but not enough to get the HBAs powered up properly. Another new failure mode to go in the book.

The gardening release

I've been developing SolView and JKstat for a while. Over time, code goes stale so I thought it was time for a good cleanup.

Rather than trying to spot all the bad code by eye, I sought a tool that would find unused code, unused imports, and generally find problems automatically. (And I already run javac with the -Xlint:unchecked flag, and use the OpenSolaris jstyle utility.)

After a little looking around, I found PMD and find it very useful. It's done an excellent job of finding poor code. Fortunately, it hasn't found any killer bugs, but has pointed out plenty of cases where I've been sloppy. One bad habit I've got into is unnecessarily declaring class fields rather than local variables, for example. So I recommend it. (I don't regard this as the end of the exercise - I plan to keep looking to see what other static analysis tools can tell me about my code.)

As a result, I've released new versions of SolView and JKstat that have been cleaned up. I haven't done much else to JKstat, although I have enabled the ability for SolView to just show the panel you're primarily interested in (such as just the services, or the general information) which makes it rather lighter weight.

JKstat, SolView, Awards

I've released new versions of JKstat and SolView.

In JKstat, I've added Kstat aggregations. These are used in an enhanced cpustate demo to show the aggregate cpu statistics of a multithreaded core, or a multicore processor. This also needed me to work out how the various cpu kstats were related, so I knew which cpu corresponded to which thread and core of a complex multithreaded/multicore system. (A version of psrinfo naturally fell out of this as something I needed for testing.) There are a couple of new charts - for cpustate and the netload demo.

For SolView, I've added access to the logfiles for SMF services, and also a tree view of the SMF services. The tree is based on the service naming hierarchy, not the dependency tree, and is an experiment to see if that's a useful description of the services (as opposed to a straight list that's a couple of hundred lines long).

These are the entries I've submitted to the OpenSolaris Community Innovation Awards Program, and I'm hoping for some success there.

You don't exist, go away!

Oh dear. I try to run ssh and I get:

You don't exist, go away!

Which is sort of correct. I'm changing all the userids (including my own) while logged into the system, so that the shell I was running this from was under the old (now invalid) userid. Still, I was a little surprised at the bluntness of the response.

Making scp go a little quicker

Transferring files with scp isn't the quickest option, but if it's the only one there's a simple way to make it go a little quicker.

scp -c blowfish source.file remote.host:/destination/file.name

Using blowfish rather than the default 3des gave me about an extra 50% or so of throughput.

This was really noticeable on my new T5120s, where I went from under 10M/s to over 13M/s for a single copy. (OK, so it can probably run lots in parallel, but I was just moving large images.)

jingle and jumble

One of the problems with developing in Java is that some relatively common tasks that ought to be simple require writing a lot of boilerplate code, or are otherwise inconvenient.

So, like many others before me, I have a bunch of classes that I use regularly. These are not clever, innovative, or terribly interesting. But they have saved me a lot of typing and repetition over the years.

I named them jingle and jumble. (There were jangle and jungle, which I think had something to do with networking and web services; I've lost those completely.)

The jingle classes help write swing applications. One of the most useful ones is a Frame registry that keeps track of how many windows you have open and allows you to close one or all of them.

In jumble, it's a case of allowing you to get a file into a string (or vice-versa) in one line.

They're used to simplify jkstat and solview.

(I don't really expect others to use them, though. I'm blogging just to note their existence.)

Subtle change on sun.com

Don't know when this was changed, but I've noticed a subtle change on Sun's website recently.

I'm sure that in the main navigation, it had Products first and Downloads second. They seem to have swapped over.

I'm not sure whether this marks a change in direction, with less emphasis on selling stuff and more on giving stuff away, or whether they're tracking visitors and ordering the navigation links by popularity.

Networking vanished

I've had a faulty X2100M2 - it's been claiming for months that it's got a failed fan.

Originally it thought all the fans had failed (although I was somewhat suspicious because the temperatures looked fine). So I replaced them once and that made it happier, but not entirely happy.

So today we tried to replace the faulty fan again. No joy, so we ended up replacing the motherboard. And after fiddling with the cables, we finally persuaded the fault light to go out.

Unfortunately, we were only halfway home. Couldn't get a peep out of the system. Nothing on the serial port, nothing on the SP, couldn't even ping it.

One of the problems was that the replacement motherboard had old firmware, so things like serial port settings were up the creek. I had to play the upgrade game. At least that got me back to sanity, and I could see the system output as it boots.

Whoa there. Couldn't bring the network up. Originally, I had bge0 and bge1; on the new motherboard it decided to call the network interfaces bge2 and bge3. Ok, rename hostname.bge0 and we're back in business, but why on earth was it that hard just to get one of the system fans to work?

SAS vs. SATA

I use Solaris zones a lot.

We've got a number of X2200s, in two variants. Some just run web front ends, and are fitted with SATA drives (once running, the only disk activity is the web server logs); the database back-ends have SAS drives.

OK, so the SAS drives are expected to be a bit quicker - we did get them for that purpose. Based solely on the rotational speed, there's about a factor of 2 difference in performance.

However, if you take zone creation time as a metric, the performance difference is rather larger than a factor of 4. Something else makes the SAS drives fly and the SATA drives crawl.

An upgrade too far

I've been using Live Upgrade on my Solaris servers recently. Normally I would prefer a fresh install, as that gives you more of an opportunity to fix up any mistakes you made, but sometimes you need to preserve the application data or can't afford the downtime.

One word of warning, though: if you're starting with Solaris 8, you can go to Solaris 10 8/07 (update 4), but not to Solaris 10 5/08 (update 5). Even when upgrading from Solaris 9 or 10 you'll need the 7zip patches, but those don't exist (yet, anyway) for Solaris 8.

Sun does quad core Opterons

So Sun are now - finally - pushing quad core Opterons in the X4140, X4240, and X4440.

The X4240 is a new one. I like it. Yes, whereas I complained before, this one does have 16 internal drives.

Living in the Ghetto

Where I work is very much a pure Microsoft shop in terms of user environment - ie. desktops.

(The company makes its money using real Unix servers.)

I'm one of the very few who actually run Solaris on a Sun workstation. And, yes, sometimes I feel like I'm being pushed into a ghetto.

A world where you have to talk to Microsoft Exchange to read your mail, which means Outlook Web Access (which, frankly, is a shockingly poor attempt at being a mail client); where you're sent documents in Office 2007 format that you can't read; where half the company intranet simply doesn't function. Catering to those of living outside the walls simply isn't in Microsoft's world view, it would seem.

So, given the list of features, I'll have to grab the OpenOffice 3 beta and give it a try.

OpenSolaris in VirtualBox

I've been playing around today with VirtualBox, after finding that the latest version claims to run on Solaris 10.

Which is true, but you need to jump through a few hoops first.

First, you need to make sure that libGL.so can be found. I guess this varies a bit depending on whether you're using mesa or have the nividia drivers, but I ended up setting LD_LIBRARY_PATH_64 to /usr/X11/lib/amd64.

Then you need libXinerama; if you're running an older version of Solaris (my test machine was running S10U3 == 11/06) then applying patch 125726-02 will do the trick.

Then if you're running 64-bit you'll need a copy of libdlpi. I just snarfed a copy off one of my test opensolaris boxes (actually indiana preview 2 - I have seen comments that the one from the official OpenSolaris release won't work as it's too new).

(Yes, I realize there might be a bit of a chicken and egg situation there!)

Then I tried booting the indiana preview 1. Which worked just great. No network, but I expected that. The only glitch I had was the key to escape from the guest - which is set to Right Ctrl by default, which I discovered I don't have. I reset that to some other key that I do have and don't use for anything else.

Having learnt from that, I had a go at the OpenSolaris 2008.05 release having found a CD that I had brought back from CommunityOne, and that worked fine (and picked up the network).

Et tu, Brute?

So Jim's posted some photos of the OpenSolaris Developer Summit.

It was great to meet so many people in person. We had some good talks, lots of discussions (not all of which actually led to results, but you can't have everything). One of the things that is clear is that open communication is vital, and we definitely had some of that.

We're on the way to building a stronger community, and there was some additional encouragement organised for us. Go Team International!

Next up, Community One.

Controlling the Install Footprint

Solaris 10 introduced two technologies that, both individually and in concert, change the way we think about installing the OS.

Zones provide very lightweight virtualization. Especially sparse-root zones, which share most of the system files (binaries such as /usr) but have their own copy of configuration files and data (/etc and /var).

SMF, the Service Management Facility, replaces the old rc system and inetd, and allows you to configure exactly what services and daemons are running on a system.

The point about SMF is that historically, the only real way to guarantee that something was disabled was to not install it at all. With SMF, it is feasible to install something and rely on SMF to make sure it won't be active. So while minimization can be done the old way, you can use SMF to minimize what's running on a much larger install.

Then think about zones, and especially sparse-root zones. Because /usr is shared, you have to install everything that's going to be used throughout all the zones. So if you need a zone that runs tomcat, you have to install tomcat globally and it's there in all the zones. (Unless you install a private copy outside the OS in the zone that needs it.) And this is where SMF comes in - you can have all the applications everywhere, but only enable them in the zones that need them.

And it's not just the applications you have now that you need to worry about - to allow for future zones, you really need to make sure that you've installed everything that you might need in the future.

And if you wish to migrate zones between machines, then you have to install everything that might be needed by any zone on your network.

The old days of "install what you need, and no more" on each system don't apply. Consolidation using zones tends to drive you towards fat installs, using SMF to manage what's active in each zone.

That's fine - up to a point. And that point is where each zone gets overloaded with the services that it might run. One part of this is manifest import, which can be the dominant factor in sparse-root zone installation times. (Thankfully work to speed this up has recently been done!)

And as time goes on, the number of services is steadily proliferating. My desktop has over 200 lines of output from svcs. I know it doesn't do 200 different things.

So I have recently been looking again at my install profiles, to see what level of additional tweaking can be done. And the emphasis here is not just on minimizing what software is installed, but minimizing what services are installed, to reduce the cost of manifest import and reduce the available number of services that have to be managed.

On my (largely untweaked) desktop, there are currently 158 SMF manifests. Broken down into packages by number, that is:

   1 SUNWaccr
   1 SUNWadmr
   1 SUNWapmsc
   1 SUNWatfsr
   1 SUNWbindr
   1 SUNWbrgr
   1 SUNWcfplr
   1 SUNWcsd
   1 SUNWdhcsr
   1 SUNWfontconfig-root
   1 SUNWftpr
   1 SUNWgnome-display-mgr-root
   1 SUNWgssc
   1 SUNWinstall-patch-utils-root
   1 SUNWipmir
   1 SUNWipplr
   1 SUNWiscsir
   1 SUNWiscsitgtr
   1 SUNWjwnsr
   1 SUNWkdcr
   1 SUNWmconr
   1 SUNWntpr
   1 SUNWocfr
   1 SUNWos86r
   1 SUNWpcr
   1 SUNWpiclr
   1 SUNWpmr
   1 SUNWppror
   1 SUNWrcapr
   1 SUNWslpr
   1 SUNWsndmr
   1 SUNWsshdr
   1 SUNWstosreg
   1 SUNWstsfr
   1 SUNWtnamr
   1 SUNWtnetr
   1 SUNWtsr
   1 SUNWwbcor
   1 SUNWxwssu
   2 SUNWbsr
   2 SUNWnfssr
   2 SUNWpoolr
   2 SUNWpsr
   2 SUNWsacom
   2 SUNWservicetagr
   2 SUNWsmmgr
   2 SUNWvbox
   2 SUNWvolr
   2 SUNWzoner
   3 SUNWckr
   3 SUNWkrbr
   3 SUNWnisr
   3 SUNWtsg
   4 SUNWsmbar
   4 SUNWxwplr
   4 SUNWypr
   5 SUNWcnsr
   5 SUNWnfscr
   6 SUNWmdr
  10 SUNWrcmdr
  49 SUNWcsr

Now I can't uninstall SUNWcsr, but there are a lot of packages there that my servers aren't going to need. So I end up removing all the WBEM, smc, service tags, telnetd, tnamd, samba, core network services, and the rcmds help a bit too.

JKstat 0.21

It's been a while since I updated JKstat. In one sense, this slowing down is indicative that the project has reached a level of comparative maturity. Alternatively, I haven't found as much time to work on it as I would like. The reality is that both are true - the first phase of producing functional software is largely finished, and I haven't had time to address the more ambitious plans I have for the next phase.

Anyway, I'm putting out an updated version of JKstat now that has just a couple of fixes. One (an excellent suggestion from Tony Curtis) is that the network accessory shows the rates in humanized units (and I ought to implement this feature more widely). The second is a bunch of NullPointerExceptions raised by empty kstats. (Yes, I eventually stumbled across some kstats that had no data at all.)

NDIS Wrapper on Indiana

I've been setting up a laptop (a Dell D600, as it happens) so I thought I would try the Indiana preview on it.

Things have moved on a bit since I did this last - the onboard ethernet is recognised out of the box. But not the wireless. Off to try the NDIS Wrapper trick.

Now, I've got the Windows driver, which gives me the .inf and .sys files, and I downloaded the latest ndis package. Now to build it.

Unlike Solaris 10, where you type make and it works first time, Indiana has been cut down to fit onto the live CD and so you need some extra stuff. Several iterations of trial and error later, and the required steps are:

pfexec pkg install SUNWgcc
pfexec pkg install SUNWhea
pfexec pkg install SUNWflexlex
pfexec pkg install SUNWgm4

(Clearly, the dependency information for these packages is incomplete. Without gm4, flex gives an error message than couldn't ever be described as useful.)

With that, the "make ndiscvt" step works. Then you need to copy the .sys and .inf files into place and run

./ndiscvt -i ndis.inf -s ndis.sys -o ndis.h

if that fails with the error

ndiscvt: line 13: e: syntax error.

then the .inf file is in utf-16 format, so you need to

iconv -f utf-16 -t ascii path/to/bcmwl5.inf > ndis.inf

and run the ndiscvt step again.

Then it really is as simple as

make ndis
pfexec cp bcmndis /kernel/drv/bcmndis
make ndisapi
pfexec cp ndisapi /kernel/misc

Looking at "/usr/X11/bin/scanpci -v", we see

pci bus 0x0002 cardnum 0x03 function 0x00: vendor 0x14e4 device 0x4320
 Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller
 CardVendor 0x1028 card 0x0001 (Dell TrueMobile 1300 WLAN Mini-PCI Card)
  STATUS    0x0010  COMMAND 0x0106
  CLASS     0x02 0x80 0x00  REVISION 0x02
  BIST      0x00  HEADER 0x00  LATENCY 0x20  CACHE 0x00
  BASE0     0xfafee000  addr 0xfafee000  MEM
  MAX_LAT   0x00  MIN_GNT 0x00  INT_PIN 0x01  INT_LINE 0x0b
  BYTE_0    0x01  BYTE_1  0x00  BYTE_2  0xc2  BYTE_3  0xff

So we need to (see the numbers for vendor and device in the first line above)

add_drv -i '"pci14e4,4320"' bcmndis

and Yay!, it attaches.

Where's my Primary Administrator gone?

One neat aspect of Solaris is RBAC, which allows you to control which actions users can perform.

A particularly blunt instrument is the 'Primary Administrator' profile. If you're a Primary Administrator, then you are effectively root - in that you can use pfexec to assume the privileges of the root account (or role).

In Indiana, for example, root is (normally) a role and the account you create at install is set up as a Primary Administrator. It's very convenient.

So I decided to implement the same mechanism on my home machines (and use RBAC to let my children do extra things without having to pester dad).

Which failed, big time. It's really easy, you just use usermod to add a profile to an account:

usermod -P 'Primary Administrator' user_name

at which point Solaris thumbed its nose at me.

UX: usermod: ERROR: Primary Administrator is not a valid profile name.  Choose another.

I decided to dig a little deeper, and then you discover that the way these profiles find their way onto the system is (ahem) strange.

The profiles are defined in some files that live in /etc/security - auth_attr, exec_attr, and prof_attr - and then /etc/user_attr controls what is assigned to users. So where do these files come from?

It turns out that different packages stick their own entries in. If you start looking around the Solaris media, then go into Solaris_XX/Product and look for */reloc/etc/security/exec_attr (and the same for prof_attr and auth_attr). These are the files that get merged into the master copy by some funky class action script. (There are things about IPS that I don't agree with, but its plan of getting rid of all these way out scripts is something that has to be good.)

OK, so looking in those files, and the Primary Administrator is delivered by the SUNWwbcor package. What's that? "Solaris WBEM Services (root)".

No wonder I hadn't got the profile. I never install WBEM or anything to do with it. Systems are much better off without it (and I couldn't ever see myself installing it on something like a home system, or indeed any system where Primary Administrator might be used). But, if you don't install that package then you're going to have to install the profile yourself. Something like the following in the Product directory on the media should do it:

cat SUNWwbcor/reloc/etc/security/auth_attr >> /etc/security/auth_attr
cat SUNWwbcor/reloc/etc/security/exec_attr >> /etc/security/exec_attr
cat SUNWwbcor/reloc/etc/security/prof_attr >> /etc/security/prof_attr

From scary to useful

Much as I was taken aback by the native cifs client that is now in OpenSolaris, there is a serious side to this.

Integration with the Microsoft world is a huge deal. You just can't ignore it. (OK, Sun did try that approach for a while.)

As the "backup guy" I was just given a simple task. Copy some old files off to tape, archiving them so they can be deleted. Simple. Only they're on a Windows server.

But now it's real easy. Just mount the share, and tar to tape.

Normally I install each SXCE build onto an old Ultra 60 that's dedicated to testing. But for this task I could justify using something just a little more modern (and something that has a faster network interface so I don't have to wait forever). OK, so a Sun Blade 1500 isn't going to set the world on fire, but it's more than capable of doing the job. And it really was just a case of mounting the share and issuing a tar command.

Job done.

X4140 - X4440

I haven't seen the announcement from Sun, but just noticed the X4140 and X4440 appear.

They look just like Opteron versions of the X4150 and X4450.

(And yes, I think it's a shame you don't have the 16-drive option for the X4440.)

The case of the unloved files

I've been investigating a strange problem on one of my servers. It started out as a simple case of NetBackup failing to back the filesystem up.

Now that's not entirely unusual - NetBackup often goes off and sulks in a corner. But this was rather different, as it didn't disappear as mysteriously as it came. Rather, it stayed put and the filesystem repeatedly refused to complete a backup. And the diagnostics are pretty much non-existent.

OK, so after a week or so of this I decide to try an alternative approach. To my surprise, I couldn't blame NetBackup.

First attempt was to try ufsdump. It started off in a promising manner, then froze completely on me.

OK, that's not good. So I try various attempts at tar. So various tar commands, writing to a remote tape or filesystem. That would work, right? Wrong!

Each attempt freezes completely on me. That's local tar piped to tar that writes over nfs; tar piped to an rsh; tar on an nfs client; tar using rmt to write to a tape.

That's odd. Now, at least I can look at the output and see how far it's got. I'm starting to make headway, as it looks like it gets to the same point each time.

OK, so I start to build a copy by tarring up various bits of the filesystem, avoiding the place where I know it breaks. Until I get into the suspect area. And yes, it still fails in the same place (but at least I've got a copy of most of the filesystem now, so can breathe easier).

The bad area is in an area that has various versions (views of the data) of the index files of a proprietary search engine. Now it looks like tar always traverses the hierarchy in the same order. OK, so if I manually list the subdirectories in an order that puts the failed one last, I can copy off the files I'm missing, right?

That was a fine theory until it froze on me again. And this is where it gets really strange. Each subdirectory has the same structure. So in each subdirectory there's a bunch of files with different suffixes. And it always fails on one particular suffix. Furthermore, it fails at about the same distance (about 38 of 40 megabytes). That's about as weird as it gets, in my experience. What on earth is there about these files that causes anything that tries to back these files up to lock up completely?

And it gets worse. I can cp this file locally. Try cp to any nfs-mounted location and the cp wedges. An rcp to any remote system wedges. And it's the same again - it wedges at the same distance into the file.

It must be something in the data stream that's contained in these files. At least I did find one way to copy them across - if I gzip them first then they go across fine.

But where is the bug that's being tickled? Is this something in the network stack?

In my next job...

Now, I'm not looking for a job, as I'm very happy where I am, but if and when I do move on there's one hard question I'm going to ask any potential employer. And it's this:

How reliable is your air conditioning?

Because my servers were all sweating away in a sauna this morning when the A/C failed. Again.

We sorted it out and nothing went down as a result. And nothing has failed, yet. (I'm expecting a rash of disk failures over the next couple of weeks.)

I don't know what it is but wherever I've worked has had air conditioning problems. This should be mature technology, but seems ever so hard to get right.

Scary

Enough to send a shiver down my spine.

So, you download a current OpenSolaris.

Then, as root:

svcadm enable smb/client

And you can - as yourself - mount a windows share, from a server called windows.server in domain DOMAIN accessed as user user_name

mkdir /tmp/G
mount -F smbfs '//DOMAIN;user_name@windows.server/G' /tmp/G

and it just works

df -k /tmp/G
Filesystem            kbytes    used   avail capacity  Mounted on
//DOMAIN;user_name@windows.server/G
                    1563815260 421453348 1142361912    27%    /tmp/G

If you just want to see what shares are available:

smbutil view '//DOMAIN;user_name@windows.server'

T5240 - disks and more disks

One of the things I've complained about in the past is the lack of internal drives in Sun servers.

The X4150 was interesting, in that it had 8 drives in a 1U chassis.

So I'm impressed with the T5240. 16 drives in 2U. Wow! I like it.

Now why didn't the X4450 do that?

OGB: constitution

Work in the OpenSolaris project is done under a constitution, which lays out how the community is structured.

Sort of, anyway. The constitution covers both the Governing Board and Community Groups, but in fact most of the activity around OpenSolaris takes place in User Groups (where people get together) and Projects (where code is written), and also in Communities of Interest (where technologies get discussed). None of these last 3 parts of the organization are defined by the constitution.

Indeed, they are hardly mentioned. Even though community groups are tasked by the constitution to initiate and manage projects to achieve their objectives, it isn't clear (because projects aren't defined) whether the projects established by a Community Group are the same as the Projects hosted on OpenSolaris.org, or whether project is used in the generic sense.

As for user groups, they are managed as projects. While I believe this to be correct in the sense that they can reuse the machinery and infrastructure, they should explicitly be called User Groups and given their own place in OpenSolaris as first class citizens.

And then we have Communities. Originally, when OpenSolaris came about, a bunch of communities were created. And these were Communities of Interest - focussed around a general technical area (for example, performance), or a specific technology (dtrace). Then the constitution came along, and created the notion of Community Groups. As part of the bootstrapping process, many of the original Communities became Community Groups, while some did not. The whole thing is a total mess, and part of that mess is overloading the notion of a community to mean two different - and often incompatible - things. I think we need to find a way to clearly separate the mechanism of governance from the day to day interactions of users in the community.

And then there's the amount of effort wasted by the current structures. Creating a project means you have to persuade a community group to authorise it (and not all communities can, remember, and of those that can persuading the machinery to work at all and getting the approval can be a painful and time-wasting process). Creating a community involves redefining the governance hierarchy and invites considerable debate.

Where to go from here, then? I think we need to call out the existence and standing of projects and user groups in the constitution; we need to make the creation of those parts of the structure be lightweight and effortless, so that anyone can just do it; we need to have a structure whereby projects and user groups are monitored and helped (sponsored, if you like); we need to revert communities to be communities of interest (which can then be created as required with no effort) and build a new governance infrastructure that just gets out of the way and lets people work on bettering OpenSolaris as fast as they're able to without putting roadblocks in their way.

OGB: expectations

It's clear that the OpenSolaris community isn't in a particularly healthy state. While there's a lot of real work being done, the community is dogged by infighting - not to mention a distrust of Sun and its motives.

Part of this comes from a lack of clarity and focus. In particular, we're guilty of not even setting expectations, let alone meeting them.

Without clear expectations, we have a problem - everyone in the community makes up their own expectations of what they can achieve and what everyone else should be doing. All those expectations are going to be different, and everyone is going to end up frustrated and disappointed.

So one important thing the incoming OGB is going to have to do is to set everyone's expectations appropriately. People (and Sun) need to understand what can and can't be achieved - even if an accurate setting of expectations means they're quite low to start with.

OGB: Bio

Glynn asked the OGB nominees to provide brief affiliations and a Bio. I've done the affiliation already, so here's a brief history of me:

I come from Nottingham, England. I read physics at St. John's College, Oxford, and stayed on to do a D. Phil. in Theoretical Astrophysics in the Department of Theoretical Physics. Computing at this point was VAX VMS based - I managed to get my hands on a VAXstation 2000.

I moved over to Toronto, to the Canadian Institute for Theoretical Astrophysics. This is where I first used Unix in anger - on my own Sun 3/50. (Almost everyone had a 3/50, so I rewrote my code and ran it across a dozen 3/50s at once.)

Back to England, to the Institute of Astronomy in Cambridge. There was a gradual incursion of Sun hardware into predominantly VAX territory and, being a Unix guy, I ended up doing the sysadmin because there wasn't anyone else at the time who would. I also set up one of the earliest websites in the UK at the IOA.

With a family to support, and having found that I had some level of competence with this computer malarkey, I joined the Medical Research Council to work as a Systems Administrator at the Human Genome Mapping Project, which offered online services to academic researchers working on the Human Genome Project. We were Sun based, by and large, and this lasted almost 11 years before the grant renewal process failed and I was out of a job.

I then spent a year commuting down the the University of Hertfordshire. This wasn't a great success, not helped by the several hours a day it took to drive down there.

I'm now working as Senior Unix Systems Administrator for ProQuest, an online publisher in Cambridge. Not only is it a good place to work, it's less then 15 minutes from home.

Along the way I've used mostly Sun systems, with IBM, SGI, and DEC alpha, and Linux disturbing the peace from time to time. I remember the pain of the SunOS 4 to Solaris transition. I've been using Unix for almost 20 years and have never (a) used a shell without command line editing, and (b) used vi. (Except for long enough to exit them for something better, that is.)

I was part of the beta programs for Solaris 7, 8, 9, and 10. Including the various update releases, and we were a Solaris 10 platinum beta site. We managed to beta test some hardware along the way - including the B1600 blade system, the V250, and the V40z (tunred it on and I was deafened; ran one benchmark and I was a convert). And then I became involved in OpenSolaris, including being part of the pilot.

X2200M2 blues

Again. Sigh.

So the X2200M2 has an upgraded firmware that updates both the BIOS and the SP. Updating is a good thing, as there are a number of known problems with old version (open relay, amongst others).

For those updating, it's important to be aware of a few issues that you might run into.

When I tried this, I essentially lost all serial access to both the BIOS and the running instance of Solaris.

If you're running something older than the tools and drivers CD 1.3, then go to the 1.3 version first, and do the newer versions in a separate step. If you don't, you'll get a CMOS checksum error and will need to clear it. I found a physical power-cycle worked.

You might have to go into the BIOS and reset to optimized defaults.

Once you've updated to the current version, then you might have to go into the BIOS (under Advanced/Remote Access) and change the serial port from COM1 to COM2. Doh!

They default baud rate has changed back to 9600. You know all the customizations (including building a modified boot image on your jumpstart server) we've had to do to set the baud rate to 115200? Don't do those bits, they now cause more harm than good. (The other customizations are still necessary, it's just the baud rate.)

So after that I do get both BIOS access and can see Solaris booting up. I don't see grub coming up as it boots, though.

Nominated for OGB

I was slightly surprised - but highly gratified - to be nominated for membership of the OpenSolaris Governing Board.

I had to think about this, as being on the OGB clearly isn't a walk in the park. There's a lot of work to be done - for whoever gets elected this time around.

There are obvious mismatches between the existing constitution and the actual functioning (if that's the right word) of the community. As such, there are a few constitutional amendments already under discussion.

One of those amendments (554) is that candidates should disclose their interests. In accordance with this:

1. I'm a systems administrator employed by ProQuest in their Cambridge office. We use Sun and Solaris, so are a customer of Sun. My management are happy for me to accept the nomination, provided (as always) that my OpenSolaris work does not interfere with my professional responsibilities. As such, the views and opinions I bring are my own, and are not representative of my employer beyond the fact that I'm working in a context where I'm paid to use Solaris. I do not believe that a conflict of interest exists.

2. I've been a user (and beta tester) of Solaris for years, and have been a long-term participant in the OpenSolaris project. As a user rather than a developer I believe I would broaden out the OGB, and can make a useful contribution towards developing our fine community.

3. I'm a core contributor in the Systems Administration and Installation and Packaging Community groups. While not a programmer by trade, I have made some modest code contributions to OpenSolaris.

If only!

Tried this on a test server:

# psrinfo -v
Status of virtual processor 0 as of: 02/18/2008 16:37:34
  on-line since 01/21/2008 10:00:38.
  The sparcv9 processor operates at 2793 MHz,
        and has a sparcv9 floating point processor.

We can dream, can't we?

What's actually happening here is that I'm trying out the sparc emulator from Transitive (which now runs on Solaris x86) and it's reporting the speed of the Opteron processors in the box.

Python considered harmful

I used to think that using java was like being in league with memory suppliers.

However...

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 10946 fred       1  60    0 3692M  270M sleep   25:34  0.00% python
 19559 joe        1  59    0 1950M  836M sleep   17:21  0.00% python
 20738 bill       1  59    0 1787M 1608M sleep    3:28  0.00% python

That's on a machine with 4G of physical memory. And given that python is being used more widely, and that data volumes are increasing, I need to do a couple of things. First, order more memory; and second, work out how to build a 64-bit copy of python with all the modules we use.

All-in-one servers

One recurring theme as I build servers is that I often want a configuration that's not available.

Sun are the worst culprit as, while they generally have excellent products, the actual range of configuration options is rather limited.

And one of the reasons I was interested in the X4150 in the first place was the ability to have 8 internal disk drives. For many things I would prefer internal storage, if I can get it.

So what is wrong with external storage? Well, it can be very expensive, because you need to get a chassis, maybe raid controllers, and HBAs, not to mention the extra rack space, power cords, cables, and having separate boxes to manage and monitor. And if all you want is a few hundred gig of space, then it's just not worth it.

So consolidate on a SAN, you say. Maybe, but SAN storage itself is rather expensive. For small amounts, the cost of HBAs and the fibre infrastructure can be prohibitive.

I've been attracted to iSCSI, but while it does actually work great, it's limited to low-bandwidth light-use scenarios. (I just have regular gigabit ethernet.)

So a solution where the storage can fit neatly in the server is very attractive. At the moment I have something that takes about 500G, but is likely to grow slightly. So I think I need about a terabyte, and it's got to be reasonably quick, faster than iSCSI anyway.

So looking at the X4150 I would probably do something like use the first 2 drives for the OS and use the upmarket raid card to create a raid-5 array across the other 6 drives. So that's 5 drives worth of data, or about 700G.

Close, but it's not quite close enough. It's just a little bit tight. It would be nice to have larger drives, but as Ben has discovered, larger capacity drives in the small (2.5") form factor just can't be had.

So if we have to go beyond that then we might go up to regular 3.5" drives (which gets you 15K rpm and 300G or 450G capacities), or you need more than 8 internal drives. In either case that implies a larger chassis. (Sun have an X4450 which is the big brother of the X4150, but that's identical as far as supported drives are concerned.)

Looking at what Sun have available, there really isn't anything. And no I don't want a thumper, not for this application anyway. (It's a shame that there aren't more variations on the thumper theme.)

Ho hum. Off to see if I can find something different. A Dell 2900 is a lumpy tower. What about a HP ProLiant DL580 G5? (The DL 320s might work, but the max memory is a little too small at a mere 8G.)

Anyone care to suggest alternative options? (Must run Solaris!)

X4150 experiences.

It's clear from the comments I've had that I'm not the only one who's had issues with X4150s.

Those of us who have been playing this game for many years manage to surmount these minor obstacles, so we can take advantage of the good things that these servers have to offer. But it does worry me that customers without the experience (or confidence and contacts) to get past the irritating issues lose out on what could be a good solution, and that a supplier loses out on a potential sale.

X4150: lit up

So I've had the X4150 booted and running for about a day now. And it's managed getting on for 8 cpu-days worth of work so far.

It's looking good. On the workload we've tested so far, it seems to be about as fast as - if not marginally faster than - a comparable opteron, such as the X4200. Overall it can chew twice the workload because it has twice the cores.

It was a bit of a drag getting there, but I think it was worth the effort.

X4150: where do I boot from?

Seems my earlier optimism was a tad misplaced. I come back a short time later and the X4150 is sitting there at the interactive prompt you get when doing a network boot. So the install finished just fine but then it booted off the network again.

(See: I knew it was a good thing not to default to install. It could have installed itself over and over in a loop.)

The cause is reasonably obvious - either the drive isn't listed as bootable or the boot order in the BIOS is wrong.

This is where Ben's tip saved me either a bit of fiddling or a walk down to the machine itself.

I get into the BIOS, and the boot order shows me the DVD, the 4 network ports, and the disk I installed to. In that order. OK, I move the disk up and reboot.

Success! We boot off the hard disk.

(There's still an open question here: I want to have all four disks bootable just in case I lose one, and I'll set them up as two sets of mirrors - one for running, and one for live upgrade. I still need to work out how to add the rest.)

X4150: No disks found

So I've managed to get to the point where I can control my new X4150 using the SP, and can get to the system console.

I have the address set in dhcp and my jumpstart server configured, so let it boot up and see what happens.

(OK. So I goofed when typing the mac address into the dhcp server the first time. So it took me an extra attempt.)

Just as an aside, the X4150 is like the X2200M2 and requires the serial port set to 115200 baud, so I've rebuilt the boot image it gets from the jumpstart server just like it says in the documentation.

Off it goes, boots up, I hit 2 for jumpstart (I don't default this so that I don't accidentally overwrite a system if it boots off the network by mistake or for maintenance), and it's going well. What normally happens is that it complains the disk in my jumpstart profile isn't valid on my system.

Not this time. "No disks found."

That's not good. One nice thing about the X4150 is the 8 disk bays on the front, and I know that 4 of them are occupied. So why can't it see them?

I was shipped a HBA and a cable kit, but hadn't found any documentation on why I would need it or how to install it. So maybe you actually do need an additional HBA to make it work, which makes me wonder why on earth this requirement isn't prominently documented, and why they don't have a functioning on-board disk controller, and why the disk bays have been carefully cabled up to the on-board connectors when that's not going to work?

OK, so I pull out the old cables and put in the replacement ones. Looks like the HBA has to go in the middle slot, otherwise it fouls the memory slots. I hope I have the cables in right and routed the correct way.

Wonder of wonders, I boot up and all the disks are visible. Jumpstart tells me the disk isn't valid on this system, but that's a trivial fix to the jumpstart profile to get the controller numbering correct and off the installation goes.

Pretty quickly too!

LOM: Consistency would be nice

Most of Sun's x64 servers have some sort of LOM, but while they're generally superficially the same (they call themselves the SP, they have a similar /SP and /SYS layout), all the different models differ in the details.

And we all know that that's where the devil is....

Why on earth are the steps to set the IP address subtly different between systems, for example?

X4150: talk to me!

OK, so the first step is to connect up the cables, power the server on, and configure the SP.

So I do that - system and management networks, connect up the serial port, tip in, and apply power.

Silence. Not a peep. Not a sausage.

Come on, now. Talk to me!

Now I think this is OK, because the same tip session from the same host works flawlessly on every other Sun x64 server I have. But still I go through the motions - different host, different cable, various types of cable.

It's still sulking.

I give Sun Service a call. I don't think I'm doing anything wrong, so we'll see what they have to say - maybe I've got a faulty unit.

Turns out there's a problem with some units shipped with the wrong settings. If this happens to you, connect up a real keyboard and monitor, and power up the box (the real box, not just the standby power to the SP). Hit F2 to go into setup, and look through the settings. The "external serial port" should be set to SP. If it's set to system, hit F9 to restore defaults, and F10 to save (I think it's those function keys). I get back to my desk and there's the regular SP login prompt.

Thanks to James of Sun Support for tracking that one down for me.

Server deja vu

Over a year ago, I had a great deal of fun and games when we got some Sun X2100M2 systems:

On to the X2100 M2
Me versus the M2
The M2 comes alive
Server Wars: The M2 strikes back

Well, it looks like I have another battle on my hands. After a stack of flawless installs of X4200 and X4500 boxes, we decided it was worth getting an X4150 to see what the Xeon offered. Our cpu-hungry applications love the idea of quad-core; our data storage likes the idea of 8 disks in a 1U system.

Installing this thing isn't going well. Watch this space...

Christopher John Tribble

My father, Christopher John Tribble, passed away peacefully early this month.

He had an operation to remove a cancer tumor about a year ago, and things seemed to be going well until he reacted badly to chemotherapy towards the end of last year, which resulted in a downward spiral.

Watching him slip away was very hard, especially the awful feeling of impotence in the face of the terrible inevitability of it all.

It's not been the best start to 2008, but my father was a pragmatic chap who didn't want anyone to make a fuss, so it's now time to get back to normal.

I've lost, amongst other things, a friend.

Solaris on a Laptop

Every so often, things go without a hitch.

Today I got a laptop to look at. It's an old Dell D600, and the requirement was to have something looking like our production Unix environment on it. One of my colleagues had already stuck a Solaris DVD and got Solaris installed, and then handed it over.

First job was to get the network working. That's easy - off to the Broadcom download site to pick up the BCME driver. (Oddly, it's in the server section of the downloads.) That installed, I was on the wired network.

Next for wireless. A quick look and this machine seemed to have a Broadcom 4320 wireless chip. So off to the OpenSolaris Laptop Community, and it looks as though I need the ndis wrapper. With that and the windows driver I downloaded from Dell, I just followed the instructions. OK, ifconfig brings up the wireless.

That's a bit crude, and it would be nice to avoid too much CLI intervention. Enter wificonfig and Inetmenu and we're all set. Having given the laptop user the 'Ginetmenu' and 'Primary Administrator' profiles, it's as easy as firing up inetmenu and selecting from the list.

And both wireless and wired networks work (almost) flawlessly.

I didn't have a chance to look at wpa support, which would be needed some of the time. But I was mightily impressed by how easily everything I did try worked out.

Jumpstart Profile Builder

The latest version of solview includes something I've been working on for a while - the ability to construct a jumpstart profile.

The aim is to get a tool to do the hard work of resolving package dependencies, so that the resulting jumpstart profile is nicely self-consistent. Doing this in the past has been very much a case of trial and error. No more!

I'll just show one screenshot here, of adding evolution on top of the reduced networking metacluster:



There's a lot more work than can be done here. The interface, while workable, is rather crude. I would like to get it to construct a reasonable jumpstart profile based on an installed system, and it would be nice to be able to feed it all my old profiles and get it to fix them up for me. And I'm sure there are many other ways it could be improved, or used in different ways.

So if you have any suggestions, give me a shout!

Solview update

I've just released a new version of solview.

If you've tried to run solview in a non global zone, then you'll find that it didn't work. (Thanks to Tony Curtis for spotting this!) This was due to an ignorant and lazy programmer. Yup, I hadn't tested it in a non global zone for a while.

This latest update actually checks that the files it relies on to grab the information it's going to display exist, and behaves properly if they don't. I'm not saying that it works correctly in all circumstances, but it does work in the cases I've tested, whereas previous versions were rather more fragile.

JKstat update

It's been a bit quiet on the JKstat front recently (summer holidays may have had something to do with that).

I've done a little bit of cleanup I've been meaning to for a while - getting rid of fixed text strings and replacing them with resources, making localization possible.

I've also added a little chart feature - when charting a statistic, it's now possible to add other statistics from that kstat to the chart.

Nothing earth-shattering, but it still represents steady progress and so there's now a new version available for download.

X2200M2 fault diagnosis

I'm just setting up some new servers - Sun X2200M2 (running Solaris, of course).

One wasn't happy. Solaris would start to boot via PXE but the system would reset (over and over, in a loop). A quick look and the fault light was on. Not good.

These systems have remote management - ILOM - but after searching and looking through the documentation I couldn't actually see how to persuade it to tell me what was wrong. It's not helped by the fact that there are several ILOM variants in use, but the one on the X2200M2 is one of the more basic ones.

After running through all the options systematically, I stumbled across:

show /SP/AgentInfo/SEL

which tells me

    Nonrecoverable ,2007/08/31 17:22:00 ,CPU1 DIMM 3 has multi-bit error

Aha! I've reseated the DIMM after swapping it with its partner, and the installation is proceeding apace.

W2100z update

I've just updated the BIOS on my home W2100z (downloaded from here).

The old BIOS on the machine was antique. I mean, it predates all the old versions on the download page. And I remember updating it at least once (to get PXE boot working), so it wasn't the factory version.

The update was pretty smooth. However, on booting back into Solaris I see a CPU pegged and very high interrupt rates. Not good.

Now, it turns out that I've stumbled across this before. And in response to that, Alan Hargreaves got the thermal zone driver backported to Solaris 10. So it's available in patch 125107.

Unfortunately getting hold of this patch is proving troublesome because sunsolve is broken again.

OK, so I go have a cup of coffee and this time, success. Patch applied and just save this blog before I reboot.

Fixed funny fonts in gpdf

If you're running Solaris 10 on x86, you may have noticed that some PDF documents (particularly, and somewhat ironically, those generated by Sun) display as garbage. Instead of readable text you get rubbish - rather like this:



What you need is the recently released patch 119813-06, which fixes bug 6375381. (That's the original bug, which shows it got fixed in nevada ages ago - 2147903 is the one that tracks Solaris 10 integration.)

So now it looks like this - a vast improvement:

That's a busy disk...

From iostat -xnz 1 on my desktop:

    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    3.0    0.0   10.5 18444348816.8 18444348816.8 6148914691236.1 \
       6148914691236.1 1844434881679 1844434881679 c1d0

Sorry if it wraps too much.

Of course, those numbers look very suspicious. I have seen this sort of silliness several times in iostat output recently though.

Sunsolve needs a makeover

Sunsolve is an excellent resource. I don't have any issues with the content - it's great stuff.

Where I do have problems is with it's reliability and performance. I couldn't get through at all for a while over the weekend, and often have problems with my automated patch scripts - which can time out, report 500 errors, or tell me I'm unauthorized, all at random, and have been that way for a long time.

I even got the following gem off the website just now:


Come on Sun - show us how good your stuff really is by making your own sites work well!

Fingered!

I found this nugget on indiana-discuss to be quite apt:

one of the keys to identifying old-time Unix admins, particularly of the SunOS persuasion, is their use of tcsh

Yup. That's got my number!

Laid low :-(

I've been suffering from a fairly severe viral infection for the last week. I've had to take several days off work, and felt pretty atrocious. I'm hoping that it's starting to ease and I'll be feeling better in the next couple of days.

And solview too

Not content with updating jkstat, I've released a minor update to solview.

New jkstat version - real progress

I've let loose a new version of JKstat, also available from the jkstat project page.

This is version 0.19, and represents real progress. I'll start with some of the fiddly stuff: I've got rid of all the individual utility scripts (they were breeding out of control) and replaced them with a single jkstat script that takes subcommands. I've removed the native methods for looking up individual statistics, as there was no possibility of error checking (like to see if the value returned was valid). There's an client for nfsstat and the start of one to emulate the kstat(1M) perl client. I've got a swing widget that allows you to add a menu to any java client from which you can launch some of the demos.

The big feature is that I can now detect changes in the underlying kstat chain. I've added another class to handle this, so it's responsible for working out whether something has changed, and which kstats have been added and removed. This wasn't quite as big a job as I had feared.

Getting the demos to use this was somewhat more difficult. In particular, the browser took far more work than I ever expected. The kstats are in a tree, they're presented using a JTree, so you just update the tree, right. Wrong! I had originally placed the kstats into a Hashtable (it's a natural things to do), passed that to the JTree. Then Tom Erickson helped me out by creating a sorted Map, and a revised JTree that could take a Map. (And why some of the java swing components haven't moved into the 21st century and become aware of the Collections framework is beyond me.) I wasted a lot of time thinking I could update the Map and get the tree to refresh its view of the world, but it just didn't work. In the end I had to get rid of that and construct my own model, doing everything manually. Ugh.

It's not finished. Not all the demos handle updates. (Those using tables do - that was easy.) And the browser has a couple of cosmetic flaws: the newly added kstats aren't currently sorted correctly, and if you're watching a kstat that gets removed it doesn't tell you that anything has happened. But it's a major step forward and I'm going to take a deep breath and use this for a while before diving back in.

Solaris metacluster sizes

As part of the Indiana discussion, I was looking at the installed sizes of various Solaris clusters of packages. The idea is to work out where all the space is going so as to be able to trim an OpenSolaris distribution onto a single CD. Depending on compression, this might allow something just over 1G of software.

A minor extension let me do the same for the various metaclusters you can choose from in the Solaris installer. And the sizes of those on a Solaris 10 update 3 x86 system are like this:

157.945M : SUNWCmreq
168.594M : SUNWCrnet
195.372M : SUNWCreq
2010.12M : SUNWCuser
2579.84M : SUNWCprog
2614.19M : SUNWCall
2614.2M : SUNWCXall

This confirms what we've known all along, that the small clusters are tiny, the other install clusters are huge, and there's a great big gap between the two extremes. More than that, the end user cluster misses out on a lot of stuff that would be really valuable to an individual user.

(I would recommend to anyone who wants to customize the packages when installing a Solaris system - either using jumpstart or interactively - that they start with either SUNWCrnet or SUNWCall, as the others don't help you much at all.)

The snag with building a CD sized distribution is that the target size is right in the middle of the gaping chasm between a fully functional (SUNWCprog) and minimized (SUNWCrnet) layout.

Fortunately, things aren't quite as bad as they look. These numbers are for a regular Solaris system. This includes a fair amount of software that isn't freely redistributable, so all the numbers shrink. Sometimes by a lot, as if you start taking StarOffice or java out then you save significant amounts of diskspace.

solview does patches

I updated solview so it now shows patches. It gets most of this for free, by looking at the installed packages. So you get the standard stuff from showrev -p, with the packages having a short description, plus it will display the readme file for the patch if it can find it. Next step is to parse the patchdiag.xref file to work out what needs updating, like pca does.

Solview, JPack

Last week I added contents file parsing to solview.

More recently, Michal Pryc announced that he was working on JPack, an easy software manager and installer written in Java.

Now, the current focus of these tools is slightly different - solview is about analysing what your system has, JPack aims to make adding new software painless.

I was planning to add package management to solview at some point, but probably won't bother now - there seems little point in duplicating the work in JPack so I could just reuse that, and hopefully I'll be able to contribute to JPack as well.

Motivated by the rather better look of the GUI in JPack, I had a little go at reorganising the layout in solview so that the package list is done as a table. It does look better. I also did some refactoring of the code - both to make it more usable by other projects (such as JPack), and to remind myself how it all worked.

So there's a new version of solview now available, with the above mentioned refactoring and layout changes, and with the slow operations done in the background with a SwingWorker.

solview - includes contents

A little bit of work this weekend and I've let out a new version of solview.

No sign of the promised jumpstart profile builder, though.

But what is present now is a contents file parser. This allows me to add a summary of the number of files and space used to each package (and to each cluster). For example:



And in addition I added a filesystem browser so you can choose a file and see what packages and clusters it's included in:



There's a slight amount of lunacy involved in parsing the contents file. Don't be surprised to see OutOfMemory errors, as the memory usage is fairly high (several hundred meg, commonly).

Phoronix and Solaris

Interesting: Phoronix now supports Solaris. See the article.

Not only an endorsement of the growth of interest in Solaris, but hopefully another source of information regarding hardware compatibility for Solaris.

JKstat 0.17 - includes charts

I've been gradually continuing work on JKstat. The latest version (0.17) has a number of code cleanups, but also includes charting capability. (Uses JFreeChart.)

You can right-click on any numerical statistic in kstatbrowser and chart that statistic. Here's an example:

JKstat 0.15

I've made a new version of JKstat available.

There are quite a few changes underneath. I'm very grateful to Tom Erickson for sending me a bunch of enhancements and fixes. Some of them made it into this version fairly directly - particularly the sorting of kstats in the browser.

This version has also been revamped and now requires Java 5. I had been thinking about modernizing the code for a while, but had always put it off. Tom again provided the impetus, as he went ahead and used generics in the fixed version he sent back to me. I reimplemented it myself, so I could actually understand what I was doing to my code.

I actually went and rewrote solview first. This was a useful exercise in itself, as one of the advantages of using generics is that it makes you think harder about the way you define classes and how your code is structured. In solview I was in the middle of cleaning the code up to define Solaris packages and install clusters as proper objects, and using generics made sure I actually got that right.

JKstat was also accepted as an OpenSolaris project, so I'm going to be moving development over there soon.

OpenSolaris priorities

Before the OGB election there was the OpenSolaris Comunity Priorities vote.

Looking at the result, and the clear winner in terms of priorities is the public defect management system. Yep, I put that at number 1 too. (I think...)

The OpenSolaris Bugs interface did get revamped recently, and that was a significant improvement. However, it's still not really doing anything for OpenSolaris - it's just a way to give a limited view of a subset of Sun's bug database.

Importantly, there's no other integration with anything else the community might be doing, and no way for a community member to interact with the system. How can it be improved?

Here are a couple of things I would like to see:

Eliminate all the crud. The database is littered with all sorts of things that aren't bugs, and still contains bugs that are antique. We need to take some of the open bugs forward, but most of the database has no relevance to the current codebase.

I think this means we need a completely separate database. And bugs that Sun generate through their support organization need to be put into the public OpenSolaris bug database the same way that Sun currently logs bugs against external projects such as Gnome.

We need to allow community members to participate in the triage process, and allow comments and followups to further refine the bugs and provide sample solutions, test cases, and prioritization. This extra interaction will build a stronger community, and provide ways for more people to get involved.

And the database needs to be linked into the whole workflow, so that there's a way to see who might be working on a particular bug and contact them.

The bug database is the crucial missing link. We want to improve and develop OpenSolaris, and to do that we first need to know what's broken and what's missing.

Polls and voter apathy

I've been sitting on the fringes of the OpenSolaris Governing Board Election. I would have liked to have been more involved, but have been struggling to find time. Strike one for apathy!

(And, while I did consider putting myself forward as a candidate, the fact that I haven't been able to effectively contribute up to this stage also implies that I wouldn't be able to do the job properly in the unlikely event that I were to be elected.)

There does seem to be a fairly low turnout so far. There have been a number of calls for people to get their finger out, and we seem to be creeping closer to a quorum.

Alan Coopersmith had a look at the contributors and communities. This shows what we've known all along - that the different communities have very different levels of participation in the governance process. Looking at the way that core contributors are distributed, it's not clear to me now that relying on communities to generate the initial list was - or is - a good thing.

There seems to be a view that reorganising the communities is going to be a big task for the incoming OGB. (If one gets elected, that is.) I'm not so sure about this. I feel that we're trying to build too much officialdom into the community structure, when in reality the communities are more social structures connected by a common interest. But more on that later.

I'm fortunate that I am classed as a core contributor and, yes, I've voted.

Perhaps core contributor status should be automatically be rescinded from those who haven't bothered to vote?

Spring Clean and Upgrade

I've been doing a bit of a spring clean and upgrade on some of my home systems.

The family PC had its memory doubled. It's survived 3 years with only 512M, which ought to be enough but it's obviously running out. We use fast user switching so there are usually 4 people logged in. That shouldn't be so bad, but it implies 4 web browsers and they guzzle memory like it's going out of fashion. I also replaced the broken DVD-ROM with a working dvd writer.

I think it's the first time the case has been opened for 3 years. And it was filthy inside, so it got a thorough vacuuming.

Next step was a couple of Sun workstations. I use one for OpenSolaris development, and I shuffled that up from build 55 to build 59. I also took a gamble on rebuilding one of the children's systems from an essentially unpatched FCS version of Solaris 10 to the latest Solaris Express release - the same build 59. Only complaint so far is the placement of the tab close buttons in Firefox 2 versus Mozilla.

Next step - run Nexenta using the VMware player.

A new use for a D1000 array

I found a new use for a D1000 array last night.

We bought a pallet-load off eBay a while back. We only actually needed one, but the whole pallet was no more expensive. So the first one got used for the purpose for which it was bought, and I've used a couple more filled up with old drives to test ZFS and the like.

We just piled the rest up in a store-room.

Which is where they had stayed for a few weeks harmlessly gathering dust.

Last night I went in there to look for some cables, and was somewhat annoyed to find that the door wouldn't open when I tried to get out. The lock release did absolutely nothing.

What you have to understand is that this was an old comms or server room. So that's a security and fire door in my way. No windows. No phone. My mobile is on my desk. It's almost soundproof as well, and it's getting late.

I did manage to attract someone's attention as they were passing in the corridor, and we tried all sorts of tricks to get the stupid door to open, with no success.

Eventually I piled all the D1000s up and stood on top of them. (Which just goes to show that buying quality kit is always worthwhile. You try this trick with cheap kit and it will fold under the weight.) Originally I was hoping to get into the false ceiling and work my way through, but even that was a bit of a stretch and very cluttered.

Eventually we took out the panel above the door and I got out that way.

See, I knew this old kit would come in useful some day!

Updated jkstat

It's been a while (almost a year) since I last updated jkstat, but I've just put version 0.13 up for download.

For those that don't know, jkstat is a Java jni interface to Solaris kstats, allowing you to access a wide range of useful data held by the Solaris kernel from a Java application.

One thing I have done is this version is try to improve the look of the sample applications, most of which has been achieved by replacing the rather clumsy GridLayout with SpringLayout.

Impressed with Xfce

I've been having a play with Xfce on one of my Solaris workstations.

And I have to say I've been very impressed. It's reasonably fast and light. It has an increasing number of applications. The file manager - Thunar - is quite nice.

There are two things that I think are in favour of Xfce. The first is that because of its common roots with gnome it can leverage all the gnome applications and GTK themes. The second is that it is very polished. There is a good section of window manager themes, and the artwork is very well done.

All in all it's well worth a look.

Sun, Intel - friends?

OK, so Sun and Intel made a big fuss about some deal or other (one which doesn't really set the pulse racing, mind). What I found ironic was the ticker underneath the main feature on the Sun home page, where they put the knife into Itanium.

I remember Sun recently selling a number of Intel powered systems. The LX50 was a nice box - even if it was pushed with SunLinux. The V60x was OK, but I quite liked the V65x, and that sort of system (6 internal drives) is something that would be quite valuable. And the B200x blade was quite a marvel - twin hyperthreaded Xeons and a decent memory configuration in a tiny package.

I expect closer collaboration will improve driver support for Solaris, but I'm also hoping for a wider range of systems designed for Solaris to give us a breadth and variety of system configurations that Sun themselves seem disinclined to supply.

Backup Blues

Taking system backups is an essential part of any IT operation, but seems to involve an awful lot of pain.

Over the years I've generally been happy with Legato (now EMC) NetWorker, sometimes known as Sun StorEdge Backup or something similar. What I really like about it is that it's very low maintenance - easy to set up and configure, and gets on with its job quietly in the background without any fuss.

I was a little concerned that the latest version has dropped the old nwadmin X-windows based GUI in favour of some Java client-server thingummy. I still don't really like the java interface, but as I previously mentioned it's pretty low maintenance so you don't need to spend much time using it, and it's generally easy enough to use. The one downside to it is that the licensing is a bit more complicated, and the license registration instructions explicitly refer to the old GUI.

I'm currently using NetWorker for one half of my backups. I put in a new server and tape library and moved the NetWorker license across. The transition has been very successful, although the primary win is that I replaced an L20 with 2 DLT-7000 drives with a C4 with 2 LTO3 drives. The old system simply didn't have the capacity or the performance - backups would take days, and some backup runs came in at over twice the capacity of the library. Now they take hours and fit onto a couple of tapes.

The other half of my backup story is more appropriate for the tile of this post. There, we're using an alternative product. OK, specifically, NetBackup. And it's much more trouble than NetWorker has ever been.

I'm trying hard to give a balanced view here. After all, NetBackup is widely used and must have some good points, right? Maybe...

The first thing is that NetBackup is very high maintenance, at least compared to NetWorker. It needs constant attention paid to it, seeming to go off and hide in a corner if you don't give it constant encouragement. I'm not surprised that large organizations have dedicated NetBackup administrators, or even teams thereof.

The second thing is that I (and I know I'm not the only one) have trouble with the way you actually define backup schedules. In NetWorker it's trivially easy - you define what you want backed up, what level of backups you want, and when to do it. In NetBackup you mess around with backup windows, frequencies, policies, and schedules. Again, this is much higher maintenance - I usually need to manually configure half a dozen screens in NetBackup as opposed to simply selecting one option under NetWorker.

The third thing is reliability. The only time I have known NetWorker fail to do a backup is if I've run out of tape. NetBackup seems to throw fits all over the place. We see regular timeouts, errors when it's outside the backup window, and occasional (once or twice a month) where the whole thing freezes up and needs a forceful restart. I've just upgraded the NetBackup server itself, and after tuning got acceptable performance (the default performance was dreadful), but the catalog backup takes over a day for something that ought to take 20 minutes, and renders the machine completely unresponsive while it's at it.

I'm tempted to replace the NetBackup installation with NetWorker, which raises a number of issues. I'm happy that it will be fairly painless on the Unix side, but I'm less sure about the Windows side which also needs backing up. While I have run NetWorker on Windows, it became apparent fairly quickly that it was basically a Unix product and looked to be in alien territory. And my experience of NetWorker's application modules (as needed for the likes of databases, exchange and the like) hasn't been entirely positive. So I'm thinking of a completely separate system for the Winddows backups.

I've tested a number of other backup products, but under Unix. And there it was obvious that most of the Windows products looked completely lost under Unix. I haven't really come across a truly cross-platform backup solution, and it's not obvious to me that such a thing actually exists.

Given how conceptually simple backups are, is it asking too much to have solutions that just work?

Fun with NVIDIA driver

I have a Sun W2100z running Solaris. It's a fantastic machine, but the graphics has never been too hot. It's got the entry level graphics card - an NVS 280 if I remember correctly - and I've been using the bundled nv driver as I wasn't aware that the accelerated driver could do much better.

I was trying to get DVD playback to work, and gxine seems to work (after I had built and installed libdvdcss), but was rather slow and I noticed that Xorg seemed to be pegging a whole cpu. In addition, it might be neat to get Looking Glass to work.

So I ambled over to NVIDIA and their Solaris Driver downloads, and noticed that my graphics card was listed as one of those supported. I downloaded and installed the latest version (9764), and that didn't work. I get an error message as it boots that seems to be from the nvidia driver, but then the system immediately reboots. Not good. (I went into failsafe from the grub menu and used pkgrm to delete the two packages to recover.)

Then I had a look in the forums and noticed a message about a legacy release (9631). Now, it doesn't really explain what the legacy release is in this case (the information on the download page about legacy releases applies to Linux), and there isn't a link to the legacy release on the download page itself, but to cut a long story short - the 9631 legacy release works fine on my entry level box. DVD playback is a bit smoother, although still not brilliant, but the cpu load is pretty negligible compared to before.

If you, like me, have an entry-level W2100z with an NVS280 graphics card, then you may be advised to use the NVIDIA legacy driver rather than the latest and greatest.

Jumpstart Profile Builder

One of my little projects is to write an interactive jumpstart profile builder, so that you can play with Solaris software selections, and construct a working jumpstart profile.

The problem currently is that you can, with some effort, create a jumpstart profile. But the only real way to test it is to use it to install a system, and either see what works and what doesn't, or look at the installation log to see what package dependencies you messed up on. Eventually you can get quite good at this, but it shouldn't be this hard.

You can select packages in the interactive install, but the dependency resolution is awful. It tells you what the dependencies that failed are, but doesn't give you a way to feed that back into the gui to resolve them. Nor does it tell you what dependencies are required as you look at the individual packages. And once you've selected your software, it will install it but you can't go from that to create a jumpstart profile to automatically install the next set of systems.

I have most of the base code already, in the shape of Solview. That knows how to turn the .clustertoc file (which is where the clusters and metaclusters are defined) into a tree representation of the packages, that could be used as the basis of a selection tree. And it can evaluate the dependencies. I've also got code that can describe a jumpstart profile.

Where I'm currently struggling a little is the actual interface. Using a java swing JTree as the base is fairly straightforward - this is what Solview does already. And I need a custom cell renderer to add checkboxes to show whether the software package is selected or not (or, in the case of an installed system, whether it's been installed or not). But then I got to thinking and I can't use the regular JCheckBox, as there are more states that I wish to display than JCheckBox gives me out of the box.

• Package or cluster selected
• Package or cluster not selected
• Cluster partially selected
• Package or cluster selected and cannot be deselected

The last case is for required packages and clusters (which you can actually delete in a jumpstart profile but which it ignores), and I could probably do simply enough by having the checkbox selected but disabled. I might also want to have a separate state for packages and clusters which have forward or reverse dependencies, so it's clear visually whether it's safe to select or deselect them.

I've had a quick look around with the help of google, and I'm obviously not the first person to have requirements of this type. And there is a fairly large amount of sample code out there. Unfortunately it appears to be fairly complex, and I would be adding a significant amount of extra code, but it doesn't appear that I have much choice.

T2000 vs bison

This wasn't a benchmark, or even meant as a test, but it was an interesting number nonetheless.

I installed bison today. As I have 3 different families of sytems (Sparc Solaris 8; Sparc Solaris 10; x86 Solaris 10) I have 3 copies of /usr/local, so build on 3 different systems. Anyway, after building bison (and the T2000 allows a very parallel make that some builds can take advantage of) I ran a make check.

The results were quite interesting. My sub $1000 Opteron 146 system took 107s. An old V880 (750MHz US-III) took 529s. The T2000 took a whopping 794s. This really isn't looking good.