poking around with dtrace

Having released jfsstat, I've been using it on my test system just to see what would come up.

I left it running looking at file accesses - lookup, access, pathconf, readlink - and started to notice some odd patterns.

The first was that /proc was getting a very steady 25 lookups and 12.5 accesses per second. OK, that's easy - I had top running (refreshing every 5 seconds) and this system has 60 processes on it, so every refresh top is doing 1 access and 2 lookup per process.

The next was that / (and this machine just has the single filesystem) was seeing about 600 lookup and 10 access requests every 10 seconds.

The third part of the pattern was that roughly every minute there was a veritable storm of operations on the / filesystem.

Now, one of the longer term aims of my writing tools like jfsstat and jkstat is that you see some strange activity, and you can click on the activity there and then and drill down deeper - probably with dtrace. I'm not there yet, but I thought I would practice my dtrace skills and do it by hand.

OK, so what I'm after is ufs lookups. What can dtrace give me?

% dtrace -l | grep ufs| grep lookup
  536     vtrace               ufs                        ufs_lookup TR_UFS_LOOKUP_END
  537     vtrace               ufs                        ufs_lookup TR_UFS_LOOKUP_START
17836        fbt               ufs                        ufs_lookup entry
17837        fbt               ufs                        ufs_lookup return

Right. Let's have a look at what's generating those lookup requests:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

  java                                                            612

So, that 10 second repeating burst of 600 lookups is java. In fact, I know this to be tomcat.

Now I run the script again to catch the massive burst of activity that happens every minute:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

  sched                                                            24
  pwd                                                              28
  sh                                                               75
  fping                                                            91
  server.sh                                                       112
  uname                                                           150
  mysqld                                                          162
  ping.sh                                                         238
  nscd                                                            491
  cron                                                            498
  init                                                            988
  load.sh                                                        1014
  rup                                                            1539
  java                                                           1836
  perl                                                           2120
  awk                                                            2394
  mysql                                                         63666

I've hit the tomcat activity burst 3 times, but the once a minute is coming from something launched by cron - a system monitoring script that runs fping and rup and pokes the results back into a mysql database. But what on earth is mysql doing making 63666 lookup requests?

(The first question I asked was - is this one instance of mysql or many? If I aggregate on pid as well as execname then I see that I'm running a lot of copies of mysql, each of which generates 786 lookup requests on the filesystem.)

Next question: what are the pathnames that are being used in the lookup request? To get this, I need to understand the ufs_lookup call itself in a little more detail. So the source tells us that the 4th argument is a struct pathname, and the string I'm after is the pn_buf member. So let's see what pathnames are being looked up. First the little tomcat burst:

dtrace -n 'fbt:ufs:ufs_lookup:entry{@[stringof(args[3]->pn_buf)]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

  /opt/XSload/tomcat/conf/Catalina/localhost                        6
  /opt/XSload/tomcat/webapps/ROOT/META-INF/context.xml                6
  /opt/XSload/tomcat/webapps/jsp-examples/META-INF/context.xml                6
...
  /opt/XSload/tomcat/webapps/xsload/WEB-INF                        18
  /opt/XSload/tomcat/webapps/xsload.war                            20
  /opt/XSload/tomcat/conf/Catalina/localhost/host-manager.xml               35
  /opt/XSload/tomcat/conf/Catalina/localhost/manager.xml               35
  /opt/XSload/tomcat/webapps/balancer/META-INF/context.xml               35
  /opt/XSload/tomcat/conf/context.xml                              90

Pretty clear, really - every 10 seconds tomcat goes round checking to see if you've gone and modified anything.

Now for the mysql burst. What are the pathnames here? There's quite a lot of output, so I've trimmed it a bit:

...
  /usr/ccs/lib/libc.so.1                                          432
  /usr/ccs/lib/libcrypt_i.so.1                                    432
  /usr/ccs/lib/libcurses.so.1                                     432
  /usr/ccs/lib/libgen.so.1                                        432
  /usr/ccs/lib/libm.so.1                                          432
  /usr/ccs/lib/libnsl.so.1                                        432
  /usr/ccs/lib/librt.so.1                                         432
  /usr/ccs/lib/libsocket.so.1                                     432
  /usr/ccs/lib/libthread.so.1                                     432
  /usr/ccs/lib/libw.so.1                                          432
  /usr/ccs/lib/libz.so.1                                          432
  /lib/ld.so.1                                                    504
  /opt/SUNWspro/lib/rw7/libCrun.so.1                              540
  /opt/SUNWspro/lib/rw7/libCstd.so.1                              540
  /opt/SUNWspro/lib/rw7/libc.so.1                                 540
  /opt/SUNWspro/lib/rw7/libcrypt_i.so.1                           540
  /opt/SUNWspro/lib/rw7/libcurses.so.1                            540
  /opt/SUNWspro/lib/rw7/libgen.so.1                               540
  /opt/SUNWspro/lib/rw7/libm.so.1                                 540
  /opt/SUNWspro/lib/rw7/libnsl.so.1                               540
  /opt/SUNWspro/lib/rw7/librt.so.1                                540
  /opt/SUNWspro/lib/rw7/libsocket.so.1                            540
  /opt/SUNWspro/lib/rw7/libthread.so.1                            540
  /opt/SUNWspro/lib/rw7/libw.so.1                                 540
  /opt/SUNWspro/lib/rw7/libz.so.1                                 540
  /opt/SUNWspro/lib/v8/libCrun.so.1                               540
  /opt/SUNWspro/lib/v8/libCstd.so.1                               540
  /usr/local/mysql/data/my.cnf                                    540
...
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis/libCstd_isa.so.1              756
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis2/libCstd_isa.so.1              756
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis/libCstd_isa.so.1              756
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis2/libCstd_isa.so.1              756
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv9/libCstd_isa.so.1              756
...
  /opt/SUNWspro/lib/libthread.so.1                                864
  /opt/SUNWspro/lib/libw.so.1                                     864
  /opt/SUNWspro/lib/libz.so.1                                     864
  /lib/libm.so.2                                                  888
  /usr/lib/libc.so.1                                              972
  /usr/lib/libcrypt_i.so.1                                        972
...
  /opt/SUNWspro/lib/v8/libgen.so.1                               1080
  /opt/SUNWspro/lib/v8/libm.so.1                                 1080
  /opt/SUNWspro/lib/v8/libnsl.so.1                               1080
  /opt/SUNWspro/lib/v8/librt.so.1                                1080
  /opt/SUNWspro/lib/v8/libsocket.so.1                            1080
  /opt/SUNWspro/lib/v8/libthread.so.1                            1080
...
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/../../libCrun.so.1             2160
  /opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/libCstd_isa.so.1             2592
  /platform/SUNW,Sun-Blade-1000/lib/../../sun4u-us3/lib/libc_psr.so.1             3360
  /platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1                4480

What on earth?

So, most of these lookup operations are the mysql binary looking for shared libraries when it starts. And in some less than obvious places too! So why is this? I go and have a look at the binary:

% dump -Lv mysql
[INDEX] Tag         Value
[1]     NEEDED          libcurses.so.1
[2]     NEEDED          libz.so.1
[3]     NEEDED          librt.so.1
[4]     NEEDED          libcrypt_i.so.1
[5]     NEEDED          libgen.so.1
[6]     NEEDED          libsocket.so.1
[7]     NEEDED          libnsl.so.1
[8]     NEEDED          libm.so.1
[9]     NEEDED          libCstd.so.1
[10]    NEEDED          libCrun.so.1
[11]    NEEDED          libw.so.1
[12]    NEEDED          libthread.so.1
[13]    NEEDED          libc.so.1
[16]    RUNPATH         /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib
[17]    RPATH           /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib

The list of libraries matches what we see being looked for, so that makes sense. The problem is that the compiled in library search path contains places that it shouldn't (and some repeated), so it needlessly searches those locations (and multiple times at that) when mysql starts up.

So, problem solved. And, as I said earlier, the idea is that you be able to see this anomalous activity in one of the jkstat tools and click on it to drill down into the system to see what's going on, so that all the dtrace is done automatically on the fly for you.

It won't be that easy, of course. I'm relying on the fbt provider, so that I need to pretty well write the dtrace script by hand for each function I wish to investigate. (There isn't even a consistent naming or calling scheme - you can't just replace ufs by procfs and expect it to work.) But fortunately we have the OpenSolaris source to look at to see what's actually going on underneath the covers.

jfsstat

I've just updated jkstat.

The notable new feature in this version is the jfsstat utility:



This uses the new kstats introduced in Nevada recently - the kstats used by fsstat. So you need to be running Nevada build 35 or newer to see these. (ON downloads - you want SXCR build 35 or later.)

The other change that's part of this is that I've started to use JTable to display things. I don't know why I've avoided this in the past - I guess that JTable seemed rather complex, but once I had got the hang of it it turns out to be very easy. I'm also using TableSorter from the Java Swing tutorial, which gives me the ability to sort the columns for free.

Enjoy!

Home Networked

I'm writing this blog entry using my W2100z workstation upstairs in the spare room.

We've spent the last couple of weekends decorating Amanda's room, and I took the opportunity to run some cat-5 cable in the upstairs rooms. The next step was to run cable down the stairs to connect my little ethernet switch to the broadband router, and enable dhcp. Hey presto! It all works.

The girls each have a SunBlade 150. Well, I've got them, and it means I don't have to worry about them getting virus infestations. So a surfing we can go! (And it means that the main computer is free instead of being taken up by someone trying to do their homework half the evening.)

I used an all-in-one kit I got from Maplins. 50m of cable, connectors, boxes, faceplates, tools, and 4 patch leads. A great buy and a real bargain.

T2000 problem

I just tried to buy a pair of Sun's T2000 machines, and then discovered (very late) that the system specification doesn't match my requirements.

It's not the system itself that's the problem, but the peripheral connectivity - or rather lack of such.

In particular, the plan was to hook up a couple of 3120 SCSI arrays. Not everything wants a fancy raid array, and FC arrays (and associated HBAs) are pretty pricy anyway. In this case, I'm looking at raw spindles for database access, but also if you look at ZFS it wants plain drives - raid hardware just gets in the way. So the plan was to have 2 SCSI channels and mirror them.

This won't work in a T2000. There's only 1 free PCI-X slot (the internal SAS controller takes the other one), and the only supported SCSI HBA is single channel anyway. So at the present time you simply can't have more than a single SCSI chain on a T2000.

To say that this is annoying is an understatement. It also limits the usefulness of the T2000 for several other projects I have in the pipeline.

Now, I could use FC storage, because the T2000 does have PCI-E slots, and there are PCI-E fibre HBAs, so it would work, but you're looking at a 50% or so increase in cost, which I regard as unacceptable. (The cost differential is particularly bad in small configurations - as you push up it becomes much less pronounced.)

Ho hum. Time to construct a plan B.

OpenSolaris Installation and Packaging

It's here! Dave Miner made the announcement that the opensolaris Installation and Packaging community, and the SVR4 packaging project are now up and running.

This is important stuff. Being able to install and manage software is a critical component of an operating system, and is something I do a lot of.

And the process - particularly the performance (or lack thereof) of the tools - could stand some improvement. Yes, they work, and they're solid, but with a bit of extra work they could be made better.

fsstat in latest OpenSolaris

I've updated my test machine to the latest OpenSolaris build.

I used the bfu archives this time. Normally I wait for it to make it into Solaris Express (SXCR) and jumpstart, but this release has two feature I wanted to investigate, and I want to try them out as soon as I can.

The two features in question are the Java binding to DTrace, and the new fsstat utility and associated kstat support. More on DTrace once I get jkstat updated to integrate with it, but I wanted to mention a little more about fsstat.

Now, I'm just a user, but I've played with fsstat a little and looked at the code, so I've got a little bit of understanding of what it's doing.

On its own, fsstat will give filesystem activity for the various filesystem types:

 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
1.22K   606    63 1.99M   569  10.2M 37.8K  993K  502M  216K  152M ufs
    0     0     0 7.73K     0  14.8K   722 8.65K 3.18M   290 6.91K proc
    0     0     0     1     0     10     0     0     0     0     0 nfs
    0     0     0     0     0      0     0     0     0     0     0 zfs
    0     0     0 87.1K     0      0     0     0     0     0     0 lofs
14.6K 2.95K 11.5K 30.8K    37  43.5K    26 32.9K 32.9M 45.9K 34.5M tmpfs
    0     0     0 3.05K     0      0     0    32 5.25K     0     0 mntfs
    0     0     0     0     0      0     0     0     0     0     0 nfs3
    0     0     0     0     0      0     0     0     0     0     0 nfs4
    0     0     0     6     0      0     0     0     0     0     0 autofs

If you give it a filesystem as an argument, it will show that:

# fsstat /tmp /var/run /etc/svc/volatile
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
  251   233     2 1.10K    21    719    26    29 34.9K    60 36.7K /tmp
   20     0     0    44     8  16.8K     0     0     0     6    67 /var/run
14.5K 2.72K 11.6K 29.8K     8  26.4K     0 32.9K 32.9M 46.0K 34.5M /etc/svc/volatile

and it takes the normal interval and count arguments.

These numbers are all backed by kstats. For example:

# kstat unix:0:vopstats_ufs
module: unix                            instance: 0
name:   vopstats_ufs                    class:    misc
        crtime                          76.4978722
        naccess                         559104
        naddmap                         1280864
        nclose                          514400
        ncmp                            5179654
        ncreate                         1222
        ndelmap                         1276280
        ndispose                        1788458
        ndump                           0
        ndumpctl                        0
        nfid                            0
        nfrlock                         6953
        nfsync                          1641
        ngetattr                        1550291
        ngetpage                        5904478
        ngetsecattr                     2882
        ninactive                       30404
        nioctl                          114402
        nlink                           18
        nlookup                         10861279
        nmap                            1024824
        nmkdir                          17
        nopen                           479509
        npageio                         14
        npathconf                       70555
        npoll                           477742
        nputpage                        39301
        nread                           1031796
        nreaddir                        38829
        nreadlink                       785043
        nrealvp                         65352
        nremove                         602
        nrename                         29
        nrmdir                          4
        nrwlock                         1292387
        nrwunlock                       1293855
        nseek                           213405
        nsetattr                        531
        nsetfl                          179084
        nsetsecattr                     26
        nshrlock                        0
        nspace                          12
        nsymlink                        16
        nvnevent                        0
        nwrite                          223174
        read_bytes                      530866166
        readdir_bytes                   18340192
        snaptime                        9588.8833902
        write_bytes                     159609739

Now, that's a lot of data about what's going on.

The kstat names aren't spectacularly obvious. They all start with unix:0:vopstats_ with the unique identifier tagged onto the end. This can either be a filesystem type (ufs, lofs, nfs, zfs are some examples) or the id of a filesystem mountpoint. So in /etc/mnttab you might see

/dev/dsk/c0t1d0s0 / ufs  rw,intr,...,dev=1d80008

(I've trimmed this for brevity). So the kstat you need to query for statistics on the root filesystem in this case is called unix:0:vopstats_1d80008.

And that's pretty much it. Now to get some more tools to make sense of all this new information!

Vanished...

After a nice weekend away I'm wondering where my machine at work has got to.

It was fine when I left it on Thursday, but sometime Friday afternoon it vanished without trace. Can't log in. Can't ping it. No nothing.

I can get into my other machine (that I use for OpenSolaris testing) just fine. That's connected into the same switch, powered off the same extension lead. So it's not a network or power problem.

Oh well, I guess I'll find out what hapened to it when I get in to the office tomorrow.

Affordable Sun Gear

Over on OpenSolaris.org, I jumped into a discussion regarding the availability - or more accurately lack of - a cheap sparc desktop machine.

At the present time, Sun sell a number of desktop sparc machines: SunBlade 150, SunBlade 1500, SunBlade 2500, and the new Ultra 45.

Frankly, why they're still selling the 150 is beyond me - it wasn't exactly quick when it was introduced in 2002 and it's a painful experience trying to use one with modern bloated software. It's horrifically expensive, and the available configurations aren't up to much. £2000 for a crippled antique? No thankyou...

The SunBlade 1500 isn't really that bad a machine. It's still over £2000, but it's not that much more than the 150 and is 2-3 times better. Still, £2000 just to get off the ground? Heavens...

I'll skip the 2500, as I think the Ultra 45 is similarly specified, but with a little more expansion and future-proofing. The starting price isn't that much more than a 1500 either, but rapdily rises.

All these boxes are out of my price range as an individual, and I couldn't really justify an employer buying them either.

The more powerful boxes are better value, but you have to pay for the privilege.

As far as I'm concerned, a reasonable entry-level box would be something like the SB1500, at about £1000 or so. Doesn't have to be fancy, but XVR-100 graphics and 512M memory minimum.

There's a similar story when it comes to sparc servers. Sun still sell the V100 and V120. OK, so there's a good market, and these machines have their uses. And they don't have much competition - there aren't many other machines of that low a spec out there.

At least with servers you definitely get into value-for-money territory as you move up the range. Certainly any of the T1 (Niagara) boxes are awesome. But there isn't much value in the sub £5000 space. Again, something like a V100 but with a modern US-IIIi processor at just over the £1000 mark would be handy.

The whole sparc low-end lineup looks incredibly stale. Even the Ultra 45 isn't much more than a SB2500 rehash (a good rehash, but still a rehash).

As might be expected, there's more in the Opteron world down at the low end. You can't really quarrel with the X2100 and Ultra 20.

But even here, Sun make it hard to get something decent. The problem (and this afflicts the whole of the range) is the lack of configuration choice. They seem to make the basic assumption that there's a cheap and nasty option that has the minimum of everything, then have another option in which most things are upgraded, and maybe another option in which everything is maxed out. That's not what I want. I don't want to have to pay for a fancy graphics card just to get a faster cpu, or the other way round. And sometimes I might actually want the base system with a 250G drive. Most PC vendors I look at allow you to select the various components of the system independently, so that I can put together a system that's balanced to meet my needs, but Sun won't let me do that.

Java DTrace

Stephen Lau announced the latest nightly OpenSolaris delivery. Looking through the changelog, I noticed:

Issues Resolved:
        PSARC case 2006/054 : DTrace JNI Binding
        BUG/RFE:6384263PSARC 2006/054 DTrace JNI Binding

Yay! Getting to DTrace from Java!

Evil JES Installer

I'm a glutton for punishment. Must be. I can't think of any other reason why I put myself through this.

I'm testing out the Java Enterprise System. Version 2005Q4 comes in the DVD kit with Solaris 10 Update 1, so I thought I would try that out, following these sample instructions.

The installer starts off OK, but then it complains that J2SE is obsolete. Say what? This is a brand new S10U1 install, and has a newer JDK than is supplied on the JES media. I selected manual upgrade (promising that I would upgrade it myself) in the hope that it wouldn't do anything stupid.

For what it's worth, Solaris 10 Update 1 ships with J2SE 5.0_06, while the JES media contains the older (and insecure) 5.0_04. There are two major issues here already:

• JES ought to have its java version in sync with the version of the OS it's shipped with
• It ought to detect a newer version and accept it as good

So I plug through the screens. (There's another one where it complains about the versions of JATO, JAXP, and JAF being out of date. Why? This is the latest all-singing all-dancing version of Solaris, hot off the press. Why aren't those components up to date?)

So I get to the end and tell it to go install. And what does it do? It downgrades the system Java to the old insecure version!

Aaaaarrrrrrgggggghhhhhh!!!!!!

This is plain bad behaviour, compounding its previous errors with a heinous crime.

JKstat updated

I've updated JKstat - my Java JNI interface to Solaris kstats.

This version adds jiostat, a graphical version of iostat. This is just a basic hack at the problem - I want to be able to select and hide statistics, sort by output, and connect associated statistics (such as a disk with its partitions, or a metadevice with its components). However, the first implementation had pretty awful performance, which is why version 0.09 got skipped.

Why do arrays have even numbers of disks?

Like it says: Why do arrays have even numbers of disks?

Most hardware disk arrays - certainly the ones that Sun sell - currently have an even number of drives in them. The StorEdge 3x00 series have 12, while the 6130 has 14.

The problem I have is that if you take away one drive to act as a hot spare, you're then left with 11 or 13. Not only is this an odd number, it's also prime.

So, what sort of sensible grouping of the drives can you come up with? I often punt and simply create a huge raid-5 volume spanning all the drives I've got left, which is simple. But there are cases when I really want to configure 2 identical sets of disks - either to mirror or to give to 2 hosts. To make this work, I have one drive left over (so I use it as a second hot spare).

Wouldn't it be neat to add an extra drive?

Roundup

Rounding up a few loose ends:

The OpenSolaris Visual Panels project has now started. This really interesting stuff, especially from my viewpoint of developing SolView and JKstat. I agree with JC Van Nieuwenhoven's comment - that managing Solaris with a GUI is a pain - and it's good to see efforts underway to fix this.

(One thing I would say, though, is that good GUIs aren't just for novices. While I might know the 16 arcane commands to configure something, would I rather have a good gui and press 1 button? If it was a good gui, yes!)

Following on from my application profiling to see if a T2000 would be a good thing, I found this article that explains a little bit more about what sorts of behaviour might throw off the statistics. Based on this, I think my machine is spending a lot of time in memcpy.

I also notice that the Ultra 45 workstation lists a 146G SAS drive as an option. I can't see this on the published price lists for the X4x00 or the T2000, but I hope it's on its way as it would help address one of the problem areas I've been having for a while.

Should I get a T2000?

So I'm looking at the new Sun T2000 boxes, and I tried the test program to see if my workload is suitable.

Now, this is a web server. That's all it does. And it's running coldfusion (ie JRun, as in Java), and Oracle, so first thoughts are that it should match pretty well. So I give pfp a whirl.

# /var/tmp/pfp -p 10
We observed 407247665 instructions separated 
in 11.17% floating point and 88.83% others. 

This workload is not recommended for UltraSPARC T1 systems.

That's not good!

OK, so that was an isolated incident. But this machine tends to stick at about the 1.5% grey area. This is typical:

# /var/tmp/pfp -p 10
We observed 1960035483 instructions separated 
in 1.61% floating point and 98.39% others. 

This workload is a potential fit for UltraSPARC T1 systems 
 and need to be tested.

Now, what I don't know is whether there's something odd about this machine, or Oracle, or Coldfusion, or the CMS sitting atop it, or the versions (oldish), or something about the fact that this is an old V880 running an old version of Solaris that pfp can't handle properly. But in any case, the T2000 doesn't look like a given.

I also tried looking at one of the machines I built myself recently, with an Apache/Tomcat/Postgres combo:

# /var/tmp/pfp 60
We observed 2132762256 instructions separated 
in 0.05% floating point and 99.95% others. 

This workload is recommended for UltraSPARC T1 systems.

That's what I expected. (And I get the same sort of thing on one of my Domino boxes.)

So I'm still unclear as to whether a T2000 would be a good bet for the old webserver.

Solaris Info Viewer

Following a discussion on the sysadmin-discuss list, I've put up SolView, a utility I put together a year or so back and then forgot about.

The idea is to have a single window that gets you to the important information about a Solaris system quickly and easily. It hasn't been extensively tested, and relies on Solaris 10 at the present time. Comments etc, especially suggestions for new capabilities or information it could display, are welcome.

Suspend/Resume

After my recent confession, and looking at start up time, I tried suspend/resume to see if that could get the system going any quicker.

Now, I had earlier problems with suspend/resume taking forever, so I tried again after updating to S10U1.

And, OK, so the resume is quicker than it was. But it's still about 3 minutes - just the same as a cold boot. There has to be something wrong here - resume should be quicker, as it just has to bring things back into memory and set them running again without having to go through the thinking step of how they got there.

System start up time

To follow on from why I use windows, I just did some (unscientific) start-up timings on my SunBlade 1500 running Solaris 10 Update 1.

Time in seconds from pressing the on button:

1. Solaris license terms: 41s
2. Console Login prompt: 104s
3. Desktop Login prompt: 133s
4. JDS start: 165s
5. JDS ready: 190s
6. Terminal ready to type: 199s

There's a 30 second hole in the middle where I'm typing in my username and password (and writing down the numbers) where 10 seconds would be more reasonable, so the full length of time to actual login is almost exactly 3 minutes. (And then another 20 seconds to open Mozilla.)

(I've broken the graphical startup into 2 phases, dtlogin start and JDS start, with the human typing in the middle. The whole graphical login process takes about a minute of the 3, with the general boot taking the other 2 minutes. OK, so it's possible to make the gui startup much quicker, but there's still the whole hardware phase and kernel boot to get past.)

What's the machine been doing all this time? A quick look at iostat immediately afterwards:

                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   74.5    3.5  824.7   37.0  0.3  0.8    4.3    9.7  13  46 c0t0d0

Or in raw terms from kstat:

 reads                           10986
 writes                          482
 nread                           124590592
 nwritten                        5022208

So during boot I read 125M of data off the disk, in 11,000 reads. And this actually only covers about 2/3 of the boot - it isn't until about a minute into the boot that the kstats are created. If you allow for the login delay while I'm typing username and password and the few seconds it takes to actually run the above commands, and you can see that the disk is actually well over 50% busy during the boot. Based on the disk activity, boot times can't improve by better than a factor of 2 unless the disk access pattern changes (larger reads than the average 11k seen here would help).

The cpu statistics can also be obtained:

 cpu_nsec_idle                   102634677402
 cpu_nsec_kernel                 44861276123
 cpu_nsec_user                   25778566225

Remember that these numbers don't cover the first 1/3 of the boot. (And they don't add up, either, as I make that 170s which is more than the 140s between crtime and snaptime.) But looking at it, the processor is less than 50% busy.

I'm not sure how to look at these numbers and convert them into the sort of boot-time improvements that might be made, but taking the 50% resource utilization at face value indicates that the portion of the boot covered by the kstat collection could be sped up by a factor 2, which takes the overall cold start from 3 minutes to 2. That wouldn't be bad, would it?

Why I use Windows

I have a confession to make.

My main home computer runs Windows XP. Now, I'm a Solaris guy, so you would expect me to have banished Windows, but no, most of the times I use a computer at home it's the Windows box that gets the nod.

There are reasons for this renegade behaviour. The two main ones are Windows ability to do suspend-to-RAM and fast user switching. Essentially, if I just want to check my email quick, then either the Windows machine is on already and I just have to switch user, or I hit the power button. In both cases it's about the time it takes me to get the seat located comfortably before I'm online and working.

Solaris can't match this. If I'm working at home then the Sun box comes on, as the superior behaviour and environment are worth waiting 10 minutes for. The boot time is awful; the JDS startup time is awful; suspend resume is only on sparc and is very much hit and miss. If we're after wider adoption, then this is a huge area we have to address.

newboot 2 - nvidia failure

OK, so having had great success upgrading my home W2100z to Solaris 10 Update 1, I tried it on my work machine.

Now, I wasn't expecting this to be entirely trouble free. My work machine gets all sorts of abuse, with various test versions of anything that might be lying about - so before it started I cleaned up an old zfs beta release, deleted all the different sorts of backup software I had been trying, and tried to clean up and back out all the hacks and kludges.

In the end, none of the problems I was anticipating surfaced. But when it came back up - no graphics. Looking at the /dev/nvidia* and /dev/fb* entries, they looked very suspect.

Turns out this is a known problem with the NVIDIA drivers and the S10U1 upgrade. And it was essentially what I had surmised by looking at the device entries, so a quick deinstall, reinstall, clear up /etc/path_to_inst and a couple of reconfiguration reboots later and everything's back in business again.

Phew!

newboot

Got my Solaris 10 1/06 DVD kit yesterday. So popped the x86 DVD into my W2100z (running Solaris 10 FCS) and let it get on with it.

Upgraded without a hitch. Everything looks good!

The boxed set includes quite a lot of stuff, so I also updated the freeware, and added the Studio 11 compilers. I'm leaving the Java Enterprise System for another time - I'm not going to risk playing with it on my main personal machine!

One interesting thing - I've got a decent monitor (a Sun 21 inch CRT) on this machine, and it had been running at 1280x1024. I hadn't really had time to investigate tweaking Xorg to get it to run at higher resolution. After upgrading, it came up on its own at 2048x1536. This was a little over the top, so I dropped it back to 1600x1200, which is the max recommended anyway, and runs at 85Hz rather than 75Hz, and makes the text a little more readable.

Everything still works, all I need now is a little more time to get back into development work.

Buried in good stuff

The good stuff keeps on coming.

OpenSolaris now has even more communities - particularly for sysadmins and appliances - to follow. As if approachability, networking, observability, zfs, and zones weren't enough. And that's only half the communities I'm interested in!

What I really need is a time machine so I'm able to keep track of it all.

Enemy action?

According to Auric Goldfinger:

Once is happenstance. Twice is coincidence. Three times is enemy action.

One: panic[cpu1]/thread=fffffe80fd47aea0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8001773c70 addr=0 occurred in module "genunix" due to a NULL pointer dereference

Two: panic[cpu0]/thread=fffffe80f83a5de0:
BAD TRAP: type=e (#pf Page fault) rp=fffffe80010e8bd0 addr=0 occurred in module "unix" due to a NULL pointer dereference

Three: panic[cpu1]/thread=fffffe8000f56c80:
BAD TRAP: type=e (#pf Page fault) rp=fffffe8000f56450 addr=0 occurred in module "conskbd" due to a NULL pointer dereference

That's 3 failures on my desktop machine in just over a month. For those interested, the stack traces are:

stack pointer for thread fffffe80fd47aea0: fffffe8001773cb0
  fffffe8001773d70 0xffffffff8a965940()
  fffffe8001773da0 port_remove_done_event+0x4b()
  fffffe8001773e10 port_associate_fd+0x2b8()
  fffffe8001773ec0 portfs+0x303()
  fffffe8001773ed0 portfs32+0x24()
  fffffe8001773f20 sys_syscall32+0xd9()

stack pointer for thread fffffe80f83a5de0: fffffe80010e8c10
  fffffe80010e8d10 tcp_close+0xff()
  fffffe80010e8d50 qdetach+0x84()
  fffffe80010e8dc0 strclose+0x3e4()
  fffffe80010e8e00 socktpi_close+0x12b()
  fffffe80010e8e30 fop_close+0x2a()
  fffffe80010e8e60 closef+0x62()
  fffffe80010e8ec0 closeandsetf+0x249()
  fffffe80010e8ed0 close+0xb()
  fffffe80010e8f20 sys_syscall32+0xd9()

stack pointer for thread fffffe8000f56c80: fffffe8000f56330
  fffffe8000f56c80 5()

Huh?

This can't be serious...

I've often wondered why Solaris patches take so long to apply.

I'm still wondering, after updating the Java patch (118668, for the technically minded). OK, so it's a big patch, but I've got a high-spec dual Opteron W2100z so it should be done in the blink of an eye.

Or maybe not. I decided to back out the old revision (this is one of those foibles I have - I tend to apply patches regularly on test boxes, so they get every revision, so I have a habit of backing out old revisions to keep things clean). The backout took 10 minutes, adding the new version 5. That's 15 minutes on one of the fastest machines around. Ouch!

One thing I did notice is that the patch backout was writing at an average of 30Mbytes/s for most of that 10 minutes. Overall, I reckon that I had about 15Gbytes of disk writes. Why on earth?????

Clearly it's up to something very clever here.

(For comparison, simply installing that version of java - which is about 135M - generated about 200Mbytes of disk writes and took about 20 seconds. And some of that is accepting the license, unpacking and verifying the archive, and building the jars. Clearly there's some scope for improvement.)

Software Stacks

I liked the idea that Ben Rockwood came up with - SIDEkick. Essentially, a complete software stack, in this case for a php-powered postgres-backed web server, in a single file.

This is along the lines of my simplicity rant the other day. What I want is things set up ready to go.

I do this myself, for most of the projects I get involved in. For a project, I build up a software stack that contains all the components necessary, with an install script that does the work - and not only installs the stack but can also put project-specific customizations in place, and install data if required. It's largely self-documenting as well, as the install script contains all the tweaks and kludges I needed to get the thing to work.

The full stack actually has quite a bundle of components. The primary ones are:

• Apache httpd
• Apache tomcat
• Tomcat Connectors
• OpenSSL
• mod_ssl
• MM
• PHP
• MySQL
• PostgreSQL
• Apache ant
• Apache Axis
• Apache XML-RPC
• SOAP::Lite

I still use Apache 1.3.X, due to problems I've had with 2.0.X and 2.2.0 not working right. Of course, not all projects require the full set.

As an example, I've used this software stack to enable easy deployment of DSpace, in addition to basic web servers and complex soap application servers.

Which raises the valid question - why build the stack myself?

There are several reasons. Being self-contained is a pretty good reason, all on its own - having everything in a single bundle with all dependencies means you don't have to worry about how to integrate it with some other component that may or may not be installed, may or may not be the right version, and may or may not be configured compatibly. It just works. You make yourself independent of other suppliers, who may change things underneath you without your knowledge. And it fully documents and supplies your requirements so that someone else can come along later and not only understand what you did but has all the bits already at hand to reproduce it.

What I would like to see is more lightweight software stacks developed - in a modular fashion so that I can just click a link and get a single file that gives me a well-defined area of functionality. Big frameworks are all very well, but there's an awful lot of complexity due to the generality that comes with a full framework, and that gets in the way of actually doing work.

This also means that I don't really want - at all - bits of the stack bundled with the operating system. These are just a pain and get in the way. I've nothing against OS suppliers having the full software stack available, but please compartmentalize it so I can safely ignore it.

Licensing Complexity

Following on the heels of my rant about keeping software simple, is another one:

Why are licensing schemes so horrendous?

A while ago, I was looking at backup solutions. I've just started to get costings together. What a nightmare!

Really, how hard can this be? But then there are different tiers of servers, prices are different for different platforms, some of it's done by volume (how many terabytes can you afford?), tape libraries by capacity in several dozen steps. There are different base products, with no explanation of the differences.

(This sort of thing screams out for an online order form where you just tick the boxes and it puts together an order for you. How do I know whether I've actually chosen the right options from 6 densely packed pages?)

Updated NVIDIA drivers [again]

Dang, after only just noticing the latest Solaris NVIDIA driver release, they've gone and updated the drivers again.

Start out Simple

One of the failings of modern IT infrastructure is that it's far too complex. Individual components are complex, and have complex interdependencies with other complex components. And some of us have to make sense of this mess every day.

Sometimes, complexity is unavoidable. That's OK. If it's a complex problem, then I expect some level of complexity in the solution. It would be better if the solution were simple, but we can't always have what we want.

Sometimes, new technologies come along that radically simplify the way that things are done. ZFS is one recent example - it takes away whole layers of complexity. But, as a rule, things get more complex over time as layer upon layer of cruft is added.

While you may need something big and complicated to solve the big and complicated version of a problem, does that mean that you need to be equally big and complicated to solve the little version of that problem? It seems that, all too often, you do need the big complicated version - with all its attendant hassles - to solve the little problems. Or, at least, that that's what we end up using.

I've always been opposed to this approach. I've always been in favour of starting off small and simple. I want to get something working, without delving deep into an impenetrable morass of configuration and tuning. Then, having got that to work (and, more to the point, having understood it), I can build on that foundation.

I've spent some time playing with the Java Enterprise System. And you know what? It's way too complex and hard to get into. I'm sure it can do wonderful things, but before it can do wonderful things I want it to do something. Anything, really - just to give me the sense of accomplishment that keeps me going to the next stage. I really can't see JES getting that much of the market, simply because most admins and organisations simple don't have the time and energy to invest in making it work at all.

Part of this is ease-of-use, but it's slightly different. It's really about the ease of getting started. And that's what complex technologies need to supply: an easy way in, to allow potential users to get started.

Updated NVIDIA drivers

For those not constantly checking, NVIDIA have released updated Solaris drivers.

(OK, so it was a few days ago, but I had missed it.)

LOSUG 2

Went along to the LOSUG meeting last night. Good to meet up with everyone again.

Mulled wine; nibbles; good talks; even the occasional mince pie finally made an appearance.

The lightning talks part worked pretty well.

Even managed a quick pint before having to leg it for the train home.

Domino Backup

One of the problems I'm working on at the moment is online backup of a Lotus Domino server running on Solaris.

Nothing too complicated, right? Just whip out your favourite backup solution , install the domino module, and you're good to go. Right? Wrong!

I've tried Legato Networker, which I've used for regular backups without any problems for the best part of a decade. Works on a trivial test, fails completely on the real thing. I've tried Backup Express from Syncsort (used by our PC systems) and haven't yet managed to persuade it to recognise that I've got a domino server installed.

I stumbled across BakBone, who make something called NetVault. I had never heard of it, but first impressions from the web site were good, and I was able to get an eval copy off their download site straight away. Installed pretty easily, and it wasn't too hard to work out how to drive it, so it's currently doing a test backup. (Performance isn't too bad, especially considering I've set it up to save to a disk based virtual library on the same disk array that the Domino server lives on.)

The real test, of course, is to wipe the Domino server out completely and see what happens if you restore it. More on that phase as it happens.

[ID 335743 kern.notice] BAD TRAP:

Bother!

As you may recall, I've been playing with apache httpd 2.2.0.

I was also looking at Derek Crudgington's comparison of Apache and Sun Webserver. So I decided to just test out Apache 1.3.34 against 2.2.0, and also going through to tomcat which was generating dynamic pages from mysql.

To cut a long story short, testing Apache 2.2.0 paniced my machine. I wasn't able to do too much damage with 1.3.34, but under stress 2.2.0 became rather sluggish, and then it and the whole machine became completely unresponsive.

Of course, it's not apache's fault. It shouldn't be capable of taking the box out. This is definitely something in Solaris that's gone awry.

A quick search of sunsolve didn't show a match, but for the enthusiast here's the (trimmed) message:

Dec  6 11:32:28 ratbert genunix: fffffe8001773b80 unix:die+da (fffffe8001773c20, 1fb955d3a)
Dec  6 11:32:28 ratbert genunix: fffffe8001773c60 unix:trap+5ea ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773c70 unix:cmntrap+11b ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773d70 genunix:list_remove+b ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773da0 genunix:port_remove_done_event+4b ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773e10 portfs:port_associate_fd+2b8 ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773ec0 portfs:portfs+303 ()
Dec  6 11:32:28 ratbert genunix: fffffe8001773ed0 portfs:portfs32+24 ()

Simple presentations

One of the things almost everyone has to do sooner or later is make presentations. And while some people can just stand up for a few minutes and hold an audience's attention, most of us need some sort of visual aids.

Personally, good old fashioned overhead foils work fine. But we're in the 21st century, and it's almost impossible to find an overhead projector.

So the general solution appears to be powerpoint or something similar. I've been using StarOffice Impress for a few years now. It sort of works, but it's a very clunky way of doing things.

Yesterday I came across Eric Meyer's S5, a simple but phenomenally powerful slide show system. Using a combination of XHTML and CSS you can easily and quickly put a simple presentation together. If you can edit basic html, you can put together a presentation. (And using superior content creation tools like emacs or vi, at that.)

As a very trivial example, I've put together a presentation on JKstat.

New Apache

Just recently, Apache httpd 2.2.0 was released.

Now, I've been a bit traditional here. I'm still using apache 1.3.34, almost always with mod_jk to talk to tomcat, and occasionally with mod_ssl for https support.

Why not upgrade? Well, there are two reasons really. One was that it wasn't at all obvious that 2.0.x was in fact an upgrade. It always felt like a retrograde step and if anything I would describe 1.3.x to 2.0.x as downgrading. The second is that actually getting 2.0.x installed was a right pain. They mistakenly switched to using autoconf, so it's much harder to get the installation and configuration right. (If it installs at all. Many a time I would find autoconf just goofing out on one of its random guesses and failing to do anything at all.)

So, is 2.2.0 any better? Well, it still uses autoconf - and it's still a very bad move - but it is possible with enough effort to circumvent most of its mistakes (although not all). But it includes ajp support to talk to tomcat, including load balancing. Which, coupled with native ssl support, should reduce the complexity of installation - if it works.

So far, running under Solaris 10 works fine for http. I haven't exhaustively tested the ajp support to see how well it handles load balancing and failover, but basically it works and looks good.

I had fun and games with getting https to work though. Essentially, the combination of httpd 2.2.0, gcc and the Solaris 10 openssl libraries didn't work. (Using curl I could get sslv2 to work but not sslv3, and neither mozilla nor firefox would have anything to d with my server.) Compiling up the latest openssl myself (which is what I do for 1.3.x anyway) with gcc works just fine. So it's either a gcc vs. cc incompatibility, or a version problem (Sun are supplying quite an old version), or some other strange incompatibility. It would be nice if I could rely on the openssl bits that come with Solaris, as openssl itself is a reasonable size and takes quite a while to build, but it looks as if I still have to do it myself.

Awash with freebies...

Seems like Xmas has come early.

Of course, Sun go on at length about the Java Enterprise System. Now, this is interesting in parts, but JES is a complicated beast and likely to be of interest primarily to - well - Enterprises.

What I like, though, is the promise of free stuff a bit further down. There has been a good emphasis on developers recently - Studio 11 and Creator, for example. But what's also now promised is free versions of Tarantella and SunRay, which are likely to be of interest to a far wider range of customers.

And as I read it, the nebulous N1, including Sun Management Center, is included in the deal too.

suspend/resume at a crawl

I've got a Sun Blade 1500 at home (one of the old red ones). Works great.

Apart from suspend/resume, that is.

I have no idea why, but both suspend and resume take an absurd amount of time. The suspend isn't too bad (slower than it should be), but resume is in the 5-10 minute range. To use an Americanism, this sucks.

(It's doubly odd because I've tried this on a Blade 150, and that's much, much quicker.)

Get it right first time!

Many years ago I wrote a simple system and network monitoring tool. It's been developed on and off over the years, but has now reached a major impasse.

Basically, I designed it wrong 10 years ago. I started out with a 2 state system. If the status is 0, then it's fine. If the status is 1, it's broken and needs fixing. Sounds reasonable, right?

Then I realized I needed to add another state, so I defined it so that if the status is 2, there's a warning condition. And all worked well for a few years.

The problem with this scheme is that the severity of the problem isn't a linear function of the status. So I end up playing all sorts of games trying to analyze the status codes trying to work out just how bad the situation really is. It would be much easier if I could simply retrieve the maximum status out of the database - no fiddling required! And I can order problems simply by sorting on the status.

Thinking about this a bit more, this is the obvious thing to do. So obvious, in fact, that I was a dullard for not thinking about this at the start. (But, when I started writing this particular monitoring tool, I wasn't thinking about what version 3 would look like 10 years down the line. And I started out by using the return code from scripts as the status, which is where 0 and 1 came from.)

Of course, I now have to consider what the best scheme might be. Do I simply have 0 for good, 1, for warning, 2 for dead? I think the 0 for good is fine. But should I do something like 255 for dead, 128 for warning, leaving me some room to add finer levels of granularity in the future?

Decisions, decisions...

Another JKstat update

I've updated JKstat to version 0.08.

It's getting better. The accessory widgets have been cleaned up and a couple of new ones added (distribution of packet sizes on bge interfaces, and dma transfer rate on ifb graphics cards). Rates are now accurately computed based on the actual snaptime, rather than approximately based on the intended refresh interval. A couple of internal changes streamline the whole system. And I've fixed it so that actually enumerating the kstats doesn't blindly read all the data, which improves performance.

With these changes, I'm much happier that it's closing in on its design goals. I was tempted to bump the version up to 0.1, but that would probably be premature based on the number of bugs that I introduced and fixed recently.

My next idea is to build a graphical iostat. Why is this of value? Well, pictures tell you a lot - the eye is very good at interpreting graphical data. You can dynamically hide uninteresting data, or expand areas of interest for a finer view (for example, you could dynamically expand a disk's I/O to show partition data). You can show historical rates, and generally have multiple views of the same data. You can use the gui to show additional context-sensitive data beyond the basic I/O data. And you could, in the future, link to other areas of functionality - such as dtrace to show what was causing all that I/O in the first place.

Bumps-a-daisy!

Had a bit of a problem yesterday. While driving to work I got bumped hard from behind, in stop-start traffic on the A1(M).

Nothing that serious - nobody was hurt, which is what really matters. The other car was a total wreck, and the rear-end of my Toyota is pretty well squashed. It's not so bad that it can't be repaired, so in a week or so it goes into the body shop and should be all fixed again ready for Christmas.

I have to say, though, that the insurance company aren't exactly covering themselves in glory here. I mean, they presumably deal with this sort of thing on a daily basis, but they do seem to be making heavy weather of it.

Properly connected

For a long time now, we've had a broadband connection to the internet, but we only had the main home PC hooked up.

No longer! I'm typing this from one of my home Sparc machines running Solaris.

What took me so long I'm not sure, but I finally went and ordered a little cable router (a non-wireless version, which seem to have vanished completely from the shops in favour of wireless models which cost twice as much and I can't take advantage of). Put in the Setup CD, follow the instructions, and it was working. Connect up my Sun, tell it to use dhcp, and I'm online.

I love it when things just work!

Now to find some really long cables to connect the machines upstairs...

Quest for small server continues...

I'm still working on my quest for a small reliable server.

I was just reading Richard Elling's blog entry on RAS and the X4100/X4200 servers. You should read this - blindingly obvious design features like not putting heat generators like disks in the airflow path for the CPUs. But he also says that most thin servers don't need more than 2 disks. Perhaps this is why I'm having so much trouble finding a server to fit my requirements!

Oh well, having exhausted Sun's catalog, I'm now looking at the likes of the HP DL385 or the Dell PE2850. Both of these are listed in the HCL, which is pretty much essential as I would naturally be running Solaris on the machine.

PostgreSQL, Sun, and Integration

Sun sure are busy with the announcements this week.

With the PostgreSQL announcement (and they don't seem to have mastered the spelling of PostgreSQL, it seems), Sun are offering to integrate and support PostgreSQL into Solaris.

Now, this has to be a good thing for both Sun and PostgreSQL. But does it help me?

I'm not really sure that integration does help me. Note that it's the integration - or bundling - that I have a problem with, not Sun supporting it or optimizing it or just supplying it.

Sun already bundle the Apache web server and Tomcat. I spend quite a lot of time setting up web servers, usually using Apache and Tomcat, often with other components (including, as it happens, PostgreSQL on occasions). And I never use the bundled versions that Sun supply. And the reason it quite simple - Sun's versions aren't the right versions, aren't set up the way I need, and are installed in the wrong place. It's much easier and safer to just install them yourself and you know exactly how they're set up and that they're going to work exactly the way you want, and that you can upgrade to the latest version at any time of your choosing.

Integration really ties you up in knots. Solaris comes with ancient versions of Gnome, and because they're integrated we're stuck with them. Not only that, because it's integral with Solaris 10, we can't apply the same version to our Solaris 9 or 8 machines. Integration locks application update to OS updates, and everybody loses.

What I want - and Sun need - is the ability to choose between sticking with a given version, or going to a new version. This requires that products such as Gnome/JDS (and the same argument applies to anything else, like Mozilla, OpenSSL, Apache, even Java) are unbundled and separated from the core OS. Then, I can select whether I want to stay with Gnome 2.6 (as in the version that comes with JDS on Solaris 10) or have a Gnome 2.12 desktop instead.

Likewise for PostgreSQL, which started this whole blog entry. For different applications, I'm going to have to support different versions - maybe on the same physical machine (using zones, for example). I need the ability to make that choice independent of the underlying OS version, otherwise you end up with an upgrade nightmare.

Deluged by good stuff

Whole load of interesting stuff coming out of Sun at the moment.

It seems that free stuff isn't just for Fridays anymore. We now get free developer tools. This is something I ought to try. I have to confess to being one of the old-school who can't see what on earth is wrong with emacs, but I'm always keen to try new things, and maybe an IDE might help out.

Then we get snippets about the new Niagara chip. Marketing have clearly got in on this one ("CoolThreads", anyone?, and there does seem to be a certain greenness in the positioning, but this looks like some serious technology.

More free stuff - the Studio 11 compilers. Making these free was inevitable, really - Studio has been free to anyone in the OpenSolaris community for some time now, and I've long felt that the excessive pricing for Studio was crippling its uptake. Good one!

Is that enough? No way! Something really big happened this week - ZFS was unleashed on the world. Sure, it got hyped (overhyped) a year ago when Solaris 10 got announced, but ZFS is the real deal - I've been privileged to have been testing it for almost a year and a half, and it does what it says on the tin. So go check out all the blogs.

Turning Opteron Down

I was putting together a server spec recently. Nothing special, just a reliable box to store 100G or so of data safely and serve it up via the web.

Easy, right?

Well, that's what I thought, and I was wrong.

I have this thing about real servers. They have to have redundant PSUs, redundant disk - mirrored. This means greater than 2 internal drives.

(Note that, according to this definition, Sun's SF280R, V210, X2100, E220R, E420R, V480, V490, V20z, and E1280 don't qualify. All are limited by 2 internal drives. They're fine for compute nodes and similar tasks, where the aim is simply to survive long enough to finish the job and decommission the node, but not for real servers. You have to have at least 3 disks to guarantee survival - and reboot - after a disk goes. OK, so you're supposed to add external arrays, but usually you can't do anything like place metadevice databases on the arrays. And also, only having 2 drives make Live Upgrade harder than need be. End of first rant.)

OK, so the next thing is that 100G of storage. It doesn't really justify getting an external array - that's fine for a terabyte, but would be a waste in this case. And, unlike something like 10G you can't just lose it on the boot drives. So 100G is an interesting number.

Grabbing 100G off a SAN doesn't look promising either. Apart from not having one to hand right now, the cost of the HBAs makes a nonsense of it for this amount of data.

So, what else? iSCSI could be interesting, as it saves you the cost of the HBAs. But it's not really mature yet, and I don't happen to have a server handy. (I don't happen to have a convenient NFS server either, which is a shame.)

OK. So the next best thing is to get a box with 4 drives - 2 to house the OS and the application binaries, and a couple extra 146G drives for the data.

So, I start of by thinking - these Sun Opteron boxes look real nice. Particularly the X4100, which can take 4 drives without the DVD. (And you don't need a DVD - it's just something else to waste money and electricity.) However, this won't work. Sun only offer 36G or 73G drives. Not enough! And there isn't a slightly bigger variant that takes more drives. OK, so Sun don't make an Opteron box that will work. Bother.

So, go to Sparc. The V240 works a treat. I like the V240. A couple of boot drives and a couple extra 146G drives and I'm all set. It's interesting that an old Sparc box is better suited than a new Opteron box.

(Not that the V240 is perfect. In the same way that it's a major disappointment that the Opteron boxes don't take 146G drives, it's disappointing that the V240 doesn't support 300G drives. Why don't Sun realize that customers want choice?)

OK, so I'm a Solaris fan, and Solaris x86 runs on a wide range of systems. A quick browse through other manufacturers websites (and some of them are nowhere near as easy to navigate as they ought to be) shows that this trend of useless system design is fairly widespread. Other manufacturers are more agile at supporting larger drive capacities, but the systems designs are similar.

In the end I decided to simply park the problem in a zone on a bigger system. It's a good solution, and was what I wanted to do anyway.

What is intriguing is that Sun used to have ideal systems for this sort of task, and have now scrapped them. The V60x allowed you to have 3 drives, so you could avoid the twin-drive trap. The V65x was a wonderful compact server and let you put 6 drives in. The V250 let you put 8 drives in the chassis, but seemed to get canned pretty quickly. It's not entirely obvious to me that genuine progress is being made.

JKstat updated

After a long hiatus, I've released an updated version of JKstat.

For those who don't know what a kstat is, it's a Solaris kernel statistic. There are a lot of these, and they give you an awful lot of information about what you Solaris system is doing.

JKstat allows you to get at the kstats from a Java application. (Solaris already includes a fabulous perl implementation. One day, I hope JKstat will be as good. It isn't yet.)

The kstats naturally form a tree structure, and I've written a graphical browser that allows you to go through the kstat tree. Like so:



(Oh dear, what has blogger done to my beautiful image? Oh well - click on it and you'll see the real thing!)

There's still a lot to be done. For one thing, I want to actually create a decent API rather than the horrible kludge that I'm using at the moment (it's this, rather than any lack of maturity or functionality, that keeps the version number at a lowly 0.07). And there are a number of existing tools that could be enhanced by a decent graphical user interface (in particular, the ability to dynamically expand or compress certain features - imagine iostat with the ability to zoom in on specific disks and show or hide the partition data on the fly). Looking further ahead, one can imagine integration with dtrace to answer the question "what is causing this activity?".

Enjoy, and if you have any comments (and, in particular, you would like a graphical display of a particular kstat, or can think of novel and useful ways of displaying kstat data) I would love to hear them.

Everyone an administrator?

James Dickens asks: Why not a server?

And it's an interesting question. Why, in a house with multiple computers, do you not have a dedicated machine somewhere and store all your files on it? It makes a lot more sense than having files spread at random amongst all those machines.

My own solution to this is a portable USB zip drive. I use this to carry stuff about between my machines, and between work and home if needed.

I'm not sure that the suggestion of using a real computer (and an Ultra 2 certainly qualified as a real computer) as the server though. Yes, I know that Solaris is an absolute doddle to administer (yes, really - once you've got to know it). But it's bad enough that everyone owning a Windows PC has to be a systems administrator, without expanding that even further. Even though it pays my wages, I'm a firm believer that when it comes to systems administration, less is definitely better.

Using a general-purpose computer doesn't necessarily make sense to me. (The one situation where it really comes into its own is if you were to use it as something like a Sun Ray Server.) But generally, some sort of appliance seems to make more sense.

And an appliance running a cut down OpenSolaris with ZFS would be a stunner.

The only downside to a server is that part of the assumption is that it's always on. I'm not sure that we should be encouraging that and the accompanying waste of power when there's so much damage already being done to the environment.

Hosted services - grid, if you like - also have limitations that are painful. They neatly solve the administration, availability, and backup problems, though. The biggest problem I see is that upload speeds on my internet connection are absolutely pathetic. Most internet connectivity is highly asymmetric - fast download, with just enough bandwidth the other way to handle the administrative packets and not a lot more. If we are to see hosted storage really become useful, then upload speeds are going to have to be significantly increased. (And, frankly, network reliability could stand a little improvement.)

Invalid system disk

I was updating Solaris on my W2100z the other night. After applying the latest recommended patches, I rebooted and went off to get a drink.

On my return I was rather startled to find displayed the old message "Invalid system disk. Press any key to continue." Oh bother - what's gone wrong?

It would be a bit of a pain, because I do quite a bit of development on this machine and I really don't want to lose it. The usual suspect in these cases would be a floppy disk, but the W2100z doesn't have one, so I was worried for a second that the main disk had been trashed.

But then I worked out what had happened. I had taken the recommended patch bundle home on a zip disk, and left it connected. So the invalid disk it was complaining about was in fact the one I had left in the USB zip drive. (So clearly the W2100z is capable of booting off a USB device. Scary.)

Panic over...

Putting Google to work

I don't know how many others use Google as a problem solving tool, but I use it all the time.

I had a problem yesterday, jumpstarting Solaris 10 onto a workstation. Couldn't find the jumpstart directory. So plan A is to drop the error into Google and see what we get.

Bingo! The right answer comes straight back. Yup, simple netmask mismatch.

It's not just that, though. I was intrigued to see that Sun use google for making Solaris development decisions.

Heck, what would we do without Google?

system() doesn't constitute an API

I like APIs. I like being able to call a function, give it some data, and tell it what to do with it, and have it do it without fuss or bother.

And I like interfaces that are stable and reliable.

However, on a typical unix-like system, certain common operations have no real API. For some operations, the normal way to do things is to run an external program.

Usually the system() call of the title is involved, but it could be popen() or some other variant. I'm using system() here as shorthand terminology for executing another program to do the work.

The two standard examples are mail and printing. You throw the data you want something done with at sendmail or lp.

This is atrocious. This isn't an API - this is merely convention that often works. Frankly, it worries me that there aren't standard programming APIs for such common tasks as sending mail and printing a file.

While mail appears to be standardized sufficiently to be largely hidden by applications, printing is another matter. The variety (and general disfunctionality) of print dialog boxes should be a clue here. Worst, of course, are those that simply have a text field that you type the print command into.

Apart from the inefficiency of launching external programs, the lack of genuine APIs limits the interactions a program can have, in particular the feedback it can get from the application it launched.

It's the 21st century. We deserve to have some decent 21st century APIs.

When did that get added?

I haven't used format to partition a disk by hand under Solaris for ages. Normally I set everything up at install time using jumpstart, and then prtvtoc and fmthard if I need to copy a partition table (if a disk fails and needs to be replaced, for example). So I happened to be working on a machine today, and was presented with:

partition> 4
Part      Tag    Flag     Cylinders         Size            Blocks
  4 unassigned    wm       0                0         (0/0/0)            0

Enter partition id tag[unassigned]: 
Enter partition permission flags[wm]: 
Enter new starting cyl[0]: 14521
Enter partition size[0b, 0c, 14521e, 0.00mb, 0.00gb]: 

Say what? What's this e thing? Well, it's what you expect - the end cylinder. Aha! My, this makes putting that last partition in so much easier. No more trying to calculate in your head how many cylinders are left...

(Looking at a couple of other machines, I think this got added in Solaris 9.)

Why don't people upgrade?

Gary mentions a topic I've always found interesting: why do people persist in running old versions of Solaris?

I know of many reasons, some of them are even valid. Most of these are in the general area of supported configurations. If a vendor (or even Sun) won't support a new version of Solaris, then you're pretty well out of luck. It's not just official support, either - sometimes the product plain won't work.

(This last point makes me wonder. Given the very strong binary compatibility guarantees in Solaris, what are some of these vendors doing to make their code stop working? Presumably they think they're being clever, but they must be putting in a lot of effort to break things.)

There are also a couple of burying head in the sand excuses I hear

• nobody else has upgraded so there must be a problem

or

• we'll wait for someone else to try it and find all the bug

both of which are bogus. Let's nail the first one for starters. Actually, plenty of people do upgrade, and don't have problems (if they read the instructions, that is). Do you want to be the one left behind? This herd instinct does seem very strong, though.

The "waiting for someone else" mentality is wrong. For one thing, Solaris doesn't go round randomly breaking functionality - compatibility is very strong and the level of bugs is very low. Secondly, the Solaris development model fixes bugs in the new version first and then fixes the older versions later (if at all), so that upgrading to a new version is the best way to reduce the number of bugs that you might be exposed to. Thirdly, it's actually already been tested (to destruction and beyond) very very hard indeed already. I know, I did it. Yes, we found bugs. But a whole lot of people inside and outside of Sun have beaten on Solaris for a long time, and most of the bugs are gone. Finally, because it's been well tested, the remaining bugs tend to be in strange distant areas that nobody has covered yet. In other words, the chances are pretty good that if you're going to get affected by a bug, it may well be specific to you, your environment, and the way you use the product - and these bugs aren't going to be picked up by other people, so it's daft to wait for other people to do the testing because it doesn't mean anything for you.

There's also the argument "it's not broken, so there's no point fixing it". Sure, it is reasonable not to immediately go out and heedlessly upgrade every time a new version comes out. And once something works leaving it alone is generally a good policy. (But I think this really ought to be expressed as "don't fiddle" rather than "never touch".) But regular maintenance is an essential part of the process, and upgrades and replacements need to be planned for. The problem with just leaving things alone is that they start to rot, and if you leave them too long you have a disaster on your hands. For example, if you don't apply any patches for 4 years, and then hit a problem, you have to apply 4 years of patches - a massive change, and a massive risk. Or if you insist on running old hardware too long, you suddenly discover that when you need to replace it you have to change the OS and application by 3 or 4 major versions and it becomes a nightmare. Essentially, planned steady change is a lot better than hanging on too long and hoping you can survive the inevitable wreck. Better many small steps than a desperate leap over a gaping chasm.

I haven't even considered all the cool new features.

So I have this message to all those still hesitating about upgrading to Solaris 10:

Come on in - the water's fine!

Yay! Gnome terminal fixed!

Every so often, there's a Solaris patch that fixes a problem that's bugged me for ages.

Today's example is a big win. After applying patches 120461 and 120289 (x86), or 120460 and 120288 (sparc), the JDS gnome terminal that comes with Solaris 10 no longer needs you to press the stupid shift key to get PageUp, PageDown, Home, and End working.

This is one of those little useability things that makes a huge difference.

To whoever fixed this - Thank You!.

Being Enterprising

I've been playing around with various bits of the Java Enterprise System.

It comes on a neat DVD in the Solaris 10 media kit. So I had a play with that at home, only to discover that it's missing one thing. Instructions. There is no documentation included. Zilch. Nothing.

The basic instructions can be found on the web, but that's not the point. It's a fairly small pdf file - about 4 megabytes, and there's plenty of space on the DVD to fit that in all the supported languages. It would make it so much easier for people to test.

Another useful document I came across is a useful guide to evaluating JES on a single system. (This would also be absolutely easy to simply copy onto the DVD.) This covers the basic steps you need to get started.

Google Earth

Surfing away last night, I came across Google Earth.

Wow!

This is a stunner. I haven't shown it to the kids yet, but one of them is doing geography homework at the moment and would find it amazing. (One reason for not showing it to them is so that I can get access to the machine.)

New Sun Boxes

So now we know what the new Sun servers are like.

And I'm pleasantly surprised. There are a couple of things that I feel are positive here:

• The entry level box, the X2100. This makes a nice addition to the range. Yes. it's dumbed down, but you don't always need all the features - sometimes you just want a server.
• I'm pleased to see the two configurations of the X4100. You can get this in either a 2-disk (plus DVD) or 4-disk variant. I like the 4-disk variant. Often, a local CD is just a waste of space and money, and I like having decnt local storage (read more than 2 disks - so you don't have to worry about quorum in a 2-disk configuration, and you can have some space to put data for a data base or some such locally). It's amazed me that the V210 (and the V1280, even) only allow 2 internal drives.

OK, so the range is still a little thin, but that's effectively twice as many variants as I was expecting, and I like the look of all of them.

All Change

Today's been a pretty big change for the Tribble household.

I started a New Job, while Amanda started a new school. Not just that, but this involves completely different schedules for getting up in the morning, getting ready, and travelling.

I can't tell you much about the new job, because I've largely been drowned in paperwork. I did manage to unpack and install my new workstation (wow!), but that's about all.

Java man pages

As the holidays draw to a close, I've been trying out a few more ideas.

Solaris comes with man pages in nroff or sgml format, that can be preformatted and indexed with catman. (Why the preformatted man pages and index aren't created by default is beyond me.) But there are still a couple of weaknesses with the man system

• There's no search system
• There's no way to follow cross-references

The Documentation DVD includes html formatted man pages which have the linking element, which is an improvement. But this doesn't solve the search problem.

However, Solaris comes with Javahelp. So I had a little play today with creating a Javahelp helpset out of the html manpages.

This worked reasonably well, up to a point. It was pretty easy to knock up the helpset file, and the map, TOC, and index files, and to create a full-text database for searching.

The Javahelp hsviewer did a reasonable job of providing a user interface. You can navigate, and the search isn't too bad.

Where it all fell down was that the html display in the viewer uses the swing JEditorPane. Which is - shall we say - a bit primitive. So while the overall solution functions reasonably well, the actual display looks pretty awful. Sufficiently bad, in fact, that it's not going to work in practice.

So the idea that I could produce something from a quick hack came to nothing.

I'm not done yet, though. As I see it, there are 3 ways to go forward:

• Recreate the html versions of the man pages with an old html syntax that the javahelp display engine is going to cope with.
• Use a different html renderer component.
• Integrate this into a web server that does the searching so you do the search and indexing in a servlet but just shove the html pages out to a browser.

At the moment, the first looks the most attractive, as this also solves the problem of what to do with other man pages that you might have.

Galaxy - About Time

According to ZDNet: Sun's 'Galaxy' servers making September debut.

Well, it's about time. And not just for these new servers specifically, but the flow of new product out of Sun has all but dried up this year and Sun need to get some new products out into the marketplace.

I generally like the look of these boxes (although the internal storage is more limiting than I would like).

What does worry me, though, is the lack of choice and diversity. I don't see any tower systems or dirt-cheap single processor boxes (like a server version of the Ultra 20, for example).

Those ole backup tapes

Some technologies are spectacularly primitive and have been largely replaced by modern equivalents. One such is backup - the idea that you use something like ufsdump to back up a Solaris filesystem has no place in the 21st century. Tools such as Legato Networker (or the Sun equivalent - whatever it may be called this week) or Veritas NetBackup should have relegated ufsdump to museums years ago.

(I have used ufsdump occasionally - usually piped direct into ufsrestore as a quick hack to copy a filesystem.)

But having closed down the systems at my former employer, deleted all the data and sent the machines off to oblivion, and burnt all the backup tapes, of course I needed to get some files back for one particular project.

So: no tapes, no tape libraries, no backup server, no legato license. But I did have a ufsdump tape made several years ago.

Every week we used to drop a copy of critical system files onto a bootstrap tape - jumpstart profiles, copies of systems configurations, basic software. (And we knew the directory trees were stable so we were happy to do the ufsdump live.) The theory was that in the event of a total disaster, we could get any old machine and put enough on it to be able to work well enough to reconstruct everything else. (Having needed to do this, it proved its worth.)

So the files were on this tape. I managed to find a DLT drive in the loft; managed to find a machine to hook it up to; managed to get the files off the tape.

(Finding a suitable machine was more tricky than I expected - I was a little surprised to go round the back of my W2100z - a SCSI based machine - and find no SCSI orifice. Fortunately I'm looking after a SunBlade 2000 for a colleague and that does. The other criterion was that I wanted to drop the files onto a USB connected zip disk, so having SCSI and USB in the same box was a help. I need to get that home network set up!)

So, problem solved. And it just goes to show that despite my reservations about tapes and old backup systems, they do actually come in handy once in a while.

The great thing about standards...

You know the old chestnut: The great thing about standards is that there are so many to choose from.

Now SCSI is a great standard. The truth is that SCSI devices do interoperate
incredibly well. At that level, it's been a phenomenal success.

But SCSI really covers a lot of things. The thing that hit me recently was connectors and cables.

I remember old SCSI cables with DB50 and centronics connectors on. I don't remember using them much - they were being phased out when I started playing this game. Then came the small 50-pin connector. Then the HD68, and more recently the VHDCI connector. Did I forget anything?

And then of course there was original SCSI, fast SCSI, fast wide SCSI, FWD, ultraSCSI, and ultra320 SCSI. And single ended plus two variants of differential.

So it's not actually that easy, given a system and a device, to say whether they will in fact interoperate. And then you need to find the right cable. (And then is the cable rated for ultraSCSI speeds?)

Which has bothered me for a while, but recently came to the fore after a bad incident caused a system to fail. So I decide to move the disks from a Netra T1 to a V240 and run the services in a zone. Easy, right? Well, not quite. The T1 has a HD68 orifice as I recall, and I think the V240 has a VHDCI outlet on the back!

So I go for a rummage in the loft and find the right cable. I'll send that across to the machines, and hopefully will be able to set the zone up and have the service back operational later this week.

The good, the bad, and the ugly

Let's start with the good:

We've now got SMBIOS integrated into OpenSolaris. I think it's the systems administrator in me showing, but I find this really exciting!

Now the bad:

I know I don't work for a living at the moment, but still keep an eye on some systems for former colleagues. And one of them suffered a power cut recently. Not just any power cut either, by the sounds of it. I've already alluded to one of the problems, and I had to roll back the SMF repository on another machine, but worse was to come.

Basically, it fried a Netra T1. I managed to persuade it to power on from the lom prompt, but it immediately starts vomiting errors. No system, no ok prompt, just errors. Looks like it can't even run POST. This looks like a dead system to me, beyond hope of repair. That's bad.

And the ugly:

SJVN has been emitting more mindless drivel. I see two choices here - either he's genuinely incapable of understanding licensing, or he's got an axe to grind and is using his journalistic position as a veneer of respect.

The article - at least the snide part about Sun - is simply plain wrong. Using CDDL as the license for OpenSolaris doesn't give Sun control. Exactly the opposite: with CDDL as the license Sun have less control of what people can do with OpenSolaris than they would have had if they had used the GPL.

And as for failing to build a significant programming community, well for one thing they already have gotten a major community, and for another thing getting a handful of part-timers to build Solaris on the cheap was never the driving force - Sun spent a huge amount of cash on building Solaris, and open-sourcing it, and it was done for sound business reasons - such as to open up new markets - not to outsource development. (Or kill the penguin, or introduce submarine patents into Linux, or any of the other stupid reasons naysayers put forward.) Of course, once the open source plan became known, a huge bunch of use started clamouring to get involved so we can modify and improve Solaris in the ways that we want.

Should I program to Tiger?

Simple question - when writing Java code, should I use the new features in Java 5.0, and thus not allow my code to work with older versions of Java?

There are, after all, some compelling reasons to do so.

In particular, I love Generics - I use Collections a lot, and being able to tell Java that the collection contains objects of a given (often fixed) type makes code so much cleaner. And again working with collections a lot, the enhanced for loop is a big win.

The question still remains - is backwards compatibility more important?

(I never found anything compelling in Java 1.4 to make breaking compatibility with 1.3 worthwhile, but the changes in 5.0 are a different matter.)

NISminus

I'm a great fan of NIS+. Not only is it very easy to set up and use, it scales well and it's dead easy to manage the data stored in it.

(Much easier than old NIS or LDAP. Sun could be suicidal and ditch it, as they've been threatening too, but replacing it with dramatically inferior solutions like NIS or LDAP is going to make a lot of people miserable.)

Of course, it doesn't always work perfectly. I just had this one case where it stopped working. The server partially booted - the NISplus server was running and serving clients. It's just that the NISplus client on the server wasn't working. So lots of other things wouldn't work. Amongst others, the NFS server. So while I could log in to a client, I couldn't access any files. Bad!

(Some testing indicates an RPC authentication problem. But basically everything goes into the RPC black box and never comes back.)

The only thing I had done was to change the IP address of the server. And yes, I had run nisupdkeys -a to update the server's addresses. So my theory is that somewhere buried deep in the bowels of NISplus is some memory of the machine's old IP address, and I can't spot it right now.

(I managed to get it working again, but I'm not sure which of the half a dozen random kludges was the one that mattered.)

Back to JNI

While still thinking about how to make progress with jkstat - basically, how closely to align it with the existing libjkstat implementation - I started another quick project to use JNI to get at Solaris system information from Java.

And, again, programming in JNI is hard going. The real problem is that it's astonishingly easy to make a mistake. And then it falls over with SIGSEGV and cores on me. Even the stack trace isn't too informative - it tells me that it's gone wrong in the native code (usually when calling back to java). As if I didn't know that! Even knowing where it blew up isn't that much help - it never tells you why. So you have to fix by inspection, and it's not the most obvious syntax.

(For example, it was blowing up at one point. Eventually I realized that I had a typo in the class name, but there was no indication in the failure that that might be the case.)

So what's the application this time? It's a java interface to PICL (the Solaris Platform Information and Control Library). The first aim is to produce a graphical picl browser (like my kstat browser). The prtpicl command is useful, and you get the information with -v, but it's not at all easy to browse through the many pages of output that can result. The picl data is tree structured, so the arrangement used in kstatbrowser works quite well - a Java JTree in the left panel to display the tree structure, and the actual data in a panel on the right.


Give me a day or two to work through the JNI errors I keep making and some code might be ready to release.

Keeping Busy

Having finished the last job, I'm at home for a while.

Not entirely idle, though. We've been to a theme park, I've started to sign on, and I've been looking at an odd little NFS problem for some former colleagues.

I've also been going through my CD collection. No, not the musical one! The pile of Sun CDs for Solaris, Java, and all the other bits of software that Sun have made that I've got the media for. After all, how many copies of the StarOffice 5.2 CD do you need? I brought my whole stock home, and have managed to prune it quite considerably.

My next project is to skim through all the sysadmin books I've got, reminding myself of what it used to be like, and what obscure things I'm supposed to know all about but nobody in the 21st century actually uses any more.

(On the first page of the first book, it tells me the key responsibilities of a Systems Administrator. High up on the list is formatting floppy diskettes. Say what?)

Submerged!

Been a bit busy lately. So much so that I've almost submerged out of sight.

With the closure of my place of work we've been working extremely hard to decommission a pretty significant computing service. It's been tough.

Our kit (aging but still serviceable) has been split up and sent off to a number of other departments - 1, 2, 3, 4, 5, 6 - not to mention some kind people who actually paid us for some of our older kit.

While our service has been shut down, it lives on in spirit.

Actually shutting down a large system cleanly was quite interesting. And, by and large, it did go down cleanly in an organised fashion, while being essentially functional right up to the bitter end. We've been shutting down and decommissioning kit steadily for weeks, and consolidating services onto fewer servers. Shutting a system down is the easy part - we've spend the last two days untangling the spaghetti of ten years worth of patch cables. One of my colleagues was still answering user support queries mid-afternoon yesterday - by early evening the system was down for good. This morning it got put into a lorry and the new owners should be receiving it soon. It just remains to send the last few bits of junk off to the local recycling company, hand in my keys, and put my feet up.

Missing the comforts of $HOME

I'm currently involved in a project with another organization to replicate large parts of our computing infrastructure on their systems.

It's been a while since I've had to use a system that I haven't personally specified and installed. All my systems are set up the way I want, matching the requirements of the applications, and work extremely well.

So it's a bit of a shock to be given an existing system and have to use that. At least it's running Solaris, so I don't have to port my code. But it feels wierd to go back to something prehistoric like NIS - we've used NISplus for over a decade - and some of the other design decisions like not using the automounter aren't decisions I would make myself. So I'm working in slightly unfamiliar territory, and it makes me realize how spoilt I am on my own systems.

(Moving some of our own application code over revealed some rather - ahem - strange implementation decisions. For example, I was fixing up a whole bunch of scripts today that had to construct the name of a user's home directory. Now, most of our users get their home directories automounted under /people, so the scripts refer to the home directory as /people/$USER. Oh dear! Why not use $HOME?)

But back to the comforts of $HOME. We've installed a wide range of useful software over the years - most of it a very long time ago now. Some of these tools come in extremely handy for certain tasks, and I've become used to having everything available - to the point that I forget that it doesn't come on a system as standard. Some of these things aren't very big, and I'll give you one small example of the sort of thing I'm talking about: rgrep. Just a recursive grep, you say. That's true - I could use grep, maybe allied with find, but rgrep is one of those little finishing touches that turns a bland computer system into my $HOME.