JKstat, SolView, Awards

I've released new versions of JKstat and SolView.

In JKstat, I've added Kstat aggregations. These are used in an enhanced cpustate demo to show the aggregate cpu statistics of a multithreaded core, or a multicore processor. This also needed me to work out how the various cpu kstats were related, so I knew which cpu corresponded to which thread and core of a complex multithreaded/multicore system. (A version of psrinfo naturally fell out of this as something I needed for testing.) There are a couple of new charts - for cpustate and the netload demo.

For SolView, I've added access to the logfiles for SMF services, and also a tree view of the SMF services. The tree is based on the service naming hierarchy, not the dependency tree, and is an experiment to see if that's a useful description of the services (as opposed to a straight list that's a couple of hundred lines long).

These are the entries I've submitted to the OpenSolaris Community Innovation Awards Program, and I'm hoping for some success there.

You don't exist, go away!

Oh dear. I try to run ssh and I get:

You don't exist, go away!

Which is sort of correct. I'm changing all the userids (including my own) while logged into the system, so that the shell I was running this from was under the old (now invalid) userid. Still, I was a little surprised at the bluntness of the response.

Making scp go a little quicker

Transferring files with scp isn't the quickest option, but if it's the only one there's a simple way to make it go a little quicker.

scp -c blowfish source.file remote.host:/destination/file.name

Using blowfish rather than the default 3des gave me about an extra 50% or so of throughput.

This was really noticeable on my new T5120s, where I went from under 10M/s to over 13M/s for a single copy. (OK, so it can probably run lots in parallel, but I was just moving large images.)

jingle and jumble

One of the problems with developing in Java is that some relatively common tasks that ought to be simple require writing a lot of boilerplate code, or are otherwise inconvenient.

So, like many others before me, I have a bunch of classes that I use regularly. These are not clever, innovative, or terribly interesting. But they have saved me a lot of typing and repetition over the years.

I named them jingle and jumble. (There were jangle and jungle, which I think had something to do with networking and web services; I've lost those completely.)

The jingle classes help write swing applications. One of the most useful ones is a Frame registry that keeps track of how many windows you have open and allows you to close one or all of them.

In jumble, it's a case of allowing you to get a file into a string (or vice-versa) in one line.

They're used to simplify jkstat and solview.

(I don't really expect others to use them, though. I'm blogging just to note their existence.)

Subtle change on sun.com

Don't know when this was changed, but I've noticed a subtle change on Sun's website recently.

I'm sure that in the main navigation, it had Products first and Downloads second. They seem to have swapped over.

I'm not sure whether this marks a change in direction, with less emphasis on selling stuff and more on giving stuff away, or whether they're tracking visitors and ordering the navigation links by popularity.

Networking vanished

I've had a faulty X2100M2 - it's been claiming for months that it's got a failed fan.

Originally it thought all the fans had failed (although I was somewhat suspicious because the temperatures looked fine). So I replaced them once and that made it happier, but not entirely happy.

So today we tried to replace the faulty fan again. No joy, so we ended up replacing the motherboard. And after fiddling with the cables, we finally persuaded the fault light to go out.

Unfortunately, we were only halfway home. Couldn't get a peep out of the system. Nothing on the serial port, nothing on the SP, couldn't even ping it.

One of the problems was that the replacement motherboard had old firmware, so things like serial port settings were up the creek. I had to play the upgrade game. At least that got me back to sanity, and I could see the system output as it boots.

Whoa there. Couldn't bring the network up. Originally, I had bge0 and bge1; on the new motherboard it decided to call the network interfaces bge2 and bge3. Ok, rename hostname.bge0 and we're back in business, but why on earth was it that hard just to get one of the system fans to work?

SAS vs. SATA

I use Solaris zones a lot.

We've got a number of X2200s, in two variants. Some just run web front ends, and are fitted with SATA drives (once running, the only disk activity is the web server logs); the database back-ends have SAS drives.

OK, so the SAS drives are expected to be a bit quicker - we did get them for that purpose. Based solely on the rotational speed, there's about a factor of 2 difference in performance.

However, if you take zone creation time as a metric, the performance difference is rather larger than a factor of 4. Something else makes the SAS drives fly and the SATA drives crawl.

An upgrade too far

I've been using Live Upgrade on my Solaris servers recently. Normally I would prefer a fresh install, as that gives you more of an opportunity to fix up any mistakes you made, but sometimes you need to preserve the application data or can't afford the downtime.

One word of warning, though: if you're starting with Solaris 8, you can go to Solaris 10 8/07 (update 4), but not to Solaris 10 5/08 (update 5). Even when upgrading from Solaris 9 or 10 you'll need the 7zip patches, but those don't exist (yet, anyway) for Solaris 8.

Sun does quad core Opterons

So Sun are now - finally - pushing quad core Opterons in the X4140, X4240, and X4440.

The X4240 is a new one. I like it. Yes, whereas I complained before, this one does have 16 internal drives.

Living in the Ghetto

Where I work is very much a pure Microsoft shop in terms of user environment - ie. desktops.

(The company makes its money using real Unix servers.)

I'm one of the very few who actually run Solaris on a Sun workstation. And, yes, sometimes I feel like I'm being pushed into a ghetto.

A world where you have to talk to Microsoft Exchange to read your mail, which means Outlook Web Access (which, frankly, is a shockingly poor attempt at being a mail client); where you're sent documents in Office 2007 format that you can't read; where half the company intranet simply doesn't function. Catering to those of living outside the walls simply isn't in Microsoft's world view, it would seem.

So, given the list of features, I'll have to grab the OpenOffice 3 beta and give it a try.

OpenSolaris in VirtualBox

I've been playing around today with VirtualBox, after finding that the latest version claims to run on Solaris 10.

Which is true, but you need to jump through a few hoops first.

First, you need to make sure that libGL.so can be found. I guess this varies a bit depending on whether you're using mesa or have the nividia drivers, but I ended up setting LD_LIBRARY_PATH_64 to /usr/X11/lib/amd64.

Then you need libXinerama; if you're running an older version of Solaris (my test machine was running S10U3 == 11/06) then applying patch 125726-02 will do the trick.

Then if you're running 64-bit you'll need a copy of libdlpi. I just snarfed a copy off one of my test opensolaris boxes (actually indiana preview 2 - I have seen comments that the one from the official OpenSolaris release won't work as it's too new).

(Yes, I realize there might be a bit of a chicken and egg situation there!)

Then I tried booting the indiana preview 1. Which worked just great. No network, but I expected that. The only glitch I had was the key to escape from the guest - which is set to Right Ctrl by default, which I discovered I don't have. I reset that to some other key that I do have and don't use for anything else.

Having learnt from that, I had a go at the OpenSolaris 2008.05 release having found a CD that I had brought back from CommunityOne, and that worked fine (and picked up the network).

Et tu, Brute?

So Jim's posted some photos of the OpenSolaris Developer Summit.

It was great to meet so many people in person. We had some good talks, lots of discussions (not all of which actually led to results, but you can't have everything). One of the things that is clear is that open communication is vital, and we definitely had some of that.

We're on the way to building a stronger community, and there was some additional encouragement organised for us. Go Team International!

Next up, Community One.

Controlling the Install Footprint

Solaris 10 introduced two technologies that, both individually and in concert, change the way we think about installing the OS.

Zones provide very lightweight virtualization. Especially sparse-root zones, which share most of the system files (binaries such as /usr) but have their own copy of configuration files and data (/etc and /var).

SMF, the Service Management Facility, replaces the old rc system and inetd, and allows you to configure exactly what services and daemons are running on a system.

The point about SMF is that historically, the only real way to guarantee that something was disabled was to not install it at all. With SMF, it is feasible to install something and rely on SMF to make sure it won't be active. So while minimization can be done the old way, you can use SMF to minimize what's running on a much larger install.

Then think about zones, and especially sparse-root zones. Because /usr is shared, you have to install everything that's going to be used throughout all the zones. So if you need a zone that runs tomcat, you have to install tomcat globally and it's there in all the zones. (Unless you install a private copy outside the OS in the zone that needs it.) And this is where SMF comes in - you can have all the applications everywhere, but only enable them in the zones that need them.

And it's not just the applications you have now that you need to worry about - to allow for future zones, you really need to make sure that you've installed everything that you might need in the future.

And if you wish to migrate zones between machines, then you have to install everything that might be needed by any zone on your network.

The old days of "install what you need, and no more" on each system don't apply. Consolidation using zones tends to drive you towards fat installs, using SMF to manage what's active in each zone.

That's fine - up to a point. And that point is where each zone gets overloaded with the services that it might run. One part of this is manifest import, which can be the dominant factor in sparse-root zone installation times. (Thankfully work to speed this up has recently been done!)

And as time goes on, the number of services is steadily proliferating. My desktop has over 200 lines of output from svcs. I know it doesn't do 200 different things.

So I have recently been looking again at my install profiles, to see what level of additional tweaking can be done. And the emphasis here is not just on minimizing what software is installed, but minimizing what services are installed, to reduce the cost of manifest import and reduce the available number of services that have to be managed.

On my (largely untweaked) desktop, there are currently 158 SMF manifests. Broken down into packages by number, that is:

   1 SUNWaccr
   1 SUNWadmr
   1 SUNWapmsc
   1 SUNWatfsr
   1 SUNWbindr
   1 SUNWbrgr
   1 SUNWcfplr
   1 SUNWcsd
   1 SUNWdhcsr
   1 SUNWfontconfig-root
   1 SUNWftpr
   1 SUNWgnome-display-mgr-root
   1 SUNWgssc
   1 SUNWinstall-patch-utils-root
   1 SUNWipmir
   1 SUNWipplr
   1 SUNWiscsir
   1 SUNWiscsitgtr
   1 SUNWjwnsr
   1 SUNWkdcr
   1 SUNWmconr
   1 SUNWntpr
   1 SUNWocfr
   1 SUNWos86r
   1 SUNWpcr
   1 SUNWpiclr
   1 SUNWpmr
   1 SUNWppror
   1 SUNWrcapr
   1 SUNWslpr
   1 SUNWsndmr
   1 SUNWsshdr
   1 SUNWstosreg
   1 SUNWstsfr
   1 SUNWtnamr
   1 SUNWtnetr
   1 SUNWtsr
   1 SUNWwbcor
   1 SUNWxwssu
   2 SUNWbsr
   2 SUNWnfssr
   2 SUNWpoolr
   2 SUNWpsr
   2 SUNWsacom
   2 SUNWservicetagr
   2 SUNWsmmgr
   2 SUNWvbox
   2 SUNWvolr
   2 SUNWzoner
   3 SUNWckr
   3 SUNWkrbr
   3 SUNWnisr
   3 SUNWtsg
   4 SUNWsmbar
   4 SUNWxwplr
   4 SUNWypr
   5 SUNWcnsr
   5 SUNWnfscr
   6 SUNWmdr
  10 SUNWrcmdr
  49 SUNWcsr

Now I can't uninstall SUNWcsr, but there are a lot of packages there that my servers aren't going to need. So I end up removing all the WBEM, smc, service tags, telnetd, tnamd, samba, core network services, and the rcmds help a bit too.

JKstat 0.21

It's been a while since I updated JKstat. In one sense, this slowing down is indicative that the project has reached a level of comparative maturity. Alternatively, I haven't found as much time to work on it as I would like. The reality is that both are true - the first phase of producing functional software is largely finished, and I haven't had time to address the more ambitious plans I have for the next phase.

Anyway, I'm putting out an updated version of JKstat now that has just a couple of fixes. One (an excellent suggestion from Tony Curtis) is that the network accessory shows the rates in humanized units (and I ought to implement this feature more widely). The second is a bunch of NullPointerExceptions raised by empty kstats. (Yes, I eventually stumbled across some kstats that had no data at all.)

NDIS Wrapper on Indiana

I've been setting up a laptop (a Dell D600, as it happens) so I thought I would try the Indiana preview on it.

Things have moved on a bit since I did this last - the onboard ethernet is recognised out of the box. But not the wireless. Off to try the NDIS Wrapper trick.

Now, I've got the Windows driver, which gives me the .inf and .sys files, and I downloaded the latest ndis package. Now to build it.

Unlike Solaris 10, where you type make and it works first time, Indiana has been cut down to fit onto the live CD and so you need some extra stuff. Several iterations of trial and error later, and the required steps are:

pfexec pkg install SUNWgcc
pfexec pkg install SUNWhea
pfexec pkg install SUNWflexlex
pfexec pkg install SUNWgm4

(Clearly, the dependency information for these packages is incomplete. Without gm4, flex gives an error message than couldn't ever be described as useful.)

With that, the "make ndiscvt" step works. Then you need to copy the .sys and .inf files into place and run

./ndiscvt -i ndis.inf -s ndis.sys -o ndis.h

if that fails with the error

ndiscvt: line 13: e: syntax error.

then the .inf file is in utf-16 format, so you need to

iconv -f utf-16 -t ascii path/to/bcmwl5.inf > ndis.inf

and run the ndiscvt step again.

Then it really is as simple as

make ndis
pfexec cp bcmndis /kernel/drv/bcmndis
make ndisapi
pfexec cp ndisapi /kernel/misc

Looking at "/usr/X11/bin/scanpci -v", we see

pci bus 0x0002 cardnum 0x03 function 0x00: vendor 0x14e4 device 0x4320
 Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller
 CardVendor 0x1028 card 0x0001 (Dell TrueMobile 1300 WLAN Mini-PCI Card)
  STATUS    0x0010  COMMAND 0x0106
  CLASS     0x02 0x80 0x00  REVISION 0x02
  BIST      0x00  HEADER 0x00  LATENCY 0x20  CACHE 0x00
  BASE0     0xfafee000  addr 0xfafee000  MEM
  MAX_LAT   0x00  MIN_GNT 0x00  INT_PIN 0x01  INT_LINE 0x0b
  BYTE_0    0x01  BYTE_1  0x00  BYTE_2  0xc2  BYTE_3  0xff

So we need to (see the numbers for vendor and device in the first line above)

add_drv -i '"pci14e4,4320"' bcmndis

and Yay!, it attaches.

Where's my Primary Administrator gone?

One neat aspect of Solaris is RBAC, which allows you to control which actions users can perform.

A particularly blunt instrument is the 'Primary Administrator' profile. If you're a Primary Administrator, then you are effectively root - in that you can use pfexec to assume the privileges of the root account (or role).

In Indiana, for example, root is (normally) a role and the account you create at install is set up as a Primary Administrator. It's very convenient.

So I decided to implement the same mechanism on my home machines (and use RBAC to let my children do extra things without having to pester dad).

Which failed, big time. It's really easy, you just use usermod to add a profile to an account:

usermod -P 'Primary Administrator' user_name

at which point Solaris thumbed its nose at me.

UX: usermod: ERROR: Primary Administrator is not a valid profile name.  Choose another.

I decided to dig a little deeper, and then you discover that the way these profiles find their way onto the system is (ahem) strange.

The profiles are defined in some files that live in /etc/security - auth_attr, exec_attr, and prof_attr - and then /etc/user_attr controls what is assigned to users. So where do these files come from?

It turns out that different packages stick their own entries in. If you start looking around the Solaris media, then go into Solaris_XX/Product and look for */reloc/etc/security/exec_attr (and the same for prof_attr and auth_attr). These are the files that get merged into the master copy by some funky class action script. (There are things about IPS that I don't agree with, but its plan of getting rid of all these way out scripts is something that has to be good.)

OK, so looking in those files, and the Primary Administrator is delivered by the SUNWwbcor package. What's that? "Solaris WBEM Services (root)".

No wonder I hadn't got the profile. I never install WBEM or anything to do with it. Systems are much better off without it (and I couldn't ever see myself installing it on something like a home system, or indeed any system where Primary Administrator might be used). But, if you don't install that package then you're going to have to install the profile yourself. Something like the following in the Product directory on the media should do it:

cat SUNWwbcor/reloc/etc/security/auth_attr >> /etc/security/auth_attr
cat SUNWwbcor/reloc/etc/security/exec_attr >> /etc/security/exec_attr
cat SUNWwbcor/reloc/etc/security/prof_attr >> /etc/security/prof_attr

From scary to useful

Much as I was taken aback by the native cifs client that is now in OpenSolaris, there is a serious side to this.

Integration with the Microsoft world is a huge deal. You just can't ignore it. (OK, Sun did try that approach for a while.)

As the "backup guy" I was just given a simple task. Copy some old files off to tape, archiving them so they can be deleted. Simple. Only they're on a Windows server.

But now it's real easy. Just mount the share, and tar to tape.

Normally I install each SXCE build onto an old Ultra 60 that's dedicated to testing. But for this task I could justify using something just a little more modern (and something that has a faster network interface so I don't have to wait forever). OK, so a Sun Blade 1500 isn't going to set the world on fire, but it's more than capable of doing the job. And it really was just a case of mounting the share and issuing a tar command.

Job done.

X4140 - X4440

I haven't seen the announcement from Sun, but just noticed the X4140 and X4440 appear.

They look just like Opteron versions of the X4150 and X4450.

(And yes, I think it's a shame you don't have the 16-drive option for the X4440.)

The case of the unloved files

I've been investigating a strange problem on one of my servers. It started out as a simple case of NetBackup failing to back the filesystem up.

Now that's not entirely unusual - NetBackup often goes off and sulks in a corner. But this was rather different, as it didn't disappear as mysteriously as it came. Rather, it stayed put and the filesystem repeatedly refused to complete a backup. And the diagnostics are pretty much non-existent.

OK, so after a week or so of this I decide to try an alternative approach. To my surprise, I couldn't blame NetBackup.

First attempt was to try ufsdump. It started off in a promising manner, then froze completely on me.

OK, that's not good. So I try various attempts at tar. So various tar commands, writing to a remote tape or filesystem. That would work, right? Wrong!

Each attempt freezes completely on me. That's local tar piped to tar that writes over nfs; tar piped to an rsh; tar on an nfs client; tar using rmt to write to a tape.

That's odd. Now, at least I can look at the output and see how far it's got. I'm starting to make headway, as it looks like it gets to the same point each time.

OK, so I start to build a copy by tarring up various bits of the filesystem, avoiding the place where I know it breaks. Until I get into the suspect area. And yes, it still fails in the same place (but at least I've got a copy of most of the filesystem now, so can breathe easier).

The bad area is in an area that has various versions (views of the data) of the index files of a proprietary search engine. Now it looks like tar always traverses the hierarchy in the same order. OK, so if I manually list the subdirectories in an order that puts the failed one last, I can copy off the files I'm missing, right?

That was a fine theory until it froze on me again. And this is where it gets really strange. Each subdirectory has the same structure. So in each subdirectory there's a bunch of files with different suffixes. And it always fails on one particular suffix. Furthermore, it fails at about the same distance (about 38 of 40 megabytes). That's about as weird as it gets, in my experience. What on earth is there about these files that causes anything that tries to back these files up to lock up completely?

And it gets worse. I can cp this file locally. Try cp to any nfs-mounted location and the cp wedges. An rcp to any remote system wedges. And it's the same again - it wedges at the same distance into the file.

It must be something in the data stream that's contained in these files. At least I did find one way to copy them across - if I gzip them first then they go across fine.

But where is the bug that's being tickled? Is this something in the network stack?

In my next job...

Now, I'm not looking for a job, as I'm very happy where I am, but if and when I do move on there's one hard question I'm going to ask any potential employer. And it's this:

How reliable is your air conditioning?

Because my servers were all sweating away in a sauna this morning when the A/C failed. Again.

We sorted it out and nothing went down as a result. And nothing has failed, yet. (I'm expecting a rash of disk failures over the next couple of weeks.)

I don't know what it is but wherever I've worked has had air conditioning problems. This should be mature technology, but seems ever so hard to get right.