A new use for a D1000 array

I found a new use for a D1000 array last night.

We bought a pallet-load off eBay a while back. We only actually needed one, but the whole pallet was no more expensive. So the first one got used for the purpose for which it was bought, and I've used a couple more filled up with old drives to test ZFS and the like.

We just piled the rest up in a store-room.

Which is where they had stayed for a few weeks harmlessly gathering dust.

Last night I went in there to look for some cables, and was somewhat annoyed to find that the door wouldn't open when I tried to get out. The lock release did absolutely nothing.

What you have to understand is that this was an old comms or server room. So that's a security and fire door in my way. No windows. No phone. My mobile is on my desk. It's almost soundproof as well, and it's getting late.

I did manage to attract someone's attention as they were passing in the corridor, and we tried all sorts of tricks to get the stupid door to open, with no success.

Eventually I piled all the D1000s up and stood on top of them. (Which just goes to show that buying quality kit is always worthwhile. You try this trick with cheap kit and it will fold under the weight.) Originally I was hoping to get into the false ceiling and work my way through, but even that was a bit of a stretch and very cluttered.

Eventually we took out the panel above the door and I got out that way.

See, I knew this old kit would come in useful some day!

Updated jkstat

It's been a while (almost a year) since I last updated jkstat, but I've just put version 0.13 up for download.

For those that don't know, jkstat is a Java jni interface to Solaris kstats, allowing you to access a wide range of useful data held by the Solaris kernel from a Java application.

One thing I have done is this version is try to improve the look of the sample applications, most of which has been achieved by replacing the rather clumsy GridLayout with SpringLayout.

Impressed with Xfce

I've been having a play with Xfce on one of my Solaris workstations.

And I have to say I've been very impressed. It's reasonably fast and light. It has an increasing number of applications. The file manager - Thunar - is quite nice.

There are two things that I think are in favour of Xfce. The first is that because of its common roots with gnome it can leverage all the gnome applications and GTK themes. The second is that it is very polished. There is a good section of window manager themes, and the artwork is very well done.

All in all it's well worth a look.

Sun, Intel - friends?

OK, so Sun and Intel made a big fuss about some deal or other (one which doesn't really set the pulse racing, mind). What I found ironic was the ticker underneath the main feature on the Sun home page, where they put the knife into Itanium.

I remember Sun recently selling a number of Intel powered systems. The LX50 was a nice box - even if it was pushed with SunLinux. The V60x was OK, but I quite liked the V65x, and that sort of system (6 internal drives) is something that would be quite valuable. And the B200x blade was quite a marvel - twin hyperthreaded Xeons and a decent memory configuration in a tiny package.

I expect closer collaboration will improve driver support for Solaris, but I'm also hoping for a wider range of systems designed for Solaris to give us a breadth and variety of system configurations that Sun themselves seem disinclined to supply.

Backup Blues

Taking system backups is an essential part of any IT operation, but seems to involve an awful lot of pain.

Over the years I've generally been happy with Legato (now EMC) NetWorker, sometimes known as Sun StorEdge Backup or something similar. What I really like about it is that it's very low maintenance - easy to set up and configure, and gets on with its job quietly in the background without any fuss.

I was a little concerned that the latest version has dropped the old nwadmin X-windows based GUI in favour of some Java client-server thingummy. I still don't really like the java interface, but as I previously mentioned it's pretty low maintenance so you don't need to spend much time using it, and it's generally easy enough to use. The one downside to it is that the licensing is a bit more complicated, and the license registration instructions explicitly refer to the old GUI.

I'm currently using NetWorker for one half of my backups. I put in a new server and tape library and moved the NetWorker license across. The transition has been very successful, although the primary win is that I replaced an L20 with 2 DLT-7000 drives with a C4 with 2 LTO3 drives. The old system simply didn't have the capacity or the performance - backups would take days, and some backup runs came in at over twice the capacity of the library. Now they take hours and fit onto a couple of tapes.

The other half of my backup story is more appropriate for the tile of this post. There, we're using an alternative product. OK, specifically, NetBackup. And it's much more trouble than NetWorker has ever been.

I'm trying hard to give a balanced view here. After all, NetBackup is widely used and must have some good points, right? Maybe...

The first thing is that NetBackup is very high maintenance, at least compared to NetWorker. It needs constant attention paid to it, seeming to go off and hide in a corner if you don't give it constant encouragement. I'm not surprised that large organizations have dedicated NetBackup administrators, or even teams thereof.

The second thing is that I (and I know I'm not the only one) have trouble with the way you actually define backup schedules. In NetWorker it's trivially easy - you define what you want backed up, what level of backups you want, and when to do it. In NetBackup you mess around with backup windows, frequencies, policies, and schedules. Again, this is much higher maintenance - I usually need to manually configure half a dozen screens in NetBackup as opposed to simply selecting one option under NetWorker.

The third thing is reliability. The only time I have known NetWorker fail to do a backup is if I've run out of tape. NetBackup seems to throw fits all over the place. We see regular timeouts, errors when it's outside the backup window, and occasional (once or twice a month) where the whole thing freezes up and needs a forceful restart. I've just upgraded the NetBackup server itself, and after tuning got acceptable performance (the default performance was dreadful), but the catalog backup takes over a day for something that ought to take 20 minutes, and renders the machine completely unresponsive while it's at it.

I'm tempted to replace the NetBackup installation with NetWorker, which raises a number of issues. I'm happy that it will be fairly painless on the Unix side, but I'm less sure about the Windows side which also needs backing up. While I have run NetWorker on Windows, it became apparent fairly quickly that it was basically a Unix product and looked to be in alien territory. And my experience of NetWorker's application modules (as needed for the likes of databases, exchange and the like) hasn't been entirely positive. So I'm thinking of a completely separate system for the Winddows backups.

I've tested a number of other backup products, but under Unix. And there it was obvious that most of the Windows products looked completely lost under Unix. I haven't really come across a truly cross-platform backup solution, and it's not obvious to me that such a thing actually exists.

Given how conceptually simple backups are, is it asking too much to have solutions that just work?

Fun with NVIDIA driver

I have a Sun W2100z running Solaris. It's a fantastic machine, but the graphics has never been too hot. It's got the entry level graphics card - an NVS 280 if I remember correctly - and I've been using the bundled nv driver as I wasn't aware that the accelerated driver could do much better.

I was trying to get DVD playback to work, and gxine seems to work (after I had built and installed libdvdcss), but was rather slow and I noticed that Xorg seemed to be pegging a whole cpu. In addition, it might be neat to get Looking Glass to work.

So I ambled over to NVIDIA and their Solaris Driver downloads, and noticed that my graphics card was listed as one of those supported. I downloaded and installed the latest version (9764), and that didn't work. I get an error message as it boots that seems to be from the nvidia driver, but then the system immediately reboots. Not good. (I went into failsafe from the grub menu and used pkgrm to delete the two packages to recover.)

Then I had a look in the forums and noticed a message about a legacy release (9631). Now, it doesn't really explain what the legacy release is in this case (the information on the download page about legacy releases applies to Linux), and there isn't a link to the legacy release on the download page itself, but to cut a long story short - the 9631 legacy release works fine on my entry level box. DVD playback is a bit smoother, although still not brilliant, but the cpu load is pretty negligible compared to before.

If you, like me, have an entry-level W2100z with an NVS280 graphics card, then you may be advised to use the NVIDIA legacy driver rather than the latest and greatest.

Jumpstart Profile Builder

One of my little projects is to write an interactive jumpstart profile builder, so that you can play with Solaris software selections, and construct a working jumpstart profile.

The problem currently is that you can, with some effort, create a jumpstart profile. But the only real way to test it is to use it to install a system, and either see what works and what doesn't, or look at the installation log to see what package dependencies you messed up on. Eventually you can get quite good at this, but it shouldn't be this hard.

You can select packages in the interactive install, but the dependency resolution is awful. It tells you what the dependencies that failed are, but doesn't give you a way to feed that back into the gui to resolve them. Nor does it tell you what dependencies are required as you look at the individual packages. And once you've selected your software, it will install it but you can't go from that to create a jumpstart profile to automatically install the next set of systems.

I have most of the base code already, in the shape of Solview. That knows how to turn the .clustertoc file (which is where the clusters and metaclusters are defined) into a tree representation of the packages, that could be used as the basis of a selection tree. And it can evaluate the dependencies. I've also got code that can describe a jumpstart profile.

Where I'm currently struggling a little is the actual interface. Using a java swing JTree as the base is fairly straightforward - this is what Solview does already. And I need a custom cell renderer to add checkboxes to show whether the software package is selected or not (or, in the case of an installed system, whether it's been installed or not). But then I got to thinking and I can't use the regular JCheckBox, as there are more states that I wish to display than JCheckBox gives me out of the box.

• Package or cluster selected
• Package or cluster not selected
• Cluster partially selected
• Package or cluster selected and cannot be deselected

The last case is for required packages and clusters (which you can actually delete in a jumpstart profile but which it ignores), and I could probably do simply enough by having the checkbox selected but disabled. I might also want to have a separate state for packages and clusters which have forward or reverse dependencies, so it's clear visually whether it's safe to select or deselect them.

I've had a quick look around with the help of google, and I'm obviously not the first person to have requirements of this type. And there is a fairly large amount of sample code out there. Unfortunately it appears to be fairly complex, and I would be adding a significant amount of extra code, but it doesn't appear that I have much choice.

T2000 vs bison

This wasn't a benchmark, or even meant as a test, but it was an interesting number nonetheless.

I installed bison today. As I have 3 different families of sytems (Sparc Solaris 8; Sparc Solaris 10; x86 Solaris 10) I have 3 copies of /usr/local, so build on 3 different systems. Anyway, after building bison (and the T2000 allows a very parallel make that some builds can take advantage of) I ran a make check.

The results were quite interesting. My sub $1000 Opteron 146 system took 107s. An old V880 (750MHz US-III) took 529s. The T2000 took a whopping 794s. This really isn't looking good.

Cheap sparcs and constrained configurations

I've recently been looking through the Sun server range, both to get some systems for present needs and to try and map out what is likely to meet our needs in the next year or two.

In some areas there are good matches and obvious choices. For example, theX2200 makes a phenomenal compute node, and the X4500 (aka thumper) is a phenomenal data storage machine.

But there are other areas where the choice is less obvious, or where what I actually want is a small variation to the configuration that Sun don't do. Here are some examples of the sort of things I want to do but that aren't available:

• A machine at the lower end of the range, such as the X2100, V210, or V125, with redundant power
• Much larger internal disk drive capacities, such as 300G drives in the V240 and similar, and 500G SATA drives in the T1000; these SAS drives really aren't helping
  
• Some new machines with full-height PCI slots so I can use my old PCI cards and connect old peripherals
• Something like an X2100 with 4 500G drive bays on the front (would make a great fileserver appliance)

In other words, more configuration flexibility, and less of the one size fits all mentality.

Things get especially problematic when you start looking at entry level sparc systems. Yes, application availability often means that we have to keep looking at sparc servers. And the available configurations aren't encouraging. The biggest issue is actually around disk, where the newer systems (T1000, T2000, V245, even the V445) are poorly provided for in terms of internal capacity and performance. In many ways, the V245 is a hugely retrograde step down from the V240. I really don't want to have to buy a SAN an FC cards to attach the machines to it, just to have enough space to load an application on them.

Problematic Printing

I've just migrated an important service off an old Solaris 8 machine onto a new(ish) server running Solaris 10.

Everything went pretty smoothly, with the exception of a few little printing problems. (I'm not a printer person, really. Whatever happened to the paperless office?)

I had just used lpadmin to set up the printers. I've read that printing in Solaris 10 is much better than it used to be (and that may be true) so that it can drive printers properly out of the box without the need for extra products like the HP JetDirect (what used to be JetAdmin) printer driver. Indeed, to the point that you're not supposed to use JetDirect any more.

In our testing we had ended up setting the type to any (rather than postscript) and using the netstandard model script. Anything more sophisticated and it mangled some of the printouts (specifically, we had some plaintext reports that had embedded characters to switch the printer to landscape that didn't work).

What we discovered today was that printer tray and media selection weren't working. I had been told that this was done in the application, but what that actually meant was that there was a wrapper shell script that called lp with some options that the jetdirect software can understand. And clearly the vanilla print system under Solaris doesn't grok those options.

So what I've had to do is to install the latest version of the JetDirect software that I can find (E10.34). And then add the specific model scripts for our model printers (and these are on the HP website, although they're a devil to find and the HP search is almost useless). And then using that to drive the printers made everything work flawlessly.

I'm sure there must be a way to pass the appropriate printer options to select the correct media type without running everything through JetDirect, but I've yet to find it. (Oh, and it would be nice if it used the same option names, to save having to track down every wrapper script that might be involved, although that's not absolutely necessary.)

Network Corruption?

I've just started seeing corrupted files on one of my X2100 M2 boxes. The visible indication is that I'm unpacking gzipped tar files from an NFS mounted disk, and was getting crc errors and invalid compressed data. Testing indicated that if I used rcp to transfer the files then I was also getting corruption. (I can check the md5 signatures, rather than just rely on gzcat to barf.) There wasn't any other evidence - I'm not seeing corruption of local files (basically, of Solaris itself), nor any network errors on the machine or the switch, and it's just the one machine that's affected.

How to test? SunVTS runs on Solaris x86, so I tried that out. Which wasn't encouraging:

INFO: Host System identified is "X2100 M2"
This Platform is not supported by SunVTS.

However, if I edit the platform.conf file it does at least run. And, so far, no memory or network errors.

Mind you, the errors I was seeing earlier seem to have disappeared...

Keeping the T2000 busy

Like many organizations, we use old computers for lightweight tasks - Sun systems just keep going. (Which can be a problem, as you can be tempted to keep them for too long when replacing them with a new cheap system would actually be a much better bet.)

So I'm putting together a couple of SunBlade 100s. They're more than powerful enough for the tasks in hand, and it saves cash. But building the software on them can take forever. Especially when you have to do it a few times before you get it right.

Enter the T2000. Type gmake -j 32 and boom! Build over.

If only ./configure could be parallelized...

T2000 - more performance

I've managed to do a couple more performance tests on my loan T2000.

the first was kicked by by the announcement that pbzip2 is available on sunfreeware. Now, bzip2 is pretty cpu intensive, so if there's a way of making it go faster then I'm all for it.

So here are some figures - in seconds - for pbzip2 as a function of the number of cpus used. I've used a small blocksize (100K) rather than the default, which helps as the test file isn't all that big.

# cpus	Time
1	16.444
2	8.686
4	4.287
8	2.791
16	1.821
32	1.784

That's pretty good. I would expect to get approximately a factor 8 gain here: it's a cpu intensive application and there are 8 physical cores. So we see about a factor 9, which is pretty good.

My next test was to load up an apache/php/mysql combo and see how that goes. The comparison machine is a SunBlade 2000 with a pair of 1.015GHz processors. (Note: this is using the same binaries - not optimised code on the T2000. Yes, Sun have optimised binaries, but that isn't really helpful in comparison, as with most applications a recompile wouldn't be feasible.)

For a simple database query operation, the T2000 took 0.160s as opposed to the 0.112s. For generating a graph, the T2000 was again slightly slower - 9.538s against 7.167s.

Again you see the single-threaded performance is at about the 60-70% of a regular UltraSparc cpu of the same clockspeed. But, this thing has 8 cores, how well does that work?

So I fired up the apache benchmark ab. And just for the database retrieval report I can get about 30 pages per second out of the SunBlade, but (with 32 concurrent requests) about 100 pages per second out of the T2000. And - according to top - the T2000 is only 50% busy. In fact, my server (apache/php/mysql) setup croaks if I push it much harder (but then it's a reporting thing designed for 1 page a minute).

So, on this test the T2000 is at least as good as 3 SunBlade 2000s. Or,equivalently, it's better than an 8x750MHz V880. Which is pretty good, as it's a fair amount cheaper and smaller.

Server Wars: The M2 Strikes Back

And I thought I was getting somewhere!

Having actually installed my X2100 M2 servers, I want to mirror the boot disk.

As an aside, I found Sun Infodoc 83605. This says:

It is recommended to use Hardware RAID controllers for the Root disk mirror, for performance and reliability reasons.

and then a little later, being consistent:

Sun Fire x2100 uses an nVidia RAID controller. Solaris OS doesn't support the nVidia driver as of now, so Solaris Volume Manager is the only option for mirroring the root disk.

OK, so I can't use hardware raid at the present time, which means (I think) that I can't hotswap the drives.

But anyway, I do the regular thing to mirror the boot disk with SVM. Works fine. I installgrub and use eeprom to define an alternate boot path. I reboot both servers. One comes back all fine and dandy.

The other one is dead. Faulty boot archive. Can't start console services. Remember that I'm having a wee bit of trouble controlling these machines, so getting to a failsafe system is a bit tricky. I fiddle for a bit, but rebooting doesn't help.

I was going to reinstall anyway, so pxe boot away. This time, I can't even see the console output. Oh dear, it's getting worse....

The M2 submits

As I have previously documented, I've been setting up a Sun X2100 M2.

It's not been the smoothest ride. And I wouldn't say that everything is completely solved, but having got it installed I followed the instructions for serial redirection and I can now ssh to the SP and get console access. That's probably all I need, although I still have a slight lack of confidence in the system - I can imagine ways in which it could get itself messed up where I get no access, but for most of the administration tasks I'll ever need to do I'm sorted.

Still can't get Serial-over-LAN to work, though. Never mind, ssh is good enough for me.

The M2 comes alive

I'm getting somewhere in my ongoing battle with the Sun X2100 M2.

I still haven't got the dratted thing to give me console access. But I have managed to install the beast in such a way that I can finish the install and log in.

The first step is to ensure that you have ipmitool installed. If you have Solaris 10, you may have it, and if so you need to patch it (119764 or 119765 according to your system). Then I can tell it to do a pxe boot next time around:

ipmitool -U root -I lanplus -H 1.2.3.4 chassis bootdev pxe

And I can power cycle it and it will do a jumpstart install.

Then in my finish script I have

sysidconfig -b /a -r /usr/openwin/bin/kdmconfig

which stops it asking me whether I want Xorg or Xsun at boot (questions which I can't answer because I have as yet found no way of typing back at it).

And I have a working system I can access using ssh. With luck I can get the installed system to play nice with the console, which is going to be essential when we move this box into production use.

Me versus the M2

I'm getting a very negative feeling about the SP on the X2100 M2 I'm trying to set up. This thing is just plain awful.

Yes, it's got some very fancy features. Basic functionality like being easily able to access the system console, and being able to operate for more than a few minutes without goofing up, or having documentation that is understandable, seem to be missing at this point.

The basic problem is that I can't get anything sensible out of the serial console. I can tip to the SP, and that works fine. But I just get junk output (if at all) on the console.

The KVM applet gadget is very nifty. And I can see output (once I've redirected to ttyb, anyway), but nothing I can do can persuade Solaris to accept keyboard input. It's just Solaris, as I can type into the GRUB screen OK. But if the install goes interactive I'm stuffed.

Anyway, I've got it installed by persuading it to jumpstart completely hands off. It insisted on asking for terminal type and locale before, although I'm not sure why. (Well, not completely - it complained about not being able to set the boot device and didn't reboot when it had finished installation, but that's OK, I can remote power-cycle it [and I've used that piece of functionality a few times today!].)

So my attempts here were to add:

-b console=ttyb

to the add_install_client invocation, and make sure that terminal and locale were defined in the sysidcfg file, and then edit the menu.lst file to add install as a argument - after:

kernel/unix

I added

 -v -m verbose install

Of course, I'm now stuck a little further on - looking at the kdmconfig screen where it's asking me to select an X server. Must find my notes about how to disable that prompt.

On to the X2100 M2

I've just been trying to set up one of the new X2100 M2s. I did a plain X2100 a while ago, and that was pretty simple - all I had to find out was the appropriate ctrl-alt-meta-shift-escape-thingy to replace F12 for the initial netboot while on the serial port. (Which I've forgotten, so if anyone could enlighten me - thanks!)

The new M2 is a different beast. The first problem was getting to the serial port. I've got my loan T2000 in the rack next to it, so I thought I would simply establish a tip session from that. However, it appears that the normal 'tip hardwire' trick doesn't work with the T2000 - there is no serial port B at all.

So I tried a different machine. And our console server. And about a dozen different cables. No joy. Eventually I did find a cable that worked, but I wasted far too long. (Don't get me started on serial cables. Every time I use them I feel this inner urge to go and throttle somebody.)

So I got an SP prompt. And this thing was - ahem - strange. After 5 minutes or so of being unable to get anything sensible out of it I punted and set the IP address manually, hooked the net management port into the network, and pointed my browser at it.

(I was somewhat dismayed to find that Sun's instructions told me to open up Internet Explorer. Oh how low have we fallen...)

So I point firefox at the LOM address and - wonder of wonders - it worked!

(There was the bit about the certificate expiring in 1979, and the mismatch between the IP address and the name on the certificate, but nothing serious.)

And you get what is actually quite a neat interface at this point. I started up the remote console, and nothing happened. Bother, edit pop-up preferences and try again. And I have the remote console.

This is all actually rather clever. It certainly looks good, and actually works pretty well too.

The next battle is to install the machine. At the moment when it tries to net boot one of our Windows Domain controllers jumps in and answers the DHCP request so my install server doesn't get a look in.

T2000 - initial performance

OK, so I've got the machine running so I thought I would try some simple performance tests.

I know that these aren't going to show the T2000 in a good light. These are simple CPU intensive single-threaded apps. (If you can call them applications.) The aim was to get a feel for just how well the machine would do.

So I have a twin 360MHz Ultra 60, a twin 1.5GHz V240, a quad 1.28GHz V440, an 8x1.0GHz T200, and a 500MHz SunBlade 100 and - for fun - a cheap X2100 with a 2.2GHz Opteron 148.

I copies the /var/sadm/install/contents file from my desktop into /tmp on each machine, and timed grep, wc, gzip, gunzip, bzip2, bunzip2 on the file (it's about 12 Meg). The times, in seconds, are:

U60:
grep    0.540
wc      0.615
gzip    3.055
gunzip  0.619
bzip2   25.589
bunzip2 4.187

SB100:
grep    0.416
wc      0.517
gzip    2.408
gunzip  0.492
bzip2   29.303
bunzip2 4.314

V240:
grep    0.136
wc      0.210
gzip    0.776
gunzip  0.152
bzip2   8.028
bunzip2 1.054

V440:
grep    0.159
wc      0.247
gzip    0.911
gunzip  0.180
bzip2   9.034
bunzip2 1.241

T2000:
grep    0.402
wc      0.656
gzip    2.772
gunzip  0.495
bzip2   17.695
bunzip2 2.285

X2100:
grep    0.079
wc      0.077
gzip    0.445
gunzip  0.092
bzip2   4.053
bunzip2 0.555

What's clear from this is the the Opteron (not entirely unexpectedly) wins by a distance. And the T2000 is handily outpaced by the V240 and V440 - even accounting for clock speed. In fact, the T2000 seems to be - for the completely unfair single tasking case - more comparable to the USIIe/USIIi in something like a V100 or Netra X1.

Of course, once you take into account the parallelism available, the T2000 might be comparable to a whole rack of the old 1U netra systems.

Now to see if some of our applications can be started up on this machine, and if we can test some applications that would suit the T2000 better.

T2000 - install performance

The first thing I did with my loan T2000 was to install Solaris on it. My first impression was that it seemed to be going fairly slowly, which was confirmed by some actual timing numbers.

I can time two parts of the install. The first is the actual Solaris installation, from the begin script to the finish script. This is only a part of the installation process, but is easy to measure. The second is my local installation, which takes place on the next boot, and installs some extra packages, runs some cleanup scripts, untars the whole of /opt/sfw, and applies current patches, and includes the reboot time. I've done 4 different systems this week, and the times (in minutes) are shown below:

Type	cpu speed	install	localinstall
Ultra 60	2x360MHz	45	73
V240	2x1.5GHz	16	40
V440	4x1.28GHz	18	34
T2000	8x1.0GHz	36	60

OK, so the install time isn't necessarily a good metric, but it's probably a fair indication of how long it's going to take to do general system administration on such a system. It's also essentially serial, which isn't good for the T1 chip. Even so, the numbers here are slightly disappointing - it's doing slightly worse for it's clock speed compared to the other sparc systems I've got available to play with today.