T2000 vs bison

This wasn't a benchmark, or even meant as a test, but it was an interesting number nonetheless.

I installed bison today. As I have 3 different families of sytems (Sparc Solaris 8; Sparc Solaris 10; x86 Solaris 10) I have 3 copies of /usr/local, so build on 3 different systems. Anyway, after building bison (and the T2000 allows a very parallel make that some builds can take advantage of) I ran a make check.

The results were quite interesting. My sub $1000 Opteron 146 system took 107s. An old V880 (750MHz US-III) took 529s. The T2000 took a whopping 794s. This really isn't looking good.

Cheap sparcs and constrained configurations

I've recently been looking through the Sun server range, both to get some systems for present needs and to try and map out what is likely to meet our needs in the next year or two.

In some areas there are good matches and obvious choices. For example, theX2200 makes a phenomenal compute node, and the X4500 (aka thumper) is a phenomenal data storage machine.

But there are other areas where the choice is less obvious, or where what I actually want is a small variation to the configuration that Sun don't do. Here are some examples of the sort of things I want to do but that aren't available:

• A machine at the lower end of the range, such as the X2100, V210, or V125, with redundant power
• Much larger internal disk drive capacities, such as 300G drives in the V240 and similar, and 500G SATA drives in the T1000; these SAS drives really aren't helping
  
• Some new machines with full-height PCI slots so I can use my old PCI cards and connect old peripherals
• Something like an X2100 with 4 500G drive bays on the front (would make a great fileserver appliance)

In other words, more configuration flexibility, and less of the one size fits all mentality.

Things get especially problematic when you start looking at entry level sparc systems. Yes, application availability often means that we have to keep looking at sparc servers. And the available configurations aren't encouraging. The biggest issue is actually around disk, where the newer systems (T1000, T2000, V245, even the V445) are poorly provided for in terms of internal capacity and performance. In many ways, the V245 is a hugely retrograde step down from the V240. I really don't want to have to buy a SAN an FC cards to attach the machines to it, just to have enough space to load an application on them.

Problematic Printing

I've just migrated an important service off an old Solaris 8 machine onto a new(ish) server running Solaris 10.

Everything went pretty smoothly, with the exception of a few little printing problems. (I'm not a printer person, really. Whatever happened to the paperless office?)

I had just used lpadmin to set up the printers. I've read that printing in Solaris 10 is much better than it used to be (and that may be true) so that it can drive printers properly out of the box without the need for extra products like the HP JetDirect (what used to be JetAdmin) printer driver. Indeed, to the point that you're not supposed to use JetDirect any more.

In our testing we had ended up setting the type to any (rather than postscript) and using the netstandard model script. Anything more sophisticated and it mangled some of the printouts (specifically, we had some plaintext reports that had embedded characters to switch the printer to landscape that didn't work).

What we discovered today was that printer tray and media selection weren't working. I had been told that this was done in the application, but what that actually meant was that there was a wrapper shell script that called lp with some options that the jetdirect software can understand. And clearly the vanilla print system under Solaris doesn't grok those options.

So what I've had to do is to install the latest version of the JetDirect software that I can find (E10.34). And then add the specific model scripts for our model printers (and these are on the HP website, although they're a devil to find and the HP search is almost useless). And then using that to drive the printers made everything work flawlessly.

I'm sure there must be a way to pass the appropriate printer options to select the correct media type without running everything through JetDirect, but I've yet to find it. (Oh, and it would be nice if it used the same option names, to save having to track down every wrapper script that might be involved, although that's not absolutely necessary.)

Network Corruption?

I've just started seeing corrupted files on one of my X2100 M2 boxes. The visible indication is that I'm unpacking gzipped tar files from an NFS mounted disk, and was getting crc errors and invalid compressed data. Testing indicated that if I used rcp to transfer the files then I was also getting corruption. (I can check the md5 signatures, rather than just rely on gzcat to barf.) There wasn't any other evidence - I'm not seeing corruption of local files (basically, of Solaris itself), nor any network errors on the machine or the switch, and it's just the one machine that's affected.

How to test? SunVTS runs on Solaris x86, so I tried that out. Which wasn't encouraging:

INFO: Host System identified is "X2100 M2"
This Platform is not supported by SunVTS.

However, if I edit the platform.conf file it does at least run. And, so far, no memory or network errors.

Mind you, the errors I was seeing earlier seem to have disappeared...

Keeping the T2000 busy

Like many organizations, we use old computers for lightweight tasks - Sun systems just keep going. (Which can be a problem, as you can be tempted to keep them for too long when replacing them with a new cheap system would actually be a much better bet.)

So I'm putting together a couple of SunBlade 100s. They're more than powerful enough for the tasks in hand, and it saves cash. But building the software on them can take forever. Especially when you have to do it a few times before you get it right.

Enter the T2000. Type gmake -j 32 and boom! Build over.

If only ./configure could be parallelized...

T2000 - more performance

I've managed to do a couple more performance tests on my loan T2000.

the first was kicked by by the announcement that pbzip2 is available on sunfreeware. Now, bzip2 is pretty cpu intensive, so if there's a way of making it go faster then I'm all for it.

So here are some figures - in seconds - for pbzip2 as a function of the number of cpus used. I've used a small blocksize (100K) rather than the default, which helps as the test file isn't all that big.

# cpus	Time
1	16.444
2	8.686
4	4.287
8	2.791
16	1.821
32	1.784

That's pretty good. I would expect to get approximately a factor 8 gain here: it's a cpu intensive application and there are 8 physical cores. So we see about a factor 9, which is pretty good.

My next test was to load up an apache/php/mysql combo and see how that goes. The comparison machine is a SunBlade 2000 with a pair of 1.015GHz processors. (Note: this is using the same binaries - not optimised code on the T2000. Yes, Sun have optimised binaries, but that isn't really helpful in comparison, as with most applications a recompile wouldn't be feasible.)

For a simple database query operation, the T2000 took 0.160s as opposed to the 0.112s. For generating a graph, the T2000 was again slightly slower - 9.538s against 7.167s.

Again you see the single-threaded performance is at about the 60-70% of a regular UltraSparc cpu of the same clockspeed. But, this thing has 8 cores, how well does that work?

So I fired up the apache benchmark ab. And just for the database retrieval report I can get about 30 pages per second out of the SunBlade, but (with 32 concurrent requests) about 100 pages per second out of the T2000. And - according to top - the T2000 is only 50% busy. In fact, my server (apache/php/mysql) setup croaks if I push it much harder (but then it's a reporting thing designed for 1 page a minute).

So, on this test the T2000 is at least as good as 3 SunBlade 2000s. Or,equivalently, it's better than an 8x750MHz V880. Which is pretty good, as it's a fair amount cheaper and smaller.

Server Wars: The M2 Strikes Back

And I thought I was getting somewhere!

Having actually installed my X2100 M2 servers, I want to mirror the boot disk.

As an aside, I found Sun Infodoc 83605. This says:

It is recommended to use Hardware RAID controllers for the Root disk mirror, for performance and reliability reasons.

and then a little later, being consistent:

Sun Fire x2100 uses an nVidia RAID controller. Solaris OS doesn't support the nVidia driver as of now, so Solaris Volume Manager is the only option for mirroring the root disk.

OK, so I can't use hardware raid at the present time, which means (I think) that I can't hotswap the drives.

But anyway, I do the regular thing to mirror the boot disk with SVM. Works fine. I installgrub and use eeprom to define an alternate boot path. I reboot both servers. One comes back all fine and dandy.

The other one is dead. Faulty boot archive. Can't start console services. Remember that I'm having a wee bit of trouble controlling these machines, so getting to a failsafe system is a bit tricky. I fiddle for a bit, but rebooting doesn't help.

I was going to reinstall anyway, so pxe boot away. This time, I can't even see the console output. Oh dear, it's getting worse....

The M2 submits

As I have previously documented, I've been setting up a Sun X2100 M2.

It's not been the smoothest ride. And I wouldn't say that everything is completely solved, but having got it installed I followed the instructions for serial redirection and I can now ssh to the SP and get console access. That's probably all I need, although I still have a slight lack of confidence in the system - I can imagine ways in which it could get itself messed up where I get no access, but for most of the administration tasks I'll ever need to do I'm sorted.

Still can't get Serial-over-LAN to work, though. Never mind, ssh is good enough for me.

The M2 comes alive

I'm getting somewhere in my ongoing battle with the Sun X2100 M2.

I still haven't got the dratted thing to give me console access. But I have managed to install the beast in such a way that I can finish the install and log in.

The first step is to ensure that you have ipmitool installed. If you have Solaris 10, you may have it, and if so you need to patch it (119764 or 119765 according to your system). Then I can tell it to do a pxe boot next time around:

ipmitool -U root -I lanplus -H 1.2.3.4 chassis bootdev pxe

And I can power cycle it and it will do a jumpstart install.

Then in my finish script I have

sysidconfig -b /a -r /usr/openwin/bin/kdmconfig

which stops it asking me whether I want Xorg or Xsun at boot (questions which I can't answer because I have as yet found no way of typing back at it).

And I have a working system I can access using ssh. With luck I can get the installed system to play nice with the console, which is going to be essential when we move this box into production use.

Me versus the M2

I'm getting a very negative feeling about the SP on the X2100 M2 I'm trying to set up. This thing is just plain awful.

Yes, it's got some very fancy features. Basic functionality like being easily able to access the system console, and being able to operate for more than a few minutes without goofing up, or having documentation that is understandable, seem to be missing at this point.

The basic problem is that I can't get anything sensible out of the serial console. I can tip to the SP, and that works fine. But I just get junk output (if at all) on the console.

The KVM applet gadget is very nifty. And I can see output (once I've redirected to ttyb, anyway), but nothing I can do can persuade Solaris to accept keyboard input. It's just Solaris, as I can type into the GRUB screen OK. But if the install goes interactive I'm stuffed.

Anyway, I've got it installed by persuading it to jumpstart completely hands off. It insisted on asking for terminal type and locale before, although I'm not sure why. (Well, not completely - it complained about not being able to set the boot device and didn't reboot when it had finished installation, but that's OK, I can remote power-cycle it [and I've used that piece of functionality a few times today!].)

So my attempts here were to add:

-b console=ttyb

to the add_install_client invocation, and make sure that terminal and locale were defined in the sysidcfg file, and then edit the menu.lst file to add install as a argument - after:

kernel/unix

I added

 -v -m verbose install

Of course, I'm now stuck a little further on - looking at the kdmconfig screen where it's asking me to select an X server. Must find my notes about how to disable that prompt.

On to the X2100 M2

I've just been trying to set up one of the new X2100 M2s. I did a plain X2100 a while ago, and that was pretty simple - all I had to find out was the appropriate ctrl-alt-meta-shift-escape-thingy to replace F12 for the initial netboot while on the serial port. (Which I've forgotten, so if anyone could enlighten me - thanks!)

The new M2 is a different beast. The first problem was getting to the serial port. I've got my loan T2000 in the rack next to it, so I thought I would simply establish a tip session from that. However, it appears that the normal 'tip hardwire' trick doesn't work with the T2000 - there is no serial port B at all.

So I tried a different machine. And our console server. And about a dozen different cables. No joy. Eventually I did find a cable that worked, but I wasted far too long. (Don't get me started on serial cables. Every time I use them I feel this inner urge to go and throttle somebody.)

So I got an SP prompt. And this thing was - ahem - strange. After 5 minutes or so of being unable to get anything sensible out of it I punted and set the IP address manually, hooked the net management port into the network, and pointed my browser at it.

(I was somewhat dismayed to find that Sun's instructions told me to open up Internet Explorer. Oh how low have we fallen...)

So I point firefox at the LOM address and - wonder of wonders - it worked!

(There was the bit about the certificate expiring in 1979, and the mismatch between the IP address and the name on the certificate, but nothing serious.)

And you get what is actually quite a neat interface at this point. I started up the remote console, and nothing happened. Bother, edit pop-up preferences and try again. And I have the remote console.

This is all actually rather clever. It certainly looks good, and actually works pretty well too.

The next battle is to install the machine. At the moment when it tries to net boot one of our Windows Domain controllers jumps in and answers the DHCP request so my install server doesn't get a look in.

T2000 - initial performance

OK, so I've got the machine running so I thought I would try some simple performance tests.

I know that these aren't going to show the T2000 in a good light. These are simple CPU intensive single-threaded apps. (If you can call them applications.) The aim was to get a feel for just how well the machine would do.

So I have a twin 360MHz Ultra 60, a twin 1.5GHz V240, a quad 1.28GHz V440, an 8x1.0GHz T200, and a 500MHz SunBlade 100 and - for fun - a cheap X2100 with a 2.2GHz Opteron 148.

I copies the /var/sadm/install/contents file from my desktop into /tmp on each machine, and timed grep, wc, gzip, gunzip, bzip2, bunzip2 on the file (it's about 12 Meg). The times, in seconds, are:

U60:
grep    0.540
wc      0.615
gzip    3.055
gunzip  0.619
bzip2   25.589
bunzip2 4.187

SB100:
grep    0.416
wc      0.517
gzip    2.408
gunzip  0.492
bzip2   29.303
bunzip2 4.314

V240:
grep    0.136
wc      0.210
gzip    0.776
gunzip  0.152
bzip2   8.028
bunzip2 1.054

V440:
grep    0.159
wc      0.247
gzip    0.911
gunzip  0.180
bzip2   9.034
bunzip2 1.241

T2000:
grep    0.402
wc      0.656
gzip    2.772
gunzip  0.495
bzip2   17.695
bunzip2 2.285

X2100:
grep    0.079
wc      0.077
gzip    0.445
gunzip  0.092
bzip2   4.053
bunzip2 0.555

What's clear from this is the the Opteron (not entirely unexpectedly) wins by a distance. And the T2000 is handily outpaced by the V240 and V440 - even accounting for clock speed. In fact, the T2000 seems to be - for the completely unfair single tasking case - more comparable to the USIIe/USIIi in something like a V100 or Netra X1.

Of course, once you take into account the parallelism available, the T2000 might be comparable to a whole rack of the old 1U netra systems.

Now to see if some of our applications can be started up on this machine, and if we can test some applications that would suit the T2000 better.

T2000 - install performance

The first thing I did with my loan T2000 was to install Solaris on it. My first impression was that it seemed to be going fairly slowly, which was confirmed by some actual timing numbers.

I can time two parts of the install. The first is the actual Solaris installation, from the begin script to the finish script. This is only a part of the installation process, but is easy to measure. The second is my local installation, which takes place on the next boot, and installs some extra packages, runs some cleanup scripts, untars the whole of /opt/sfw, and applies current patches, and includes the reboot time. I've done 4 different systems this week, and the times (in minutes) are shown below:

Type	cpu speed	install	localinstall
Ultra 60	2x360MHz	45	73
V240	2x1.5GHz	16	40
V440	4x1.28GHz	18	34
T2000	8x1.0GHz	36	60

OK, so the install time isn't necessarily a good metric, but it's probably a fair indication of how long it's going to take to do general system administration on such a system. It's also essentially serial, which isn't good for the T1 chip. Even so, the numbers here are slightly disappointing - it's doing slightly worse for it's clock speed compared to the other sparc systems I've got available to play with today.

T2000 - past the first roadblock

Just got past the first roadblock with my loan T2000.

Plugged it in, powered it on, and got no network connectivity.

That's odd. All the lights look fine on both the T2000 and the switch. But I can't get the link to come up properly. I get

Timed out waiting for Autonegotation to complete
Check cable and try again
Link Down

So I pull it out of the foundry switch and into a cisco. And then I get

100 Mbps full duplex  Link up

OK, it's a shame that I can't get it working at gigabit but at least I can jumpstart it now, and I'll worry about getting it running at full speed later.

New Toys

I've been racking some new kit today. A couple of simple boxes for mail relays (and spam filters), and a backup server.

We've gone for the new Sun X2100 M2 for the mail systems. Something lightweight, but fast to handle the load. And very affordable at that.

We got an X2100 a while back, and it's been a huge success. We run our trouble ticket system on it, and it's not just faster - it's revolutionized the way we work. Before, doing anything with the trouble ticket system was so painful that it discouraged anyone from logging or handling tickets. Now, it just flies. We're hoping for the same from the mail system, which (and I guess everyone is getting the same experience) is getting swamped by the spam deluge.

The backup system is nothing special, I've put in a V240 with a C4 library. The only thing to say about this is that the C4 library is a deep beast - it barely fits into the Sun 900 rack and that's without being cabled up yet!

And I was just sitting down after that when a try-and-buy T2000 showed up. That's in the rack too, and it all looks very impressive.

We're interested to see how well the coolthreads system works. In general, I see increasing use of opteron systems, but we still have a number of sparc based applications that we're going to have to run. It stands a good chance - apache, tomcat, with a database or proprietary search engine at the back - so we're optimistic it'll be a success.

Solview Updated

I've updated my solview utility to version 0.3.

What is solview, I hear you ask? It's a little java gui to show useful information about a Solaris system. It shows installed software packages (and installation software clusters); services and their status; and the output from various informational commands.

This new version enhances the package and cluster information. It's split up into several tabbed panels, and now shows cluster membership, the installed status of clusters, and can calculate full dependencies.

What happened to version 0.2? Well, that version included a rather spectacular failure to enhance the software display by parsing the contents file. I expect it is possible to get java to extract useful information from this file without running out of memory, but I didn't succeed in doing so. So that attempt got scrapped and I moved on to version 0.3.

How hard can it be?

Yesterday I tried to write a CD on the family PC. I've been taking digital photos almost exclusively for the past couple of years, and we went through the accumulation and decided to get a few printed off properly. Simple - just burn them to a CD and take it to the nearest supermarket.

So the family PC has a writable CD drive, and a popular user-friendly operating system that's supposed to make this a breeze. So it should be a doddle, right? Wrong! I go to my photos and click where it says "Copy all items to CD", and failed completely to produce a CD with any pictures on. Several coasters, for sure, but no viable CDs. (My understanding of the way this is supposed to work is that it just copies the files to a holding cell, and when you open a writable CD it's supposed to ask you if you want to burn the CD. Nope, just trashed the CD every time.)

The PC has a number of utilities, that either came preinstalled, or as part of some other package, that allow you to write to a CD. None of those did the job either (I know the thing works, because I've written backups or VideoCDs, but I wanted something terribly simple. No dice.)

Unwilling to waste any more time, I simply copied the files to a real computer running Solaris. A couple of simple commands later and I had burnt a CD, first time, no problem.

Oh, and the pictures came out great.

Back in Circulation

I haven't disappeared. I haven't given up. I'm still alive.

The last month has been pretty hectic for me. It being summer and the kids having time off school, we took some holiday.

Also, as I mentioned earlier, I've changed jobs. I'm now working for Proquest Information and Learning, looking after their Sun Solaris servers. Half the first month was spent on holiday (arranged long before, and Proquest were happy with that - they were keen for me to make a start as soon as possible rather than wait until after my vacation). So far, I've been enjoying myself enormously. It's a big challenge and an environment rather different to what I've been used to, but I'm starting to get my teeth into it. The lifestyle is much better, and I've been cycling to work quite a lot (although, if you read the holiday story, it's not as if I have much choice at the moment).

Self Preservation

It's been unpleasantly warm here in the UK over the last month. I'm sure other places have it hotter, but we're not used to it and not geared up for it.

I have some old work colleagues who I set up some systems for, and I got an urgent phone call just before going on holiday: "the air conditioning failed yesterday and the power in the data centre was unceremoniously yanked - could you check the systems out?".

It turns out the servers - Sun machines - were fine. With a fine sense of self preservation they had shut themselves down gracefully due to the heat:

Jul 19 18:00:14 foobar rmclomv: [ID 690426 kern.crit]
  SC initiating soft host system shutdown due to fault at MB.T_ENC.
Jul 19 18:00:15 foobar rmclomv: [ID 222544 kern.error]
  SC Request to Power Off Host.
Jul 19 18:00:16 foobar rmclomv: [ID 428258 kern.crit]
  TEMP_SENSOR @ MB.T_ENC has exceeded high soft shutdown threshold.

So far, all appears to be well, and I haven't seen any further errors.

Enhancing the JDS desktop look

While the JDS desktop that currently comes with Solaris is essentially functional, I've always found it rather bland. In one sense it's a good thing - it has no major irritations (most window managers or desktop environments have something that drives me up the wall).

Still, I would like to get a bit more character out of the desktop. Unfortunately, the available themes are pretty thin on the ground. Unlike the huge array of themes that you can get for Window Maker, you're really very restricted.

I actually quite liked the new theme that the latest JDS builds come with, but it doesn't seem to be separately available.

I've found a couple of themes that I like. I've used eXPerience for most of the last year. I've got my Windows XP box set to the olive green theme, and find that color combination works quite well for me. And I've just been playing with Clear Blue, which works pretty well. It's a bit pale on my current monitor, and I might have to tone down some of the other applications I use to blend in a little better, but it's looking good.

There's still a lot of room for more gorgeous looking desktop themes, though.