Monday, March 17, 2008

OGB: constitution

Work in the OpenSolaris project is done under a constitution, which lays out how the community is structured.

Sort of, anyway. The constitution covers both the Governing Board and Community Groups, but in fact most of the activity around OpenSolaris takes place in User Groups (where people get together) and Projects (where code is written), and also in Communities of Interest (where technologies get discussed). None of these last 3 parts of the organization are defined by the constitution.

Indeed, they are hardly mentioned. Even though community groups are tasked by the constitution to initiate and manage projects to achieve their objectives, it isn't clear (because projects aren't defined) whether the projects established by a Community Group are the same as the Projects hosted on OpenSolaris.org, or whether project is used in the generic sense.

As for user groups, they are managed as projects. While I believe this to be correct in the sense that they can reuse the machinery and infrastructure, they should explicitly be called User Groups and given their own place in OpenSolaris as first class citizens.

And then we have Communities. Originally, when OpenSolaris came about, a bunch of communities were created. And these were Communities of Interest - focussed around a general technical area (for example, performance), or a specific technology (dtrace). Then the constitution came along, and created the notion of Community Groups. As part of the bootstrapping process, many of the original Communities became Community Groups, while some did not. The whole thing is a total mess, and part of that mess is overloading the notion of a community to mean two different - and often incompatible - things. I think we need to find a way to clearly separate the mechanism of governance from the day to day interactions of users in the community.

And then there's the amount of effort wasted by the current structures. Creating a project means you have to persuade a community group to authorise it (and not all communities can, remember, and of those that can persuading the machinery to work at all and getting the approval can be a painful and time-wasting process). Creating a community involves redefining the governance hierarchy and invites considerable debate.

Where to go from here, then? I think we need to call out the existence and standing of projects and user groups in the constitution; we need to make the creation of those parts of the structure be lightweight and effortless, so that anyone can just do it; we need to have a structure whereby projects and user groups are monitored and helped (sponsored, if you like); we need to revert communities to be communities of interest (which can then be created as required with no effort) and build a new governance infrastructure that just gets out of the way and lets people work on bettering OpenSolaris as fast as they're able to without putting roadblocks in their way.

Tuesday, March 11, 2008

OGB: expectations

It's clear that the OpenSolaris community isn't in a particularly healthy state. While there's a lot of real work being done, the community is dogged by infighting - not to mention a distrust of Sun and its motives.

Part of this comes from a lack of clarity and focus. In particular, we're guilty of not even setting expectations, let alone meeting them.

Without clear expectations, we have a problem - everyone in the community makes up their own expectations of what they can achieve and what everyone else should be doing. All those expectations are going to be different, and everyone is going to end up frustrated and disappointed.

So one important thing the incoming OGB is going to have to do is to set everyone's expectations appropriately. People (and Sun) need to understand what can and can't be achieved - even if an accurate setting of expectations means they're quite low to start with.

Sunday, March 09, 2008

OGB: Bio

Glynn asked the OGB nominees to provide brief affiliations and a Bio. I've done the affiliation already, so here's a brief history of me:

I come from Nottingham, England. I read physics at St. John's College, Oxford, and stayed on to do a D. Phil. in Theoretical Astrophysics in the Department of Theoretical Physics. Computing at this point was VAX VMS based - I managed to get my hands on a VAXstation 2000.

I moved over to Toronto, to the Canadian Institute for Theoretical Astrophysics. This is where I first used Unix in anger - on my own Sun 3/50. (Almost everyone had a 3/50, so I rewrote my code and ran it across a dozen 3/50s at once.)

Back to England, to the Institute of Astronomy in Cambridge. There was a gradual incursion of Sun hardware into predominantly VAX territory and, being a Unix guy, I ended up doing the sysadmin because there wasn't anyone else at the time who would. I also set up one of the earliest websites in the UK at the IOA.

With a family to support, and having found that I had some level of competence with this computer malarkey, I joined the Medical Research Council to work as a Systems Administrator at the Human Genome Mapping Project, which offered online services to academic researchers working on the Human Genome Project. We were Sun based, by and large, and this lasted almost 11 years before the grant renewal process failed and I was out of a job.

I then spent a year commuting down the the University of Hertfordshire. This wasn't a great success, not helped by the several hours a day it took to drive down there.

I'm now working as Senior Unix Systems Administrator for ProQuest, an online publisher in Cambridge. Not only is it a good place to work, it's less then 15 minutes from home.

Along the way I've used mostly Sun systems, with IBM, SGI, and DEC alpha, and Linux disturbing the peace from time to time. I remember the pain of the SunOS 4 to Solaris transition. I've been using Unix for almost 20 years and have never (a) used a shell without command line editing, and (b) used vi. (Except for long enough to exit them for something better, that is.)

I was part of the beta programs for Solaris 7, 8, 9, and 10. Including the various update releases, and we were a Solaris 10 platinum beta site. We managed to beta test some hardware along the way - including the B1600 blade system, the V250, and the V40z (tunred it on and I was deafened; ran one benchmark and I was a convert). And then I became involved in OpenSolaris, including being part of the pilot.

Monday, March 03, 2008

X2200M2 blues

Again. Sigh.

So the X2200M2 has an upgraded firmware that updates both the BIOS and the SP. Updating is a good thing, as there are a number of known problems with old version (open relay, amongst others).

For those updating, it's important to be aware of a few issues that you might run into.

When I tried this, I essentially lost all serial access to both the BIOS and the running instance of Solaris.

If you're running something older than the tools and drivers CD 1.3, then go to the 1.3 version first, and do the newer versions in a separate step. If you don't, you'll get a CMOS checksum error and will need to clear it. I found a physical power-cycle worked.

You might have to go into the BIOS and reset to optimized defaults.

Once you've updated to the current version, then you might have to go into the BIOS (under Advanced/Remote Access) and change the serial port from COM1 to COM2. Doh!

They default baud rate has changed back to 9600. You know all the customizations (including building a modified boot image on your jumpstart server) we've had to do to set the baud rate to 115200? Don't do those bits, they now cause more harm than good. (The other customizations are still necessary, it's just the baud rate.)

So after that I do get both BIOS access and can see Solaris booting up. I don't see grub coming up as it boots, though.

Sunday, March 02, 2008

Nominated for OGB

I was slightly surprised - but highly gratified - to be nominated for membership of the OpenSolaris Governing Board.

I had to think about this, as being on the OGB clearly isn't a walk in the park. There's a lot of work to be done - for whoever gets elected this time around.

There are obvious mismatches between the existing constitution and the actual functioning (if that's the right word) of the community. As such, there are a few constitutional amendments already under discussion.

One of those amendments (554) is that candidates should disclose their interests. In accordance with this:

1. I'm a systems administrator employed by ProQuest in their Cambridge office. We use Sun and Solaris, so are a customer of Sun. My management are happy for me to accept the nomination, provided (as always) that my OpenSolaris work does not interfere with my professional responsibilities. As such, the views and opinions I bring are my own, and are not representative of my employer beyond the fact that I'm working in a context where I'm paid to use Solaris. I do not believe that a conflict of interest exists.

2. I've been a user (and beta tester) of Solaris for years, and have been a long-term participant in the OpenSolaris project. As a user rather than a developer I believe I would broaden out the OGB, and can make a useful contribution towards developing our fine community.

3. I'm a core contributor in the Systems Administration and Installation and Packaging Community groups. While not a programmer by trade, I have made some modest code contributions to OpenSolaris.

Monday, February 18, 2008

If only!

Tried this on a test server:

# psrinfo -v
Status of virtual processor 0 as of: 02/18/2008 16:37:34
on-line since 01/21/2008 10:00:38.
The sparcv9 processor operates at 2793 MHz,
and has a sparcv9 floating point processor.

We can dream, can't we?

What's actually happening here is that I'm trying out the sparc emulator from Transitive (which now runs on Solaris x86) and it's reporting the speed of the Opteron processors in the box.

Python considered harmful

I used to think that using java was like being in league with memory suppliers.

However...

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
10946 fred 1 60 0 3692M 270M sleep 25:34 0.00% python
19559 joe 1 59 0 1950M 836M sleep 17:21 0.00% python
20738 bill 1 59 0 1787M 1608M sleep 3:28 0.00% python

That's on a machine with 4G of physical memory. And given that python is being used more widely, and that data volumes are increasing, I need to do a couple of things. First, order more memory; and second, work out how to build a 64-bit copy of python with all the modules we use.

All-in-one servers

One recurring theme as I build servers is that I often want a configuration that's not available.

Sun are the worst culprit as, while they generally have excellent products, the actual range of configuration options is rather limited.

And one of the reasons I was interested in the X4150 in the first place was the ability to have 8 internal disk drives. For many things I would prefer internal storage, if I can get it.

So what is wrong with external storage? Well, it can be very expensive, because you need to get a chassis, maybe raid controllers, and HBAs, not to mention the extra rack space, power cords, cables, and having separate boxes to manage and monitor. And if all you want is a few hundred gig of space, then it's just not worth it.

So consolidate on a SAN, you say. Maybe, but SAN storage itself is rather expensive. For small amounts, the cost of HBAs and the fibre infrastructure can be prohibitive.

I've been attracted to iSCSI, but while it does actually work great, it's limited to low-bandwidth light-use scenarios. (I just have regular gigabit ethernet.)

So a solution where the storage can fit neatly in the server is very attractive. At the moment I have something that takes about 500G, but is likely to grow slightly. So I think I need about a terabyte, and it's got to be reasonably quick, faster than iSCSI anyway.

So looking at the X4150 I would probably do something like use the first 2 drives for the OS and use the upmarket raid card to create a raid-5 array across the other 6 drives. So that's 5 drives worth of data, or about 700G.

Close, but it's not quite close enough. It's just a little bit tight. It would be nice to have larger drives, but as Ben has discovered, larger capacity drives in the small (2.5") form factor just can't be had.

So if we have to go beyond that then we might go up to regular 3.5" drives (which gets you 15K rpm and 300G or 450G capacities), or you need more than 8 internal drives. In either case that implies a larger chassis. (Sun have an X4450 which is the big brother of the X4150, but that's identical as far as supported drives are concerned.)

Looking at what Sun have available, there really isn't anything. And no I don't want a thumper, not for this application anyway. (It's a shame that there aren't more variations on the thumper theme.)

Ho hum. Off to see if I can find something different. A Dell 2900 is a lumpy tower. What about a HP ProLiant DL580 G5? (The DL 320s might work, but the max memory is a little too small at a mere 8G.)

Anyone care to suggest alternative options? (Must run Solaris!)

Sunday, February 17, 2008

X4150 experiences.

It's clear from the comments I've had that I'm not the only one who's had issues with X4150s.

Those of us who have been playing this game for many years manage to surmount these minor obstacles, so we can take advantage of the good things that these servers have to offer. But it does worry me that customers without the experience (or confidence and contacts) to get past the irritating issues lose out on what could be a good solution, and that a supplier loses out on a potential sale.

Saturday, February 09, 2008

X4150: lit up

So I've had the X4150 booted and running for about a day now. And it's managed getting on for 8 cpu-days worth of work so far.

It's looking good. On the workload we've tested so far, it seems to be about as fast as - if not marginally faster than - a comparable opteron, such as the X4200. Overall it can chew twice the workload because it has twice the cores.

It was a bit of a drag getting there, but I think it was worth the effort.

Friday, February 08, 2008

X4150: where do I boot from?

Seems my earlier optimism was a tad misplaced. I come back a short time later and the X4150 is sitting there at the interactive prompt you get when doing a network boot. So the install finished just fine but then it booted off the network again.

(See: I knew it was a good thing not to default to install. It could have installed itself over and over in a loop.)

The cause is reasonably obvious - either the drive isn't listed as bootable or the boot order in the BIOS is wrong.

This is where Ben's tip saved me either a bit of fiddling or a walk down to the machine itself.

I get into the BIOS, and the boot order shows me the DVD, the 4 network ports, and the disk I installed to. In that order. OK, I move the disk up and reboot.

Success! We boot off the hard disk.

(There's still an open question here: I want to have all four disks bootable just in case I lose one, and I'll set them up as two sets of mirrors - one for running, and one for live upgrade. I still need to work out how to add the rest.)

X4150: No disks found

So I've managed to get to the point where I can control my new X4150 using the SP, and can get to the system console.

I have the address set in dhcp and my jumpstart server configured, so let it boot up and see what happens.

(OK. So I goofed when typing the mac address into the dhcp server the first time. So it took me an extra attempt.)

Just as an aside, the X4150 is like the X2200M2 and requires the serial port set to 115200 baud, so I've rebuilt the boot image it gets from the jumpstart server just like it says in the documentation.

Off it goes, boots up, I hit 2 for jumpstart (I don't default this so that I don't accidentally overwrite a system if it boots off the network by mistake or for maintenance), and it's going well. What normally happens is that it complains the disk in my jumpstart profile isn't valid on my system.

Not this time. "No disks found."

That's not good. One nice thing about the X4150 is the 8 disk bays on the front, and I know that 4 of them are occupied. So why can't it see them?

I was shipped a HBA and a cable kit, but hadn't found any documentation on why I would need it or how to install it. So maybe you actually do need an additional HBA to make it work, which makes me wonder why on earth this requirement isn't prominently documented, and why they don't have a functioning on-board disk controller, and why the disk bays have been carefully cabled up to the on-board connectors when that's not going to work?

OK, so I pull out the old cables and put in the replacement ones. Looks like the HBA has to go in the middle slot, otherwise it fouls the memory slots. I hope I have the cables in right and routed the correct way.

Wonder of wonders, I boot up and all the disks are visible. Jumpstart tells me the disk isn't valid on this system, but that's a trivial fix to the jumpstart profile to get the controller numbering correct and off the installation goes.

Pretty quickly too!

Thursday, February 07, 2008

LOM: Consistency would be nice

Most of Sun's x64 servers have some sort of LOM, but while they're generally superficially the same (they call themselves the SP, they have a similar /SP and /SYS layout), all the different models differ in the details.

And we all know that that's where the devil is....

Why on earth are the steps to set the IP address subtly different between systems, for example?

X4150: talk to me!

OK, so the first step is to connect up the cables, power the server on, and configure the SP.

So I do that - system and management networks, connect up the serial port, tip in, and apply power.

Silence. Not a peep. Not a sausage.

Come on, now. Talk to me!

Now I think this is OK, because the same tip session from the same host works flawlessly on every other Sun x64 server I have. But still I go through the motions - different host, different cable, various types of cable.

It's still sulking.

I give Sun Service a call. I don't think I'm doing anything wrong, so we'll see what they have to say - maybe I've got a faulty unit.

Turns out there's a problem with some units shipped with the wrong settings. If this happens to you, connect up a real keyboard and monitor, and power up the box (the real box, not just the standby power to the SP). Hit F2 to go into setup, and look through the settings. The "external serial port" should be set to SP. If it's set to system, hit F9 to restore defaults, and F10 to save (I think it's those function keys). I get back to my desk and there's the regular SP login prompt.

Thanks to James of Sun Support for tracking that one down for me.

Server deja vu

Over a year ago, I had a great deal of fun and games when we got some Sun X2100M2 systems:

On to the X2100 M2
Me versus the M2
The M2 comes alive
Server Wars: The M2 strikes back

Well, it looks like I have another battle on my hands. After a stack of flawless installs of X4200 and X4500 boxes, we decided it was worth getting an X4150 to see what the Xeon offered. Our cpu-hungry applications love the idea of quad-core; our data storage likes the idea of 8 disks in a 1U system.

Installing this thing isn't going well. Watch this space...

Sunday, January 20, 2008

Christopher John Tribble

My father, Christopher John Tribble, passed away peacefully early this month.

He had an operation to remove a cancer tumor about a year ago, and things seemed to be going well until he reacted badly to chemotherapy towards the end of last year, which resulted in a downward spiral.

Watching him slip away was very hard, especially the awful feeling of impotence in the face of the terrible inevitability of it all.

It's not been the best start to 2008, but my father was a pragmatic chap who didn't want anyone to make a fuss, so it's now time to get back to normal.

I've lost, amongst other things, a friend.

Wednesday, October 10, 2007

Solaris on a Laptop

Every so often, things go without a hitch.

Today I got a laptop to look at. It's an old Dell D600, and the requirement was to have something looking like our production Unix environment on it. One of my colleagues had already stuck a Solaris DVD and got Solaris installed, and then handed it over.

First job was to get the network working. That's easy - off to the Broadcom download site to pick up the BCME driver. (Oddly, it's in the server section of the downloads.) That installed, I was on the wired network.

Next for wireless. A quick look and this machine seemed to have a Broadcom 4320 wireless chip. So off to the OpenSolaris Laptop Community, and it looks as though I need the ndis wrapper. With that and the windows driver I downloaded from Dell, I just followed the instructions. OK, ifconfig brings up the wireless.

That's a bit crude, and it would be nice to avoid too much CLI intervention. Enter wificonfig and Inetmenu and we're all set. Having given the laptop user the 'Ginetmenu' and 'Primary Administrator' profiles, it's as easy as firing up inetmenu and selecting from the list.

And both wireless and wired networks work (almost) flawlessly.

I didn't have a chance to look at wpa support, which would be needed some of the time. But I was mightily impressed by how easily everything I did try worked out.

Thursday, September 27, 2007

Jumpstart Profile Builder

The latest version of solview includes something I've been working on for a while - the ability to construct a jumpstart profile.

The aim is to get a tool to do the hard work of resolving package dependencies, so that the resulting jumpstart profile is nicely self-consistent. Doing this in the past has been very much a case of trial and error. No more!

I'll just show one screenshot here, of adding evolution on top of the reduced networking metacluster:



There's a lot more work than can be done here. The interface, while workable, is rather crude. I would like to get it to construct a reasonable jumpstart profile based on an installed system, and it would be nice to be able to feed it all my old profiles and get it to fix them up for me. And I'm sure there are many other ways it could be improved, or used in different ways.

So if you have any suggestions, give me a shout!

Sunday, September 09, 2007

Solview update

I've just released a new version of solview.

If you've tried to run solview in a non global zone, then you'll find that it didn't work. (Thanks to Tony Curtis for spotting this!) This was due to an ignorant and lazy programmer. Yup, I hadn't tested it in a non global zone for a while.

This latest update actually checks that the files it relies on to grab the information it's going to display exist, and behaves properly if they don't. I'm not saying that it works correctly in all circumstances, but it does work in the cases I've tested, whereas previous versions were rather more fragile.

JKstat update

It's been a bit quiet on the JKstat front recently (summer holidays may have had something to do with that).

I've done a little bit of cleanup I've been meaning to for a while - getting rid of fixed text strings and replacing them with resources, making localization possible.

I've also added a little chart feature - when charting a statistic, it's now possible to add other statistics from that kstat to the chart.

Nothing earth-shattering, but it still represents steady progress and so there's now a new version available for download.

Monday, September 03, 2007

X2200M2 fault diagnosis

I'm just setting up some new servers - Sun X2200M2 (running Solaris, of course).

One wasn't happy. Solaris would start to boot via PXE but the system would reset (over and over, in a loop). A quick look and the fault light was on. Not good.

These systems have remote management - ILOM - but after searching and looking through the documentation I couldn't actually see how to persuade it to tell me what was wrong. It's not helped by the fact that there are several ILOM variants in use, but the one on the X2200M2 is one of the more basic ones.

After running through all the options systematically, I stumbled across:

show /SP/AgentInfo/SEL

which tells me

Nonrecoverable ,2007/08/31 17:22:00 ,CPU1 DIMM 3 has multi-bit error

Aha! I've reseated the DIMM after swapping it with its partner, and the installation is proceeding apace.

Saturday, September 01, 2007

W2100z update

I've just updated the BIOS on my home W2100z (downloaded from here).

The old BIOS on the machine was antique. I mean, it predates all the old versions on the download page. And I remember updating it at least once (to get PXE boot working), so it wasn't the factory version.

The update was pretty smooth. However, on booting back into Solaris I see a CPU pegged and very high interrupt rates. Not good.

Now, it turns out that I've stumbled across this before. And in response to that, Alan Hargreaves got the thermal zone driver backported to Solaris 10. So it's available in patch 125107.

Unfortunately getting hold of this patch is proving troublesome because sunsolve is broken again.

OK, so I go have a cup of coffee and this time, success. Patch applied and just save this blog before I reboot.

Tuesday, August 28, 2007

Fixed funny fonts in gpdf

If you're running Solaris 10 on x86, you may have noticed that some PDF documents (particularly, and somewhat ironically, those generated by Sun) display as garbage. Instead of readable text you get rubbish - rather like this:



What you need is the recently released patch 119813-06, which fixes bug 6375381. (That's the original bug, which shows it got fixed in nevada ages ago - 2147903 is the one that tracks Solaris 10 integration.)

So now it looks like this - a vast improvement:

Wednesday, August 22, 2007

That's a busy disk...

From iostat -xnz 1 on my desktop:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 3.0 0.0 10.5 18444348816.8 18444348816.8 6148914691236.1 \
6148914691236.1 1844434881679 1844434881679 c1d0

Sorry if it wraps too much.

Of course, those numbers look very suspicious. I have seen this sort of silliness several times in iostat output recently though.

Monday, August 20, 2007

Sunsolve needs a makeover

Sunsolve is an excellent resource. I don't have any issues with the content - it's great stuff.

Where I do have problems is with it's reliability and performance. I couldn't get through at all for a while over the weekend, and often have problems with my automated patch scripts - which can time out, report 500 errors, or tell me I'm unauthorized, all at random, and have been that way for a long time.

I even got the following gem off the website just now:


Come on Sun - show us how good your stuff really is by making your own sites work well!

Saturday, August 18, 2007

Fingered!

I found this nugget on indiana-discuss to be quite apt:
one of the keys to identifying old-time Unix admins, particularly of the SunOS persuasion, is their use of tcsh
Yup. That's got my number!

Saturday, July 07, 2007

Laid low :-(

I've been suffering from a fairly severe viral infection for the last week. I've had to take several days off work, and felt pretty atrocious. I'm hoping that it's starting to ease and I'll be feeling better in the next couple of days.

Sunday, June 24, 2007

And solview too

Not content with updating jkstat, I've released a minor update to solview.

New jkstat version - real progress

I've let loose a new version of JKstat, also available from the jkstat project page.

This is version 0.19, and represents real progress. I'll start with some of the fiddly stuff: I've got rid of all the individual utility scripts (they were breeding out of control) and replaced them with a single jkstat script that takes subcommands. I've removed the native methods for looking up individual statistics, as there was no possibility of error checking (like to see if the value returned was valid). There's an client for nfsstat and the start of one to emulate the kstat(1M) perl client. I've got a swing widget that allows you to add a menu to any java client from which you can launch some of the demos.

The big feature is that I can now detect changes in the underlying kstat chain. I've added another class to handle this, so it's responsible for working out whether something has changed, and which kstats have been added and removed. This wasn't quite as big a job as I had feared.

Getting the demos to use this was somewhat more difficult. In particular, the browser took far more work than I ever expected. The kstats are in a tree, they're presented using a JTree, so you just update the tree, right. Wrong! I had originally placed the kstats into a Hashtable (it's a natural things to do), passed that to the JTree. Then Tom Erickson helped me out by creating a sorted Map, and a revised JTree that could take a Map. (And why some of the java swing components haven't moved into the 21st century and become aware of the Collections framework is beyond me.) I wasted a lot of time thinking I could update the Map and get the tree to refresh its view of the world, but it just didn't work. In the end I had to get rid of that and construct my own model, doing everything manually. Ugh.

It's not finished. Not all the demos handle updates. (Those using tables do - that was easy.) And the browser has a couple of cosmetic flaws: the newly added kstats aren't currently sorted correctly, and if you're watching a kstat that gets removed it doesn't tell you that anything has happened. But it's a major step forward and I'm going to take a deep breath and use this for a while before diving back in.

Monday, June 18, 2007

Solaris metacluster sizes

As part of the Indiana discussion, I was looking at the installed sizes of various Solaris clusters of packages. The idea is to work out where all the space is going so as to be able to trim an OpenSolaris distribution onto a single CD. Depending on compression, this might allow something just over 1G of software.

A minor extension let me do the same for the various metaclusters you can choose from in the Solaris installer. And the sizes of those on a Solaris 10 update 3 x86 system are like this:

157.945M : SUNWCmreq
168.594M : SUNWCrnet
195.372M : SUNWCreq
2010.12M : SUNWCuser
2579.84M : SUNWCprog
2614.19M : SUNWCall
2614.2M : SUNWCXall

This confirms what we've known all along, that the small clusters are tiny, the other install clusters are huge, and there's a great big gap between the two extremes. More than that, the end user cluster misses out on a lot of stuff that would be really valuable to an individual user.

(I would recommend to anyone who wants to customize the packages when installing a Solaris system - either using jumpstart or interactively - that they start with either SUNWCrnet or SUNWCall, as the others don't help you much at all.)

The snag with building a CD sized distribution is that the target size is right in the middle of the gaping chasm between a fully functional (SUNWCprog) and minimized (SUNWCrnet) layout.

Fortunately, things aren't quite as bad as they look. These numbers are for a regular Solaris system. This includes a fair amount of software that isn't freely redistributable, so all the numbers shrink. Sometimes by a lot, as if you start taking StarOffice or java out then you save significant amounts of diskspace.

Sunday, June 03, 2007

solview does patches

I updated solview so it now shows patches. It gets most of this for free, by looking at the installed packages. So you get the standard stuff from showrev -p, with the packages having a short description, plus it will display the readme file for the patch if it can find it. Next step is to parse the patchdiag.xref file to work out what needs updating, like pca does.

Sunday, May 13, 2007

Solview, JPack

Last week I added contents file parsing to solview.

More recently, Michal Pryc announced that he was working on JPack, an easy software manager and installer written in Java.

Now, the current focus of these tools is slightly different - solview is about analysing what your system has, JPack aims to make adding new software painless.

I was planning to add package management to solview at some point, but probably won't bother now - there seems little point in duplicating the work in JPack so I could just reuse that, and hopefully I'll be able to contribute to JPack as well.

Motivated by the rather better look of the GUI in JPack, I had a little go at reorganising the layout in solview so that the package list is done as a table. It does look better. I also did some refactoring of the code - both to make it more usable by other projects (such as JPack), and to remind myself how it all worked.

So there's a new version of solview now available, with the above mentioned refactoring and layout changes, and with the slow operations done in the background with a SwingWorker.

Monday, May 07, 2007

solview - includes contents

A little bit of work this weekend and I've let out a new version of solview.

No sign of the promised jumpstart profile builder, though.

But what is present now is a contents file parser. This allows me to add a summary of the number of files and space used to each package (and to each cluster). For example:



And in addition I added a filesystem browser so you can choose a file and see what packages and clusters it's included in:



There's a slight amount of lunacy involved in parsing the contents file. Don't be surprised to see OutOfMemory errors, as the memory usage is fairly high (several hundred meg, commonly).

Monday, April 23, 2007

Phoronix and Solaris

Interesting: Phoronix now supports Solaris. See the article.

Not only an endorsement of the growth of interest in Solaris, but hopefully another source of information regarding hardware compatibility for Solaris.

Sunday, April 15, 2007

JKstat 0.17 - includes charts

I've been gradually continuing work on JKstat. The latest version (0.17) has a number of code cleanups, but also includes charting capability. (Uses JFreeChart.)

You can right-click on any numerical statistic in kstatbrowser and chart that statistic. Here's an example:

Sunday, March 25, 2007

JKstat 0.15

I've made a new version of JKstat available.

There are quite a few changes underneath. I'm very grateful to Tom Erickson for sending me a bunch of enhancements and fixes. Some of them made it into this version fairly directly - particularly the sorting of kstats in the browser.

This version has also been revamped and now requires Java 5. I had been thinking about modernizing the code for a while, but had always put it off. Tom again provided the impetus, as he went ahead and used generics in the fixed version he sent back to me. I reimplemented it myself, so I could actually understand what I was doing to my code.

I actually went and rewrote solview first. This was a useful exercise in itself, as one of the advantages of using generics is that it makes you think harder about the way you define classes and how your code is structured. In solview I was in the middle of cleaning the code up to define Solaris packages and install clusters as proper objects, and using generics made sure I actually got that right.

JKstat was also accepted as an OpenSolaris project, so I'm going to be moving development over there soon.

Saturday, March 24, 2007

OpenSolaris priorities

Before the OGB election there was the OpenSolaris Comunity Priorities vote.

Looking at the result, and the clear winner in terms of priorities is the public defect management system. Yep, I put that at number 1 too. (I think...)

The OpenSolaris Bugs interface did get revamped recently, and that was a significant improvement. However, it's still not really doing anything for OpenSolaris - it's just a way to give a limited view of a subset of Sun's bug database.

Importantly, there's no other integration with anything else the community might be doing, and no way for a community member to interact with the system. How can it be improved?

Here are a couple of things I would like to see:

Eliminate all the crud. The database is littered with all sorts of things that aren't bugs, and still contains bugs that are antique. We need to take some of the open bugs forward, but most of the database has no relevance to the current codebase.

I think this means we need a completely separate database. And bugs that Sun generate through their support organization need to be put into the public OpenSolaris bug database the same way that Sun currently logs bugs against external projects such as Gnome.

We need to allow community members to participate in the triage process, and allow comments and followups to further refine the bugs and provide sample solutions, test cases, and prioritization. This extra interaction will build a stronger community, and provide ways for more people to get involved.

And the database needs to be linked into the whole workflow, so that there's a way to see who might be working on a particular bug and contact them.

The bug database is the crucial missing link. We want to improve and develop OpenSolaris, and to do that we first need to know what's broken and what's missing.

Polls and voter apathy

I've been sitting on the fringes of the OpenSolaris Governing Board Election. I would have liked to have been more involved, but have been struggling to find time. Strike one for apathy!

(And, while I did consider putting myself forward as a candidate, the fact that I haven't been able to effectively contribute up to this stage also implies that I wouldn't be able to do the job properly in the unlikely event that I were to be elected.)

There does seem to be a fairly low turnout so far. There have been a number of calls for people to get their finger out, and we seem to be creeping closer to a quorum.

Alan Coopersmith had a look at the contributors and communities. This shows what we've known all along - that the different communities have very different levels of participation in the governance process. Looking at the way that core contributors are distributed, it's not clear to me now that relying on communities to generate the initial list was - or is - a good thing.

There seems to be a view that reorganising the communities is going to be a big task for the incoming OGB. (If one gets elected, that is.) I'm not so sure about this. I feel that we're trying to build too much officialdom into the community structure, when in reality the communities are more social structures connected by a common interest. But more on that later.

I'm fortunate that I am classed as a core contributor and, yes, I've voted.

Perhaps core contributor status should be automatically be rescinded from those who haven't bothered to vote?

Thursday, March 08, 2007

Spring Clean and Upgrade

I've been doing a bit of a spring clean and upgrade on some of my home systems.

The family PC had its memory doubled. It's survived 3 years with only 512M, which ought to be enough but it's obviously running out. We use fast user switching so there are usually 4 people logged in. That shouldn't be so bad, but it implies 4 web browsers and they guzzle memory like it's going out of fashion. I also replaced the broken DVD-ROM with a working dvd writer.

I think it's the first time the case has been opened for 3 years. And it was filthy inside, so it got a thorough vacuuming.

Next step was a couple of Sun workstations. I use one for OpenSolaris development, and I shuffled that up from build 55 to build 59. I also took a gamble on rebuilding one of the children's systems from an essentially unpatched FCS version of Solaris 10 to the latest Solaris Express release - the same build 59. Only complaint so far is the placement of the tab close buttons in Firefox 2 versus Mozilla.

Next step - run Nexenta using the VMware player.

Tuesday, February 20, 2007

A new use for a D1000 array

I found a new use for a D1000 array last night.

We bought a pallet-load off eBay a while back. We only actually needed one, but the whole pallet was no more expensive. So the first one got used for the purpose for which it was bought, and I've used a couple more filled up with old drives to test ZFS and the like.

We just piled the rest up in a store-room.

Which is where they had stayed for a few weeks harmlessly gathering dust.

Last night I went in there to look for some cables, and was somewhat annoyed to find that the door wouldn't open when I tried to get out. The lock release did absolutely nothing.

What you have to understand is that this was an old comms or server room. So that's a security and fire door in my way. No windows. No phone. My mobile is on my desk. It's almost soundproof as well, and it's getting late.

I did manage to attract someone's attention as they were passing in the corridor, and we tried all sorts of tricks to get the stupid door to open, with no success.

Eventually I piled all the D1000s up and stood on top of them. (Which just goes to show that buying quality kit is always worthwhile. You try this trick with cheap kit and it will fold under the weight.) Originally I was hoping to get into the false ceiling and work my way through, but even that was a bit of a stretch and very cluttered.

Eventually we took out the panel above the door and I got out that way.

See, I knew this old kit would come in useful some day!

Sunday, February 11, 2007

Updated jkstat

It's been a while (almost a year) since I last updated jkstat, but I've just put version 0.13 up for download.

For those that don't know, jkstat is a Java jni interface to Solaris kstats, allowing you to access a wide range of useful data held by the Solaris kernel from a Java application.

One thing I have done is this version is try to improve the look of the sample applications, most of which has been achieved by replacing the rather clumsy GridLayout with SpringLayout.

Thursday, February 08, 2007

Impressed with Xfce

I've been having a play with Xfce on one of my Solaris workstations.

And I have to say I've been very impressed. It's reasonably fast and light. It has an increasing number of applications. The file manager - Thunar - is quite nice.

There are two things that I think are in favour of Xfce. The first is that because of its common roots with gnome it can leverage all the gnome applications and GTK themes. The second is that it is very polished. There is a good section of window manager themes, and the artwork is very well done.

All in all it's well worth a look.

Monday, January 22, 2007

Sun, Intel - friends?

OK, so Sun and Intel made a big fuss about some deal or other (one which doesn't really set the pulse racing, mind). What I found ironic was the ticker underneath the main feature on the Sun home page, where they put the knife into Itanium.

I remember Sun recently selling a number of Intel powered systems. The LX50 was a nice box - even if it was pushed with SunLinux. The V60x was OK, but I quite liked the V65x, and that sort of system (6 internal drives) is something that would be quite valuable. And the B200x blade was quite a marvel - twin hyperthreaded Xeons and a decent memory configuration in a tiny package.

I expect closer collaboration will improve driver support for Solaris, but I'm also hoping for a wider range of systems designed for Solaris to give us a breadth and variety of system configurations that Sun themselves seem disinclined to supply.

Sunday, January 14, 2007

Backup Blues

Taking system backups is an essential part of any IT operation, but seems to involve an awful lot of pain.

Over the years I've generally been happy with Legato (now EMC) NetWorker, sometimes known as Sun StorEdge Backup or something similar. What I really like about it is that it's very low maintenance - easy to set up and configure, and gets on with its job quietly in the background without any fuss.

I was a little concerned that the latest version has dropped the old nwadmin X-windows based GUI in favour of some Java client-server thingummy. I still don't really like the java interface, but as I previously mentioned it's pretty low maintenance so you don't need to spend much time using it, and it's generally easy enough to use. The one downside to it is that the licensing is a bit more complicated, and the license registration instructions explicitly refer to the old GUI.

I'm currently using NetWorker for one half of my backups. I put in a new server and tape library and moved the NetWorker license across. The transition has been very successful, although the primary win is that I replaced an L20 with 2 DLT-7000 drives with a C4 with 2 LTO3 drives. The old system simply didn't have the capacity or the performance - backups would take days, and some backup runs came in at over twice the capacity of the library. Now they take hours and fit onto a couple of tapes.

The other half of my backup story is more appropriate for the tile of this post. There, we're using an alternative product. OK, specifically, NetBackup. And it's much more trouble than NetWorker has ever been.

I'm trying hard to give a balanced view here. After all, NetBackup is widely used and must have some good points, right? Maybe...

The first thing is that NetBackup is very high maintenance, at least compared to NetWorker. It needs constant attention paid to it, seeming to go off and hide in a corner if you don't give it constant encouragement. I'm not surprised that large organizations have dedicated NetBackup administrators, or even teams thereof.

The second thing is that I (and I know I'm not the only one) have trouble with the way you actually define backup schedules. In NetWorker it's trivially easy - you define what you want backed up, what level of backups you want, and when to do it. In NetBackup you mess around with backup windows, frequencies, policies, and schedules. Again, this is much higher maintenance - I usually need to manually configure half a dozen screens in NetBackup as opposed to simply selecting one option under NetWorker.

The third thing is reliability. The only time I have known NetWorker fail to do a backup is if I've run out of tape. NetBackup seems to throw fits all over the place. We see regular timeouts, errors when it's outside the backup window, and occasional (once or twice a month) where the whole thing freezes up and needs a forceful restart. I've just upgraded the NetBackup server itself, and after tuning got acceptable performance (the default performance was dreadful), but the catalog backup takes over a day for something that ought to take 20 minutes, and renders the machine completely unresponsive while it's at it.

I'm tempted to replace the NetBackup installation with NetWorker, which raises a number of issues. I'm happy that it will be fairly painless on the Unix side, but I'm less sure about the Windows side which also needs backing up. While I have run NetWorker on Windows, it became apparent fairly quickly that it was basically a Unix product and looked to be in alien territory. And my experience of NetWorker's application modules (as needed for the likes of databases, exchange and the like) hasn't been entirely positive. So I'm thinking of a completely separate system for the Winddows backups.

I've tested a number of other backup products, but under Unix. And there it was obvious that most of the Windows products looked completely lost under Unix. I haven't really come across a truly cross-platform backup solution, and it's not obvious to me that such a thing actually exists.

Given how conceptually simple backups are, is it asking too much to have solutions that just work?

Wednesday, December 27, 2006

Fun with NVIDIA driver

I have a Sun W2100z running Solaris. It's a fantastic machine, but the graphics has never been too hot. It's got the entry level graphics card - an NVS 280 if I remember correctly - and I've been using the bundled nv driver as I wasn't aware that the accelerated driver could do much better.

I was trying to get DVD playback to work, and gxine seems to work (after I had built and installed libdvdcss), but was rather slow and I noticed that Xorg seemed to be pegging a whole cpu. In addition, it might be neat to get Looking Glass to work.

So I ambled over to NVIDIA and their Solaris Driver downloads, and noticed that my graphics card was listed as one of those supported. I downloaded and installed the latest version (9764), and that didn't work. I get an error message as it boots that seems to be from the nvidia driver, but then the system immediately reboots. Not good. (I went into failsafe from the grub menu and used pkgrm to delete the two packages to recover.)

Then I had a look in the forums and noticed a message about a legacy release (9631). Now, it doesn't really explain what the legacy release is in this case (the information on the download page about legacy releases applies to Linux), and there isn't a link to the legacy release on the download page itself, but to cut a long story short - the 9631 legacy release works fine on my entry level box. DVD playback is a bit smoother, although still not brilliant, but the cpu load is pretty negligible compared to before.

If you, like me, have an entry-level W2100z with an NVS280 graphics card, then you may be advised to use the NVIDIA legacy driver rather than the latest and greatest.

Tuesday, December 05, 2006

Jumpstart Profile Builder

One of my little projects is to write an interactive jumpstart profile builder, so that you can play with Solaris software selections, and construct a working jumpstart profile.

The problem currently is that you can, with some effort, create a jumpstart profile. But the only real way to test it is to use it to install a system, and either see what works and what doesn't, or look at the installation log to see what package dependencies you messed up on. Eventually you can get quite good at this, but it shouldn't be this hard.

You can select packages in the interactive install, but the dependency resolution is awful. It tells you what the dependencies that failed are, but doesn't give you a way to feed that back into the gui to resolve them. Nor does it tell you what dependencies are required as you look at the individual packages. And once you've selected your software, it will install it but you can't go from that to create a jumpstart profile to automatically install the next set of systems.

I have most of the base code already, in the shape of Solview. That knows how to turn the .clustertoc file (which is where the clusters and metaclusters are defined) into a tree representation of the packages, that could be used as the basis of a selection tree. And it can evaluate the dependencies. I've also got code that can describe a jumpstart profile.

Where I'm currently struggling a little is the actual interface. Using a java swing JTree as the base is fairly straightforward - this is what Solview does already. And I need a custom cell renderer to add checkboxes to show whether the software package is selected or not (or, in the case of an installed system, whether it's been installed or not). But then I got to thinking and I can't use the regular JCheckBox, as there are more states that I wish to display than JCheckBox gives me out of the box.
  • Package or cluster selected
  • Package or cluster not selected
  • Cluster partially selected
  • Package or cluster selected and cannot be deselected
The last case is for required packages and clusters (which you can actually delete in a jumpstart profile but which it ignores), and I could probably do simply enough by having the checkbox selected but disabled. I might also want to have a separate state for packages and clusters which have forward or reverse dependencies, so it's clear visually whether it's safe to select or deselect them.

I've had a quick look around with the help of google, and I'm obviously not the first person to have requirements of this type. And there is a fairly large amount of sample code out there. Unfortunately it appears to be fairly complex, and I would be adding a significant amount of extra code, but it doesn't appear that I have much choice.

Wednesday, November 22, 2006

T2000 vs bison

This wasn't a benchmark, or even meant as a test, but it was an interesting number nonetheless.

I installed bison today. As I have 3 different families of sytems (Sparc Solaris 8; Sparc Solaris 10; x86 Solaris 10) I have 3 copies of /usr/local, so build on 3 different systems. Anyway, after building bison (and the T2000 allows a very parallel make that some builds can take advantage of) I ran a make check.

The results were quite interesting. My sub $1000 Opteron 146 system took 107s. An old V880 (750MHz US-III) took 529s. The T2000 took a whopping 794s. This really isn't looking good.

Friday, November 17, 2006

Cheap sparcs and constrained configurations

I've recently been looking through the Sun server range, both to get some systems for present needs and to try and map out what is likely to meet our needs in the next year or two.

In some areas there are good matches and obvious choices. For example, theX2200 makes a phenomenal compute node, and the X4500 (aka thumper) is a phenomenal data storage machine.

But there are other areas where the choice is less obvious, or where what I actually want is a small variation to the configuration that Sun don't do. Here are some examples of the sort of things I want to do but that aren't available:

  • A machine at the lower end of the range, such as the X2100, V210, or V125, with redundant power
  • Much larger internal disk drive capacities, such as 300G drives in the V240 and similar, and 500G SATA drives in the T1000; these SAS drives really aren't helping
  • Some new machines with full-height PCI slots so I can use my old PCI cards and connect old peripherals
  • Something like an X2100 with 4 500G drive bays on the front (would make a great fileserver appliance)
In other words, more configuration flexibility, and less of the one size fits all mentality.

Things get especially problematic when you start looking at entry level sparc systems. Yes, application availability often means that we have to keep looking at sparc servers. And the available configurations aren't encouraging. The biggest issue is actually around disk, where the newer systems (T1000, T2000, V245, even the V445) are poorly provided for in terms of internal capacity and performance. In many ways, the V245 is a hugely retrograde step down from the V240. I really don't want to have to buy a SAN an FC cards to attach the machines to it, just to have enough space to load an application on them.

Monday, October 23, 2006

Problematic Printing

I've just migrated an important service off an old Solaris 8 machine onto a new(ish) server running Solaris 10.

Everything went pretty smoothly, with the exception of a few little printing problems. (I'm not a printer person, really. Whatever happened to the paperless office?)

I had just used lpadmin to set up the printers. I've read that printing in Solaris 10 is much better than it used to be (and that may be true) so that it can drive printers properly out of the box without the need for extra products like the HP JetDirect (what used to be JetAdmin) printer driver. Indeed, to the point that you're not supposed to use JetDirect any more.

In our testing we had ended up setting the type to any (rather than postscript) and using the netstandard model script. Anything more sophisticated and it mangled some of the printouts (specifically, we had some plaintext reports that had embedded characters to switch the printer to landscape that didn't work).

What we discovered today was that printer tray and media selection weren't working. I had been told that this was done in the application, but what that actually meant was that there was a wrapper shell script that called lp with some options that the jetdirect software can understand. And clearly the vanilla print system under Solaris doesn't grok those options.

So what I've had to do is to install the latest version of the JetDirect software that I can find (E10.34). And then add the specific model scripts for our model printers (and these are on the HP website, although they're a devil to find and the HP search is almost useless). And then using that to drive the printers made everything work flawlessly.

I'm sure there must be a way to pass the appropriate printer options to select the correct media type without running everything through JetDirect, but I've yet to find it. (Oh, and it would be nice if it used the same option names, to save having to track down every wrapper script that might be involved, although that's not absolutely necessary.)

Thursday, October 19, 2006

Network Corruption?

I've just started seeing corrupted files on one of my X2100 M2 boxes. The visible indication is that I'm unpacking gzipped tar files from an NFS mounted disk, and was getting crc errors and invalid compressed data. Testing indicated that if I used rcp to transfer the files then I was also getting corruption. (I can check the md5 signatures, rather than just rely on gzcat to barf.) There wasn't any other evidence - I'm not seeing corruption of local files (basically, of Solaris itself), nor any network errors on the machine or the switch, and it's just the one machine that's affected.

How to test? SunVTS runs on Solaris x86, so I tried that out. Which wasn't encouraging:

INFO: Host System identified is "X2100 M2"
This Platform is not supported by SunVTS.

However, if I edit the platform.conf file it does at least run. And, so far, no memory or network errors.

Mind you, the errors I was seeing earlier seem to have disappeared...

Thursday, October 12, 2006

Keeping the T2000 busy

Like many organizations, we use old computers for lightweight tasks - Sun systems just keep going. (Which can be a problem, as you can be tempted to keep them for too long when replacing them with a new cheap system would actually be a much better bet.)

So I'm putting together a couple of SunBlade 100s. They're more than powerful enough for the tasks in hand, and it saves cash. But building the software on them can take forever. Especially when you have to do it a few times before you get it right.

Enter the T2000. Type gmake -j 32 and boom! Build over.

If only ./configure could be parallelized...

Friday, October 06, 2006

T2000 - more performance

I've managed to do a couple more performance tests on my loan T2000.

the first was kicked by by the announcement that pbzip2 is available on sunfreeware. Now, bzip2 is pretty cpu intensive, so if there's a way of making it go faster then I'm all for it.

So here are some figures - in seconds - for pbzip2 as a function of the number of cpus used. I've used a small blocksize (100K) rather than the default, which helps as the test file isn't all that big.








# cpusTime
116.444
28.686
44.287
82.791
161.821
321.784

That's pretty good. I would expect to get approximately a factor 8 gain here: it's a cpu intensive application and there are 8 physical cores. So we see about a factor 9, which is pretty good.

My next test was to load up an apache/php/mysql combo and see how that goes. The comparison machine is a SunBlade 2000 with a pair of 1.015GHz processors. (Note: this is using the same binaries - not optimised code on the T2000. Yes, Sun have optimised binaries, but that isn't really helpful in comparison, as with most applications a recompile wouldn't be feasible.)

For a simple database query operation, the T2000 took 0.160s as opposed to the 0.112s. For generating a graph, the T2000 was again slightly slower - 9.538s against 7.167s.

Again you see the single-threaded performance is at about the 60-70% of a regular UltraSparc cpu of the same clockspeed. But, this thing has 8 cores, how well does that work?

So I fired up the apache benchmark ab. And just for the database retrieval report I can get about 30 pages per second out of the SunBlade, but (with 32 concurrent requests) about 100 pages per second out of the T2000. And - according to top - the T2000 is only 50% busy. In fact, my server (apache/php/mysql) setup croaks if I push it much harder (but then it's a reporting thing designed for 1 page a minute).

So, on this test the T2000 is at least as good as 3 SunBlade 2000s. Or,equivalently, it's better than an 8x750MHz V880. Which is pretty good, as it's a fair amount cheaper and smaller.

Wednesday, October 04, 2006

Server Wars: The M2 Strikes Back

And I thought I was getting somewhere!

Having actually installed my X2100 M2 servers, I want to mirror the boot disk.

As an aside, I found Sun Infodoc 83605. This says:

It is recommended to use Hardware RAID controllers for the Root disk mirror, for performance and reliability reasons.


and then a little later, being consistent:

Sun Fire x2100 uses an nVidia RAID controller. Solaris OS doesn't support the nVidia driver as of now, so Solaris Volume Manager is the only option for mirroring the root disk.


OK, so I can't use hardware raid at the present time, which means (I think) that I can't hotswap the drives.

But anyway, I do the regular thing to mirror the boot disk with SVM. Works fine. I installgrub and use eeprom to define an alternate boot path. I reboot both servers. One comes back all fine and dandy.

The other one is dead. Faulty boot archive. Can't start console services. Remember that I'm having a wee bit of trouble controlling these machines, so getting to a failsafe system is a bit tricky. I fiddle for a bit, but rebooting doesn't help.

I was going to reinstall anyway, so pxe boot away. This time, I can't even see the console output. Oh dear, it's getting worse....

Tuesday, October 03, 2006

The M2 submits

As I have previously documented, I've been setting up a Sun X2100 M2.

It's not been the smoothest ride. And I wouldn't say that everything is completely solved, but having got it installed I followed the instructions for serial redirection and I can now ssh to the SP and get console access. That's probably all I need, although I still have a slight lack of confidence in the system - I can imagine ways in which it could get itself messed up where I get no access, but for most of the administration tasks I'll ever need to do I'm sorted.

Still can't get Serial-over-LAN to work, though. Never mind, ssh is good enough for me.

Monday, October 02, 2006

The M2 comes alive

I'm getting somewhere in my ongoing battle with the Sun X2100 M2.

I still haven't got the dratted thing to give me console access. But I have managed to install the beast in such a way that I can finish the install and log in.

The first step is to ensure that you have ipmitool installed. If you have Solaris 10, you may have it, and if so you need to patch it (119764 or 119765 according to your system). Then I can tell it to do a pxe boot next time around:

ipmitool -U root -I lanplus -H 1.2.3.4 chassis bootdev pxe

And I can power cycle it and it will do a jumpstart install.

Then in my finish script I have

sysidconfig -b /a -r /usr/openwin/bin/kdmconfig

which stops it asking me whether I want Xorg or Xsun at boot (questions which I can't answer because I have as yet found no way of typing back at it).

And I have a working system I can access using ssh. With luck I can get the installed system to play nice with the console, which is going to be essential when we move this box into production use.

Friday, September 29, 2006

Me versus the M2

I'm getting a very negative feeling about the SP on the X2100 M2 I'm trying to set up. This thing is just plain awful.

Yes, it's got some very fancy features. Basic functionality like being easily able to access the system console, and being able to operate for more than a few minutes without goofing up, or having documentation that is understandable, seem to be missing at this point.

The basic problem is that I can't get anything sensible out of the serial console. I can tip to the SP, and that works fine. But I just get junk output (if at all) on the console.

The KVM applet gadget is very nifty. And I can see output (once I've redirected to ttyb, anyway), but nothing I can do can persuade Solaris to accept keyboard input. It's just Solaris, as I can type into the GRUB screen OK. But if the install goes interactive I'm stuffed.

Anyway, I've got it installed by persuading it to jumpstart completely hands off. It insisted on asking for terminal type and locale before, although I'm not sure why. (Well, not completely - it complained about not being able to set the boot device and didn't reboot when it had finished installation, but that's OK, I can remote power-cycle it [and I've used that piece of functionality a few times today!].)

So my attempts here were to add:

-b console=ttyb

to the add_install_client invocation, and make sure that terminal and locale were defined in the sysidcfg file, and then edit the menu.lst file to add install as a argument - after:

kernel/unix

I added

-v -m verbose install


Of course, I'm now stuck a little further on - looking at the kdmconfig screen where it's asking me to select an X server. Must find my notes about how to disable that prompt.

Thursday, September 28, 2006

On to the X2100 M2

I've just been trying to set up one of the new X2100 M2s. I did a plain X2100 a while ago, and that was pretty simple - all I had to find out was the appropriate ctrl-alt-meta-shift-escape-thingy to replace F12 for the initial netboot while on the serial port. (Which I've forgotten, so if anyone could enlighten me - thanks!)

The new M2 is a different beast. The first problem was getting to the serial port. I've got my loan T2000 in the rack next to it, so I thought I would simply establish a tip session from that. However, it appears that the normal 'tip hardwire' trick doesn't work with the T2000 - there is no serial port B at all.

So I tried a different machine. And our console server. And about a dozen different cables. No joy. Eventually I did find a cable that worked, but I wasted far too long. (Don't get me started on serial cables. Every time I use them I feel this inner urge to go and throttle somebody.)

So I got an SP prompt. And this thing was - ahem - strange. After 5 minutes or so of being unable to get anything sensible out of it I punted and set the IP address manually, hooked the net management port into the network, and pointed my browser at it.

(I was somewhat dismayed to find that Sun's instructions told me to open up Internet Explorer. Oh how low have we fallen...)

So I point firefox at the LOM address and - wonder of wonders - it worked!

(There was the bit about the certificate expiring in 1979, and the mismatch between the IP address and the name on the certificate, but nothing serious.)

And you get what is actually quite a neat interface at this point. I started up the remote console, and nothing happened. Bother, edit pop-up preferences and try again. And I have the remote console.

This is all actually rather clever. It certainly looks good, and actually works pretty well too.

The next battle is to install the machine. At the moment when it tries to net boot one of our Windows Domain controllers jumps in and answers the DHCP request so my install server doesn't get a look in.

T2000 - initial performance

OK, so I've got the machine running so I thought I would try some simple performance tests.

I know that these aren't going to show the T2000 in a good light. These are simple CPU intensive single-threaded apps. (If you can call them applications.) The aim was to get a feel for just how well the machine would do.

So I have a twin 360MHz Ultra 60, a twin 1.5GHz V240, a quad 1.28GHz V440, an 8x1.0GHz T200, and a 500MHz SunBlade 100 and - for fun - a cheap X2100 with a 2.2GHz Opteron 148.

I copies the /var/sadm/install/contents file from my desktop into /tmp on each machine, and timed grep, wc, gzip, gunzip, bzip2, bunzip2 on the file (it's about 12 Meg). The times, in seconds, are:


U60:
grep 0.540
wc 0.615
gzip 3.055
gunzip 0.619
bzip2 25.589
bunzip2 4.187

SB100:
grep 0.416
wc 0.517
gzip 2.408
gunzip 0.492
bzip2 29.303
bunzip2 4.314

V240:
grep 0.136
wc 0.210
gzip 0.776
gunzip 0.152
bzip2 8.028
bunzip2 1.054

V440:
grep 0.159
wc 0.247
gzip 0.911
gunzip 0.180
bzip2 9.034
bunzip2 1.241

T2000:
grep 0.402
wc 0.656
gzip 2.772
gunzip 0.495
bzip2 17.695
bunzip2 2.285

X2100:
grep 0.079
wc 0.077
gzip 0.445
gunzip 0.092
bzip2 4.053
bunzip2 0.555


What's clear from this is the the Opteron (not entirely unexpectedly) wins by a distance. And the T2000 is handily outpaced by the V240 and V440 - even accounting for clock speed. In fact, the T2000 seems to be - for the completely unfair single tasking case - more comparable to the USIIe/USIIi in something like a V100 or Netra X1.

Of course, once you take into account the parallelism available, the T2000 might be comparable to a whole rack of the old 1U netra systems.

Now to see if some of our applications can be started up on this machine, and if we can test some applications that would suit the T2000 better.

T2000 - install performance

The first thing I did with my loan T2000 was to install Solaris on it. My first impression was that it seemed to be going fairly slowly, which was confirmed by some actual timing numbers.

I can time two parts of the install. The first is the actual Solaris installation, from the begin script to the finish script. This is only a part of the installation process, but is easy to measure. The second is my local installation, which takes place on the next boot, and installs some extra packages, runs some cleanup scripts, untars the whole of /opt/sfw, and applies current patches, and includes the reboot time. I've done 4 different systems this week, and the times (in minutes) are shown below:






Typecpu speedinstalllocalinstall
Ultra 602x360MHz4573
V2402x1.5GHz1640
V4404x1.28GHz1834
T20008x1.0GHz3660


OK, so the install time isn't necessarily a good metric, but it's probably a fair indication of how long it's going to take to do general system administration on such a system. It's also essentially serial, which isn't good for the T1 chip. Even so, the numbers here are slightly disappointing - it's doing slightly worse for it's clock speed compared to the other sparc systems I've got available to play with today.

Wednesday, September 27, 2006

T2000 - past the first roadblock

Just got past the first roadblock with my loan T2000.

Plugged it in, powered it on, and got no network connectivity.

That's odd. All the lights look fine on both the T2000 and the switch. But I can't get the link to come up properly. I get

Timed out waiting for Autonegotation to complete
Check cable and try again
Link Down

So I pull it out of the foundry switch and into a cisco. And then I get

100 Mbps full duplex Link up

OK, it's a shame that I can't get it working at gigabit but at least I can jumpstart it now, and I'll worry about getting it running at full speed later.

Tuesday, September 26, 2006

New Toys

I've been racking some new kit today. A couple of simple boxes for mail relays (and spam filters), and a backup server.

We've gone for the new Sun X2100 M2 for the mail systems. Something lightweight, but fast to handle the load. And very affordable at that.

We got an X2100 a while back, and it's been a huge success. We run our trouble ticket system on it, and it's not just faster - it's revolutionized the way we work. Before, doing anything with the trouble ticket system was so painful that it discouraged anyone from logging or handling tickets. Now, it just flies. We're hoping for the same from the mail system, which (and I guess everyone is getting the same experience) is getting swamped by the spam deluge.

The backup system is nothing special, I've put in a V240 with a C4 library. The only thing to say about this is that the C4 library is a deep beast - it barely fits into the Sun 900 rack and that's without being cabled up yet!

And I was just sitting down after that when a try-and-buy T2000 showed up. That's in the rack too, and it all looks very impressive.

We're interested to see how well the coolthreads system works. In general, I see increasing use of opteron systems, but we still have a number of sparc based applications that we're going to have to run. It stands a good chance - apache, tomcat, with a database or proprietary search engine at the back - so we're optimistic it'll be a success.

Tuesday, September 05, 2006

Solview Updated

I've updated my solview utility to version 0.3.

What is solview, I hear you ask? It's a little java gui to show useful information about a Solaris system. It shows installed software packages (and installation software clusters); services and their status; and the output from various informational commands.

This new version enhances the package and cluster information. It's split up into several tabbed panels, and now shows cluster membership, the installed status of clusters, and can calculate full dependencies.

What happened to version 0.2? Well, that version included a rather spectacular failure to enhance the software display by parsing the contents file. I expect it is possible to get java to extract useful information from this file without running out of memory, but I didn't succeed in doing so. So that attempt got scrapped and I moved on to version 0.3.

Sunday, August 20, 2006

How hard can it be?

Yesterday I tried to write a CD on the family PC. I've been taking digital photos almost exclusively for the past couple of years, and we went through the accumulation and decided to get a few printed off properly. Simple - just burn them to a CD and take it to the nearest supermarket.

So the family PC has a writable CD drive, and a popular user-friendly operating system that's supposed to make this a breeze. So it should be a doddle, right? Wrong! I go to my photos and click where it says "Copy all items to CD", and failed completely to produce a CD with any pictures on. Several coasters, for sure, but no viable CDs. (My understanding of the way this is supposed to work is that it just copies the files to a holding cell, and when you open a writable CD it's supposed to ask you if you want to burn the CD. Nope, just trashed the CD every time.)

The PC has a number of utilities, that either came preinstalled, or as part of some other package, that allow you to write to a CD. None of those did the job either (I know the thing works, because I've written backups or VideoCDs, but I wanted something terribly simple. No dice.)

Unwilling to waste any more time, I simply copied the files to a real computer running Solaris. A couple of simple commands later and I had burnt a CD, first time, no problem.

Oh, and the pictures came out great.

Friday, August 18, 2006

Back in Circulation

I haven't disappeared. I haven't given up. I'm still alive.

The last month has been pretty hectic for me. It being summer and the kids having time off school, we took some holiday.

Also, as I mentioned earlier, I've changed jobs. I'm now working for Proquest Information and Learning, looking after their Sun Solaris servers. Half the first month was spent on holiday (arranged long before, and Proquest were happy with that - they were keen for me to make a start as soon as possible rather than wait until after my vacation). So far, I've been enjoying myself enormously. It's a big challenge and an environment rather different to what I've been used to, but I'm starting to get my teeth into it. The lifestyle is much better, and I've been cycling to work quite a lot (although, if you read the holiday story, it's not as if I have much choice at the moment).

Sunday, July 30, 2006

Self Preservation

It's been unpleasantly warm here in the UK over the last month. I'm sure other places have it hotter, but we're not used to it and not geared up for it.

I have some old work colleagues who I set up some systems for, and I got an urgent phone call just before going on holiday: "the air conditioning failed yesterday and the power in the data centre was unceremoniously yanked - could you check the systems out?".

It turns out the servers - Sun machines - were fine. With a fine sense of self preservation they had shut themselves down gracefully due to the heat:

Jul 19 18:00:14 foobar rmclomv: [ID 690426 kern.crit]
SC initiating soft host system shutdown due to fault at MB.T_ENC.
Jul 19 18:00:15 foobar rmclomv: [ID 222544 kern.error]
SC Request to Power Off Host.
Jul 19 18:00:16 foobar rmclomv: [ID 428258 kern.crit]
TEMP_SENSOR @ MB.T_ENC has exceeded high soft shutdown threshold.

So far, all appears to be well, and I haven't seen any further errors.

Thursday, July 13, 2006

Enhancing the JDS desktop look

While the JDS desktop that currently comes with Solaris is essentially functional, I've always found it rather bland. In one sense it's a good thing - it has no major irritations (most window managers or desktop environments have something that drives me up the wall).

Still, I would like to get a bit more character out of the desktop. Unfortunately, the available themes are pretty thin on the ground. Unlike the huge array of themes that you can get for Window Maker, you're really very restricted.

I actually quite liked the new theme that the latest JDS builds come with, but it doesn't seem to be separately available.

I've found a couple of themes that I like. I've used eXPerience for most of the last year. I've got my Windows XP box set to the olive green theme, and find that color combination works quite well for me. And I've just been playing with Clear Blue, which works pretty well. It's a bit pale on my current monitor, and I might have to tone down some of the other applications I use to blend in a little better, but it's looking good.

There's still a lot of room for more gorgeous looking desktop themes, though.

Wednesday, July 05, 2006

S10U2 companion problems

Recently, the update 2 of Solaris 10 was released.

Along with this comes the companion CD - a collection of unsupported freeware.

However, the S10U2 companion DVD has a number of problems. Apart from the old one of being distributed as a single ISO image with both sparc and x86 binaries, so that you have to download both architectures even if you only wanta one, some new problems have appeared.
  • There isn't an installer. THere are over a hundred packages here and installing them individually by hand can get a bit tedious.
  • Worse, some packages depend on packages that are no longer shipped. I had to install SFWgcc2l, SFWgcc34l, SFWgfile, SFWgtext, SFWshutl, and probably SFWgm4 off the S10U1 companion to get the dependencies and missing components back.
  • Some useful utilities (some of them listed above, but also things like bluefish) have just disappeared.
  • Coreutils has been added, without the g prefixes.

For all that pain, is there any gain? I can't see any. I can't see any new packages that add any value, we've lost functionality, the installed packages don't work, and it's not even as if we have anything useful like a more up to date version of KDE.

I recommend going back to the S10U1 companion.

Thursday, June 22, 2006

#opensolaris

One part of the OpenSolaris first anniversary (or birthday) last week was an IRC chat party.

I'm not entirely sure I'm fluent with IRC yet. I used to chat interactively 15-20 years ago, but haven't done much since. One problem is that I get interrupted or distracted quite a lot. This isn't much of a problem for email discussions, but it's rather easy to lose the thread of a chat conversation.

First couple of times I used the builtin mozilla client. Just tried the ChatZilla extension for Firefox, which is basically the mozilla client, and it works fine.

Wednesday, June 21, 2006

Solaris Install

Mark Mayo lists his top 10 Solaris Installation Annoyances.

I'm not going to try and pretend that the Solaris installer isn't anything other than bad, but consider the following:

I've done thousands of Solaris installs over the last decade or so. I can probably count the number of times I've used the interactive installer on the fingers of one hand. No, I didn't like it much either.

The fact that most admins with a clue use jumpstart means that the interactive installer has received precious little attention. Even if the interactive installer was the best thing since sliced bread, I would never use it at work. That's not to say that the interactive installer can't be improved - it can, and must be.

I've not used a modern Linux installer (say, Ubuntu) but have used RedHat, Mandrake, Suse, Fedora. And I wouldn't necessarily say they're actually much better. Sure, they might look prettier, and feel slicker, but were often fragile (installer just bombs out) and unreliable (system all messed up afterwards). The Solaris installer might be butt-ugly and dog-slow, but produced working systems if you were prepared to suffer to the end. And if people are using interactive installers professionally, they need to get with the program and adopt some automatic provisioning scheme.

1. As for why it's slow, some work has been done on this. The biggest problem is that the distribution is compressed with bzip2. It goes way quicker if you swap out bzip2 for gzip.

2,3,4. Never happen if you use jumpstart.

5. I would argue against any attempt to change the default shell to bash. (Now, if they suggested tcsh, that might be received more favourably.)

6. Err, who on earth logs in as root?

7,8. We ought to chuck vi and vim and replace 'em with emacs...

9. Urrgh. Sendmail.....

10. Secure by default has finally made it into Nevada. But automation under jumpstart would normally knock out services you don't need.

Opera 9, Solaris x86

Not only is there a new version of Opera, but it's available for Solaris x86 as well!

Hooray!

An unfair comparison:

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
107 ptribble 4 49 0 1024M 971M sleep 219:15 1.75% firefox-bin
25551 ptribble 1 49 0 64M 52M sleep 0:07 0.02% opera

I still prefer firefox, but that might change. Don't know why, it just seems to work for me better than other browsers. One thing that opera does have that's useful is the ability to duplicate a tab. And I am getting rather fed up of having to restart firefox every day or two to keep memory utilization within bounds on my workstation (or cpu utilization down on the XP box at home).

Thursday, June 15, 2006

Moving On

Not as newsworthy as Scoble, but I'm changing jobs soon.

I've been in my current job for about 9 months, and it's been tough. I've never really settled properly into the organization, and there have been a lot of frustrations that I haven't been able to identify properly or address. Combine that with spending between 2 and 3 hours a day commuting, and it wasn't really a situation I wanted to continue in.

Last week got fairly frantic. I saw an advert for an attractive position with a company only a few minutes from home at the beginning of the week. Sent off my CV, got called by the agency, went for interview, went back for a second interview the next day. I liked what I saw, and they obviously liked me enough to offer me the post, so I start there in about a month's time.

I'm looking forward to it. And not just the job - 2 hours of my life back every weekday is going to make a huge difference.

Wednesday, June 14, 2006

Striding Out

A year ago today, OpenSolaris was launched and started taking its first baby steps.

Over the last year, a lot has happened. More code has been released. We've had lots of lively discussions. Many projects have started. OpenSolaris distributions have been created. There's been serious progress on building a Source Code Management system that fits the needs of Sun and the community. Members of the outside community have been filing bugs and contributing fixes.

Progress in some areas has been slow, but that's fine. We've been learning how to walk before we can run, and we love Solaris too much to risk running too fast and falling over and breaking anything. But the project is gaining strength and starting to stride out more confidently.

For my own part, I've been involved in the community since the early days of the pilot. Job issues meant I had to back off on the code contribution front for a while, but I've recently managed to get some fixes put back, with more on the way. This work seems to have been picked up by ZDnet Australia and even a Sun feature.

So we've had a good year, and I feel privileged to have been allowed to be a part of it.

, here's to many more successful years!

How fast is my SCSI?

Discovered a quick way of working out at what speed SCSI devices are connected to a Sun server at.

This only works for some SCSI cards - it seems to be a bit hit and miss. I've generally had some success on my SPARC workgroup servers, but other machines give zilch.

And the actual data transfer rate is something else again. But at least you can check that things are being set up at the rated speeds.

The command is:

prtpicl -v | grep sync-speed

and you then have to work out what the devices are.

Some examples. Here's a V240 with an attached 3310 array:

:target0-sync-speed 160000
:target1-sync-speed 160000
:target2-sync-speed 160000
:target3-sync-speed 160000
:target0-sync-speed 320000
which shows the intrenal drives at Ultra160, and the external array at Ultra320.

Here's a V440 with a pair of 3120 arrays:

:target5-sync-speed 5000
:target8-sync-speed 320000
:target9-sync-speed 320000
:targeta-sync-speed 320000
:targetb-sync-speed 320000
:target5-sync-speed 5000
:target8-sync-speed 320000
:target9-sync-speed 320000
:targeta-sync-speed 320000
:targetb-sync-speed 320000
:target0-sync-speed 320000
:target1-sync-speed 320000
:target2-sync-speed 320000
:target3-sync-speed 320000
all the disks are at Ultra320, and you see an SES target on each array that's come in at 5M.

Here's an old E450:

:target0-sync-speed 40000
:target1-sync-speed 40000
:target2-sync-speed 40000
:target3-sync-speed 40000
:target6-sync-speed 20000
:target1-sync-speed 10000
which has the disks at 40M, the DVD at 20M, and the tape at 10M.

Friday, June 09, 2006

Ultra 25

So where did the Ultra 25 come from? I don't recall any product announcement or big splash.

Still, that's not much of a surprise really, as the machine doesn't look too impressive. Only a 1.34GHz chip, which is slower than the SunBlade 1500 it presumably replaces.

I also note that the operating systems it supports include

Solaris 9 (Available September 2006)

Does this mean that there's going to be a new hardware release of Solaris 9 in September?