Thursday, May 11, 2006

MyISAM vs InnoDB

I've been using MySQL for the database componenet of a number of projects over the years.

Usually, I've used the MyISAM storage engine. It's fast (that's the reason for using MySQL in the first place), and generally reliable.

MyISAM isn't robust against system failure though. System crashes, reboots, and power cuts tend not to be handled very well. (Yeah, I know, they shouldn't happen in the first place, but this is the real world.)

I don't need the transactional capabilities of InnoDB (from the functional point of view even MyISAM is overkill for most of what I do), but something more robust would help.

So I thought I would do a quick check of the impact of using InnoDB on the system. This isn't a benchmark, it's completely unscientific, and all that, but it told me what I needed to know.

Running InnoDB, one minute's worth of disk activity looks like:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 7.0 0.8 34.1 0.0 0.1 1.6 12.4 1 5 d0

whereas with MyISAM it looks like:

r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.1 2.3 0.8 3.0 0.0 0.0 3.5 14.5 1 1 d0


OK, so InnoDB generates 3 times as many writes, and 10 times as much data transfer, as MyISAM. And the mysqld process consumes correspondingly more cpu time (although it's still very small - much less than 1% of a processor), so the system load average is a bit lower with MyISAM too.

I don't think this rules out InnoDB, although it does indicate that there is a significant cost to changing, and to scale up by a factor 10 (which I'm going to need to do, and then some) would likely have problems if I was using InnoDB. If I go down that route, I need to do more optimisation of the system design and the database client interactions.

Wednesday, May 10, 2006

Sinking ship?

Is this the end for SGI?

It's been a long painful slide, but inexorably downwards, and I can't really see any way back (short of a government funded rescue package).

I remember sitting through a sales presentation from SGI some years ago. We were told of their grand plans to throw away their well-respected workstation technology and become a shifter of (not quite standard or compatible with anybody elses) PCs running Windows NT; to throw away IRIX and adopt Linux; and to throw away their own RISC chips and go down with Itanium. (Itanium wasn't even shipping at the time.)

We regarded this as doubly suicidal. Not only were the products entirely unattractive to us (as existing customers with investments in their hardware and software platforms, incompatibility is a big turn-off), but we could see the plan failing and were thus reluctant to invest in products from a company that - in a single presentation - had gone from a front-runner to certainly doomed.

Sunday, May 07, 2006

Outdated libraries

Every so often, the ugly interaction of Solaris and the undisciplined open software world completely breaks a project I'm working on.

Latest case in point - libxml2. Solaris ships a horrifically antiquated version (2.6.10, to be exact) with Solaris 10 (and nevada, currently). Snag is, PHP requires 2.6.11 or later and won't build.

There are a couple of bugs open: 6362386 and 6193876. Not to mention 6378879.

It's been this way for 16 months. Surely time enough for something to have happened?

I realise that, because certain core components of Solaris itself rely upon libxml2, it cannot be upgraded without due care. If that is the case, then those components should use a private, compatible, copy, and allow the publically available version to be kept reasonably current.

Unfortunately the download site appears to be down at the moment, so I'll have to grab the source and build my own up-to-date copy tomorrow before I can make progress.

It's not as if libxml2 is the only external component that is somewhat antiquated. In fact, quite a lot of it is getting sufficiently old as to be useless, and if Solaris is going to be used as a platform for other open-source applications then some serious updating is going to have to be done.

Great Open Source

Looked fairly innocuous to start with, but this blog on ZDnet points to a list of the top 50 open source projects. Check it out - there are some real nuggets, and starting points for further exploration.

Friday, May 05, 2006

Solaris to/from Linux?

So Matty gave some reasons why people might consider switching from Solaris to Linux.

This is all a matter of opinion, of course. Despite computers being binary systems, everything in IT is shades of grey (often very dirty shades of grey). So here's my take on Matty's points, in the same order:

1. Integrated LAMP stack. Actually, all my Solaris boxes could have a full SAMP stack installed, but I would never use it. I wouldn't (and didn't) use it on Linux either. In both cases I would much rather install the application stack separately. It's very much safer that way - not only can I be sure that all the components are at specified known levels, and are the same across platforms, but I can be sure that my OS vendor isn't going to screw me around. I always had to wipe all evidence of apache, mysql, and friends from a RedHat system in order to get operational stability; unfortunately it appears that Sun are heading down the same misguided path by bundling more services in Solaris.

2. At my previous place of employment, you could always tell when the next version of Fedora had been released - all the group of developers were surly and miserable for a week because there desktop had been randomly rearranged and they had to relearn it. I reckon each of them lost a week's productivity as a result. Sun aren't much better - open source software doesn't care for operational stability, and every time Sun update Gnome or JDS in Solaris nothing works. And I'm now convinced we're going backwards. (As an aside, I've had many more problems with desktop apps not working right on Fedora than Solaris, but that may have been because the Solaris versions didn't try to push the bleeding edge so hard.)

3. The JES stack deserves a good kicking. My experience of it has been woeful. So I agree with that part. The other side of the coin is regular updates. As I said in point 1, I don't want my OS provider to randomly modify the working of key applications. So I wouldn't use either of them.

4. My experience of ISVs is that they hate supporting Linux. And I can't blame them - the qualification effort is horrendous. I asked one the other week about Solaris x86 support, and their answer was "we keep getting asked about that, but it's not on our roadmap". Anothe one went "phew!" when we said we were going to deploy on Solaris - he was then confident it would work out of the box. But, yes, there does seem to be more visibility for Linux - I suspect vendors think that's what customers want, when most of the time any alignment being customer requirements and vendor offerings is random at best. At least with Solaris, if it works it works, rather than having to stick to one particular release of one particular Linux distribution with a specific set of patches.

5. Not so; managing applications and patches on Solaris is an absolute doddle. Provided you ignore completely Sun's perpetually failed attempts to improve the process - patchpro, smpatch, prodreg, update connection are all worthy only of the dustbin. Fortunately it's trivial to write your own tools, or use pkg-get or pca which are massively better than anything Sun have come up with so far.

6. Why not upgrade? Modulo bugs in occasional releases (and these are the sorts of bugs that mean that not every build qualifies as a regular Express release) regular upgrade or live upgrade work fine. There's no need to futz with bfu unless you really want to.

7. At least with Solaris you have the option of zones! Are zones a universal panacea? No. But they are enormously useful for a whole range of operations where you need to consolidate or isolate services. I can believe smpatch update could take days, but then I wouldn't do it like that - I wouldn't use smpatch, for starters, and I wouldn't create the zones and then apply the patches. For this sort of thing our mode of operation was to simply migrate services onto a zone on a different box, and then rebuild machines, if the patch overhead was too great. (Based on the philosophy of never patching or rebooting a live service.) The fact is that there's no getting away from the fact that by creating 25 zones, you've increased the update cost 25 times. It is certainly true, though, that there's room for improvement in the patch tools. They are significantly slower than can be accounted for by the actual work they're doing.

8. It's a tricky balancing act. And I thin Sun have done pretty well with OpenSolaris. There was massive concern that the open source process would destroy Solaris' core strengths, and it doesn't seem to have done so yet. My hope was that it would address some of Solaris' weaknesses, and I don't think it's actually done that yet, but the level of engagement is there, and the signs are promising. To be honest, I'm not sure I would want to use an OS that would allow outsiders to put code back to the kernel source tree - does Red Hat allow me to modify their kernel source?

One thing that worries me about end-user putbacks is that the whole thing has to be vetted and managed, and that has to be done by Sun (because their position is that if it goes into OpenSolaris then it goes into Solaris - there is only one codebase), so it costs Sun more to have a community member fix something than to have one of their own engineers do it. Given the financials of the past few years, this doesn't strike me as an optimal situation.

There are some good points here - it's not as if Solaris hasn't got any weaknesses. The real killer is the availability of commercial software. If you're only offered it under Linux then you're going to have to put up with increased costs in terms of licensing, support, hardware, and staffing (or just pretend they don't exist), or choose another product. But there are many places where one person's ideal is anathema to someone else, and we just have to accept that there is no one solution to every problem.

Thursday, May 04, 2006

Well, that didn't work...

I notice that Eric made a decent comment about my recent rant on the decline of the desktop.

Now, I've tried xfce a couple of times in the past, but it has similarities to CDE that have always put me off. But I've never tried fluxbox (although I did try blackbox back in the day).

So I thought it deserved a try. Downloaded and installed it (which was fine apart from the usual problems).

I started it up, and everything seemed unusually sluggish. That's odd, this is supposed to be light and fast. I then made the mistake of clicking on the background and selecting 'About'. This sent the whole thing into a tailspin. 4G of memory usage later and my machine is like stuck in treacle.

Worth a try, but I think I have to write that one off.

I'm leaning back towards windowmaker. (Anyone know which black hole the windowmaker website has disappeared down at the moment?)

Wednesday, May 03, 2006

tuxpaint

One of my girls came home from school yesterday raving about Tux Paint.

It looks pretty good stuff to me. (The fact that one of my girls is mad about penguins might also make it popular.)

They're using it on Windows at School, but it's cross-platform, so I built it for Solaris too.

I had to go through the prerequistes:
In all cases I did a

(setenv CC cc ; setenv CXX CC ; setenv F77 f77 ; ./configure )
gmake
gmake install

to build, using the free Sun Compiler, as libtool couldn't cope with gcc for some reason.

Then to build tuxpaint

gmake CC=gcc


This wasn't quite enough. I needed to add an explicit

-lpng -lsocket

to the link list, and then it fails because it needs strcasestr().

OK, so I grabbed a shim copy of strcasestr from OpenSolaris (for example, this one), compiled it, and linked that in.

And, because it used a simple makefile, fixing up the problems was trivial.

Oh, and I've filed a couple of bugs. One against tuxpaint so that it handles a missing strcasestr (or doesn't require it at all); and an RFE that Solaris include this function (as opposed to the several private copies dotted about in the source), as this isn't the first time I've seen applications want it.

Saturday, April 29, 2006

Wither the desktop?

It's a personal thing. My desktop is a virtual space in which I work, play, and socialize. As such, I want it to work my way.

This covers several aspects: the way it looks (colour scheme, imagery); the way it behaves (in response to my requests); what it does (which programs are available).

In unix-like environments, the desktop was originally a number of independent windows looked after by a window manager. This has evolved into the (more or less) tightly integrated desktop environments such as GNOME and KDE that we have today.

In years gone by...

Some of the early window managers were fairly basic. I remember wm and uwm, but I really liked the old Ardent window manager. It was friendly and very customizable.

Time went on and the original generation of window managers bit the dust. It seemed that twm was the new standard. I never used twm itself much, as I was starting to run into problems caused by having too many windows open. So I ended up using tvtwm
instead.

Customizing tvtwm was fairly easy (assuming you're happy to edit configuration files) and the degree of customization available was quite extensive. You had complete control over all the mouse button events, for example. I defined all my own menus. And you could define your colour scheme extremely precisely - I had the window decorations for each type of window in different colours, so I could more easily spot a particular application on screen.

Then along came desktop environments - a window manager with an associated set of libraries, and a (more or less) complete toolset. Open Look and CDE were the well known ones, although I'm sure there were others.

However, I found one thing in common with both Open Look and CDE. I hated them both. Utterly and completely. They both forced me into unnatural and counterintuitive ways of working, and I can't stand either of them for more than 5 minutes.

We're now into the brave new world of Gnome and KDE. Again, the aim is a complete and all encompassing desktop environment with a set of libraries and a complete set of applications.

Unlike Open Look and CDE, I've found I can tolerate both KDE and Gnome. Sure, they're slow, and both have irritating features, but both are good enough that I don't start swearing at the screen after a couple of minutes.

Have we really made progress in the last decade? I'm not sure we have. Some of the applications we have now certainly have more functionality than was available 10 years ago, but I don't see that they are necessarily better suited to today's problems than the applications of 10 years ago were to the problems of the time.

Put it another way. Ten years ago the applications and environments I had available met my needs of the time more than adequately. That's no longer true. And it's not as if my requirements have actually changed all that much - certainly less than the overall computing landscape has.

I'm not optimistic about the future. I was less than impressed with the Gnome-based JDS that will shortly go into OpenSolaris. I see very little of interest, and a lot of regressions.

What I also see is a lack of variety, a lack of excitement, no spark. Coupled with an increasing inability to do the basics, and the desktop is withering away.

When it comes down to it, I'm not actually asking for very much. I want to be able to set the focus policy, define the actions to be taken on mouse clicks, define what menus appear, and under what circumstances, and with what contents, and define the shape, colour, and imagery of decorations. And then use the applications that I want.

The desktop environments appear to have become less customizable, not more. Consider the available themes. There are thousands for WindowMaker (and some of them are quite decent). How many Gnome themes are there? Now, OK, a WindowMaker theme doesn't really do very much, but it does what you want.

It's not as if the desktop frameworks have provided a solid foundation on which to build better applications, either. Most of the standard applications that ship are pretty poor. I would much rather have a dedicated window manager and ally that with best of breed applications than have a bunch of applications that happen to be built using the same toolkit.

In summary, I feel that desktop development has headed off down a cul-de-sac, and we need to get back on the main road to make real progress.

Thursday, April 27, 2006

More distractions...

As if I didn't have enough distractions, I've been playing with Google SketchUp.

(And yes, this means on the Windows box...)

I've used various CAD and 3-D programs over the years (and the majority of consumer ones felt like they were just some third-rate CAD lookalike), and they were pretty hard work. SketchUp is brilliant - so easy to build a model.

Wednesday, April 26, 2006

Die, configure, die!!!

So I'm building a new version of Eterm and, as usual, the ./configure script is getting in the way.

First it couldn't find libast. That was legitimate - so I install libast and expect all to be well.

So now it constructs a compiler command line that gcc chokes on.

That's fine, I tell it to use cc instead and it can't find libast. The reason is simple - it's putting the -last ahead of the -L switch to tell it where the library is.

So I fix that and the compile just blows up. Looking more closely:

cc -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/local/i clude
-I/usr/local/i clude -I/usr/ope wi /i clude -O -c
actions.c -KPIC -DPIC -o .libs/actions.lo

Look closely: it's whacked all the n's out of the include directories.

So I randomly shuffle my PATH (presumably it's found a sed it doesn't like, but isn't that the sort of thing that autoconf is supposed to solve?) and eventually get it to work.

Won't compile. The code simply isn't liked by the Sun compiler.

Ho hum. Back to gcc - let's try a different version. OK, so I get it to configure and make, but it still fails, this time with an Undefined symbol error.

The upshot of all this is that autoconf has just wasted 10 minutes of my time and completely failed to produce a successful result at the end of it all.

Sadly, this is becoming the norm.

Tuesday, April 25, 2006

Retro gaming

There aren't enough hours in the day.

I'm trying to wrap up a bunch of contributions to OpenSolaris, and have managed to get some of them submitted, but I find myself getting distracted.

I just got Pro Pinball: Timeshock for PSOne off eBay (and that has to be the best Pinball simulation ever) so I'm spending a little time on that.

I just got the PS2 Atari Anthology and it's a blast. I remember most of the games. The great thing about having it on the PS2 is that it doesn't cost a fortune to play! The downside is that the controls don't always translate to a console gamepad all that well so some of the games can be quite tricky to play.

And then I've been playing some of the old ZX Spectrum games on World of Spectrum, using the Java emulator.

Gotta go, there's a hi-score to beat...

Sunday, April 16, 2006

Graphical DTrace

So the Chime Visualization Tool for DTrace is now available.

I'm itching to try this out, and to see how you can build bridges with kstat (by means of jkstat or something different).

Unfortunately there appears to be a network problem with my test machine. Or more specifically there's a problem somewhere in the path between where I'm sat and where it's at. I've managed to determine that the machine itself is fine, but doing any work on it is impossible. It being a holiday I don't expect it to be fixed for a day or two.

Wednesday, April 12, 2006

Extra Software

No operating system - no matter how good - comes with a complete set of every piece of software you're likely to want. There are always cases where an end-user needs to add additional software to meet specific needs. Corporate servers need business software; developers may want to live on the bleeding edge.

I used to maintain a lot of stuff myself. But this has become harder and harder over time, as build systems become more complex and less intelligent, and dependencies becore more entwined and harder to resolve.

One place I now use to keep stuff up to date - and to try software out easily without having to invest the effort in building the whole dependency tree from scratch myself - is Blastwave.

It's very use. Install pkg-get and go install. It grabs the software you want and installs it and any dependencies you need. All simple and painless.

Of course, what it also shows you is how bad this dependency tracking gets. I wanted to try out a couple of pieces of software this afternoon - dia and enlightenment as it happens - and off it went, installing package after package. It's a lot easier than doing it myself, but complexity is on the rise and I'm not sure how close we are to total meltdown.

Install investigations

There's been a reaonable amount of discussion recently regarding Solaris installation speed. Indeed, there's even some idea of what the problem is.

However, closer investigation reveals that rewriting the contents file isn't the limiting factor in a Solaris install. It's not even in the top 3, which are:
  • The time taken to uncompress the data (they're bzip2 compressed)
  • The time taken write Solaris to the disk
  • The SMF manifest import on first boot

Currently, the contents file rewrite is having a race with the pkg tools overhead for 4th place.

Why this disconnect between the obvious problem - and it is a problem - of rewriting this large file many times and its relatively minor importance to Solaris install times? After all, for a slightly trimmed full install Solaris itself accounts for 3.5G of writes, and rewriting the contents file is twice that.

The point is, though, that rewriting the contents file involves a small number of large writes, which are quick. Solaris itself is many small files, so it generates something like 100 times the number of I/O operations.

Not only that, but it's possible to tweak the package installation order to minimize the amount of rewritten data. Simply install all the small packages early and leave the big packages that bloat the contents file until last. Doing so could - in principle - reduce the contents file rewrites by an order of magnitude.

This affects zone installs as well. For zone install, the uncompress cost doesn't exist at all. And for a sparse root zone, there's no Solaris to write to disk - it's loopback mounted from the global zone. So the contents file is much more important as a limiting factor for zone creation performance. However, I've managed to halve the contents file rewrites by tweaking the package installation order. I've not got that much control over the installation order, as it seems to depend on both dependency calculations and the order that opendir() goes through /var/sadm/pkg, but even then a gain of a factor 2 was fairly easy.

This isn't to say that the management of the contents file isn't an interesting and important subject that can lead to some benefits, but the relative importance of it in install performance can easily be substantially overstated. There's other low-hanging fruit to have a go at!

Monday, April 10, 2006

./configure is evil

For years I've used emacs as my editor.I don't want to get into religious wars here - if you want to use vi, then that's fine too. I happen to like emacs because I started out with EDT and TPU under VMS, and when I moved off the VAX I had to find an alternative - and I was able to make emacs play ball pretty easily.

I don't use an IDE, or indeed any custom authoring tools for that matter. I write web pages in emacs, as plain HTML.

Si I was glad to find out that I'm not entirely alone. Bill Rushmore just wrote about Emacs as a Java IDE. Spurred on by this, I thought about upgrading the version of emacs that I use.

I've been stuck on GNU emacs, version 19, for a very long time. The reason I haven't upgraded to later version is that they're too darn slow. (The same goes for XEmacs.) Which is one reason for not using an IDE - an editor has to start faster than I can start typing.

I now have a dual Opteron desktop, so it should be possible to get the latest version to start up fast enough, right?

I don't know, the darn thing won't build.

So I download emacs, unpack it on my Solaris box, see what configure options I have, and type:

./configure --prefix=/usr/local/versions/emacs-21.4 --without-gcc
--with-xpm --with-jpeg --with-tiff --with-gif --with-png

Why without-gcc? I need this to go real fast, so I want to use the latest studio compiler and crank it up.

It fails. No xpm, png, in fact no nothing.

configure:2694: /usr/ccs/lib/cpp conftest.c >/dev/null 2>conftest.out
"/usr/include/sys/isa_defs.h", line 500: undefined control

Well, there's a fundamental problem. The configure script is trying to be too clever by half, and is calling cpp directly. Don't do that. Run it the same way the compiler does, and things will work much better.

So I tell it to use "cc -E". This gets it past some of the cpp stuff, but there are two new problems that crop up:

configure:5430: cc -o conftest -I/usr/openwin/include - g - O
-I/usr/openwin/include -L/usr/openwin/lib conftest.c -lpng -lz -lm
- lX11 - lkvm - lelf - lsocket - lnsl - lkstat 1>&5
ld: fatal: file g: open failed: No such file or directory
ld: fatal: file O: open failed: No such file or directory
ld: fatal: file lX11: open failed: No such file or directory
ld: fatal: file lkvm: open failed: No such file or directory
ld: fatal: file lelf: open failed: No such file or directory
ld: fatal: file lsocket: open failed: No such file or directory
ld: fatal: file lnsl: open failed: No such file or directory
ld: fatal: file lkstat: open failed: No such file or directory

There's a major thing wrong here. Why has it randomly decided to put spaces in? (And why does it think it needs libelf, libkvm, and libkstat just to see if it can find a png function? I'll let it off some of the other libraries, as although it doesn't need to specify them all there were times in the past when you had to.)

That's not all:

"junk.c", line 586: invalid input token: 8.elc
...

So it looks like it's trying to use the C preprocessor in different ways.

Foiled again.

Really, I can't see why anyone would think that getting a configure script to make a bunch of random guesses (and often get them wrong) is anything other than a stupid idea. There was a time when unix variants differed dramatically and you needed a way to tell them apart, but that time has long gone. As it is, we've now got to the point where the amount of software that compiles and works is heading towards zero, and the worst thing is that - unlike in the old days when it would have taken a couple of seconds to fix the Makefile to make it work - actually correcting the error when it goofs up like this is almost impossible.

Wednesday, March 22, 2006

poking around with dtrace

Having released jfsstat, I've been using it on my test system just to see what would come up.

I left it running looking at file accesses - lookup, access, pathconf, readlink - and started to notice some odd patterns.

The first was that /proc was getting a very steady 25 lookups and 12.5 accesses per second. OK, that's easy - I had top running (refreshing every 5 seconds) and this system has 60 processes on it, so every refresh top is doing 1 access and 2 lookup per process.

The next was that / (and this machine just has the single filesystem) was seeing about 600 lookup and 10 access requests every 10 seconds.

The third part of the pattern was that roughly every minute there was a veritable storm of operations on the / filesystem.

Now, one of the longer term aims of my writing tools like jfsstat and jkstat is that you see some strange activity, and you can click on the activity there and then and drill down deeper - probably with dtrace. I'm not there yet, but I thought I would practice my dtrace skills and do it by hand.

OK, so what I'm after is ufs lookups. What can dtrace give me?

% dtrace -l | grep ufs| grep lookup
536 vtrace ufs ufs_lookup TR_UFS_LOOKUP_END
537 vtrace ufs ufs_lookup TR_UFS_LOOKUP_START
17836 fbt ufs ufs_lookup entry
17837 fbt ufs ufs_lookup return

Right. Let's have a look at what's generating those lookup requests:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

java 612

So, that 10 second repeating burst of 600 lookups is java. In fact, I know this to be tomcat.

Now I run the script again to catch the massive burst of activity that happens every minute:

% dtrace -n 'fbt:ufs:ufs_lookup:entry{@[execname]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

sched 24
pwd 28
sh 75
fping 91
server.sh 112
uname 150
mysqld 162
ping.sh 238
nscd 491
cron 498
init 988
load.sh 1014
rup 1539
java 1836
perl 2120
awk 2394
mysql 63666

I've hit the tomcat activity burst 3 times, but the once a minute is coming from something launched by cron - a system monitoring script that runs fping and rup and pokes the results back into a mysql database. But what on earth is mysql doing making 63666 lookup requests?

(The first question I asked was - is this one instance of mysql or many? If I aggregate on pid as well as execname then I see that I'm running a lot of copies of mysql, each of which generates 786 lookup requests on the filesystem.)

Next question: what are the pathnames that are being used in the lookup request? To get this, I need to understand the ufs_lookup call itself in a little more detail. So the source tells us that the 4th argument is a struct pathname, and the string I'm after is the pn_buf member. So let's see what pathnames are being looked up. First the little tomcat burst:

dtrace -n 'fbt:ufs:ufs_lookup:entry{@[stringof(args[3]->pn_buf)]=count();}'
dtrace: description 'fbt:ufs:ufs_lookup:entry' matched 1 probe
^C

/opt/XSload/tomcat/conf/Catalina/localhost 6
/opt/XSload/tomcat/webapps/ROOT/META-INF/context.xml 6
/opt/XSload/tomcat/webapps/jsp-examples/META-INF/context.xml 6
...
/opt/XSload/tomcat/webapps/xsload/WEB-INF 18
/opt/XSload/tomcat/webapps/xsload.war 20
/opt/XSload/tomcat/conf/Catalina/localhost/host-manager.xml 35
/opt/XSload/tomcat/conf/Catalina/localhost/manager.xml 35
/opt/XSload/tomcat/webapps/balancer/META-INF/context.xml 35
/opt/XSload/tomcat/conf/context.xml 90

Pretty clear, really - every 10 seconds tomcat goes round checking to see if you've gone and modified anything.

Now for the mysql burst. What are the pathnames here? There's quite a lot of output, so I've trimmed it a bit:

...
/usr/ccs/lib/libc.so.1 432
/usr/ccs/lib/libcrypt_i.so.1 432
/usr/ccs/lib/libcurses.so.1 432
/usr/ccs/lib/libgen.so.1 432
/usr/ccs/lib/libm.so.1 432
/usr/ccs/lib/libnsl.so.1 432
/usr/ccs/lib/librt.so.1 432
/usr/ccs/lib/libsocket.so.1 432
/usr/ccs/lib/libthread.so.1 432
/usr/ccs/lib/libw.so.1 432
/usr/ccs/lib/libz.so.1 432
/lib/ld.so.1 504
/opt/SUNWspro/lib/rw7/libCrun.so.1 540
/opt/SUNWspro/lib/rw7/libCstd.so.1 540
/opt/SUNWspro/lib/rw7/libc.so.1 540
/opt/SUNWspro/lib/rw7/libcrypt_i.so.1 540
/opt/SUNWspro/lib/rw7/libcurses.so.1 540
/opt/SUNWspro/lib/rw7/libgen.so.1 540
/opt/SUNWspro/lib/rw7/libm.so.1 540
/opt/SUNWspro/lib/rw7/libnsl.so.1 540
/opt/SUNWspro/lib/rw7/librt.so.1 540
/opt/SUNWspro/lib/rw7/libsocket.so.1 540
/opt/SUNWspro/lib/rw7/libthread.so.1 540
/opt/SUNWspro/lib/rw7/libw.so.1 540
/opt/SUNWspro/lib/rw7/libz.so.1 540
/opt/SUNWspro/lib/v8/libCrun.so.1 540
/opt/SUNWspro/lib/v8/libCstd.so.1 540
/usr/local/mysql/data/my.cnf 540
...
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus+vis2/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9+vis2/libCstd_isa.so.1 756
/opt/SUNWspro/prod/usr/lib/cpu/sparcv9/libCstd_isa.so.1 756
...
/opt/SUNWspro/lib/libthread.so.1 864
/opt/SUNWspro/lib/libw.so.1 864
/opt/SUNWspro/lib/libz.so.1 864
/lib/libm.so.2 888
/usr/lib/libc.so.1 972
/usr/lib/libcrypt_i.so.1 972
...
/opt/SUNWspro/lib/v8/libgen.so.1 1080
/opt/SUNWspro/lib/v8/libm.so.1 1080
/opt/SUNWspro/lib/v8/libnsl.so.1 1080
/opt/SUNWspro/lib/v8/librt.so.1 1080
/opt/SUNWspro/lib/v8/libsocket.so.1 1080
/opt/SUNWspro/lib/v8/libthread.so.1 1080
...
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/../../libCrun.so.1 2160
/opt/SUNWspro/prod/usr/lib/cpu/sparcv8plus/libCstd_isa.so.1 2592
/platform/SUNW,Sun-Blade-1000/lib/../../sun4u-us3/lib/libc_psr.so.1 3360
/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1 4480

What on earth?

So, most of these lookup operations are the mysql binary looking for shared libraries when it starts. And in some less than obvious places too! So why is this? I go and have a look at the binary:

% dump -Lv mysql
[INDEX] Tag Value
[1] NEEDED libcurses.so.1
[2] NEEDED libz.so.1
[3] NEEDED librt.so.1
[4] NEEDED libcrypt_i.so.1
[5] NEEDED libgen.so.1
[6] NEEDED libsocket.so.1
[7] NEEDED libnsl.so.1
[8] NEEDED libm.so.1
[9] NEEDED libCstd.so.1
[10] NEEDED libCrun.so.1
[11] NEEDED libw.so.1
[12] NEEDED libthread.so.1
[13] NEEDED libc.so.1
[16] RUNPATH /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib
[17] RPATH /opt/SUNWspro/lib/rw7:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/opt/SUNWspro/lib/v8:/opt/SUNWspro/lib:/usr/ccs/lib:/usr/lib

The list of libraries matches what we see being looked for, so that makes sense. The problem is that the compiled in library search path contains places that it shouldn't (and some repeated), so it needlessly searches those locations (and multiple times at that) when mysql starts up.

So, problem solved. And, as I said earlier, the idea is that you be able to see this anomalous activity in one of the jkstat tools and click on it to drill down into the system to see what's going on, so that all the dtrace is done automatically on the fly for you.

It won't be that easy, of course. I'm relying on the fbt provider, so that I need to pretty well write the dtrace script by hand for each function I wish to investigate. (There isn't even a consistent naming or calling scheme - you can't just replace ufs by procfs and expect it to work.) But fortunately we have the OpenSolaris source to look at to see what's actually going on underneath the covers.

Tuesday, March 21, 2006

jfsstat

I've just updated jkstat.

The notable new feature in this version is the jfsstat utility:



This uses the new kstats introduced in Nevada recently - the kstats used by fsstat. So you need to be running Nevada build 35 or newer to see these. (ON downloads - you want SXCR build 35 or later.)

The other change that's part of this is that I've started to use JTable to display things. I don't know why I've avoided this in the past - I guess that JTable seemed rather complex, but once I had got the hang of it it turns out to be very easy. I'm also using TableSorter from the Java Swing tutorial, which gives me the ability to sort the columns for free.

Enjoy!

Friday, March 17, 2006

Home Networked

I'm writing this blog entry using my W2100z workstation upstairs in the spare room.

We've spent the last couple of weekends decorating Amanda's room, and I took the opportunity to run some cat-5 cable in the upstairs rooms. The next step was to run cable down the stairs to connect my little ethernet switch to the broadband router, and enable dhcp. Hey presto! It all works.

The girls each have a SunBlade 150. Well, I've got them, and it means I don't have to worry about them getting virus infestations. So a surfing we can go! (And it means that the main computer is free instead of being taken up by someone trying to do their homework half the evening.)

I used an all-in-one kit I got from Maplins. 50m of cable, connectors, boxes, faceplates, tools, and 4 patch leads. A great buy and a real bargain.

Wednesday, March 08, 2006

T2000 problem

I just tried to buy a pair of Sun's T2000 machines, and then discovered (very late) that the system specification doesn't match my requirements.

It's not the system itself that's the problem, but the peripheral connectivity - or rather lack of such.

In particular, the plan was to hook up a couple of 3120 SCSI arrays. Not everything wants a fancy raid array, and FC arrays (and associated HBAs) are pretty pricy anyway. In this case, I'm looking at raw spindles for database access, but also if you look at ZFS it wants plain drives - raid hardware just gets in the way. So the plan was to have 2 SCSI channels and mirror them.

This won't work in a T2000. There's only 1 free PCI-X slot (the internal SAS controller takes the other one), and the only supported SCSI HBA is single channel anyway. So at the present time you simply can't have more than a single SCSI chain on a T2000.

To say that this is annoying is an understatement. It also limits the usefulness of the T2000 for several other projects I have in the pipeline.

Now, I could use FC storage, because the T2000 does have PCI-E slots, and there are PCI-E fibre HBAs, so it would work, but you're looking at a 50% or so increase in cost, which I regard as unacceptable. (The cost differential is particularly bad in small configurations - as you push up it becomes much less pronounced.)

Ho hum. Time to construct a plan B.

Friday, March 03, 2006

OpenSolaris Installation and Packaging

It's here! Dave Miner made the announcement that the Installation and Packaging community, and the SVR4 packaging project are now up and running.

This is important stuff. Being able to install and manage software is a critical component of an operating system, and is something I do a lot of.

And the process - particularly the performance (or lack thereof) of the tools - could stand some improvement. Yes, they work, and they're solid, but with a bit of extra work they could be made better.