Tuesday, September 25, 2012

Mangling the contents file for fun and profit

I was recently using Live Upgrade to update a Solaris 10 system, when it went and chucked the following error at me and refused to go any further:

WARNING: Directory </usr/openv> zone <global> lies on a filesystem shared between BEs, remapping path to </usr/openv-S10U10>.
WARNING: Device <storage/backup> is shared between BEs, remapping to <storage/backup-S10U10>.
Mounting ABE <S10U10>.
ERROR: error retrieving mountpoint source for dataset <storage/backup-S10U10>
ERROR: failed to mount file system <storage/backup-S10U10> on </.alt.tmp.b-yPg.mnt/usr/openv-S10U10>

What I have is Netbackup installed in /usr/openv, and this is a separate filesystem (in fact, it's on a completely separate pool on completely separate drives).

The underlying problem here is that Solaris thinks there's part of the OS installed in /usr/openv, so it gets included in the scope of the upgrade. I've seen this in other cases where someone has stuck Netbackup off to the side, and it drags way too much stuff into scope.

On backup clients, the simplest thing to do is remove the client and reinstall it when you're done. You might get a free upgrade of the backup client along the way.

In this case I was working on the master backup server and didn't want to touch the installation at all.

One advantage of SVR4 packaging is that the package "database" is just a bunch of text files. If anything gets messed up you can open them up in your favourite editor and fix things up. (In one case, I remember simply copying /var/sadm off a similar system and nothing noticed the difference.)

The list of what files are installed is the contents file (found at /var/sadm/install/contents). The Live Upgrade process looks at this file to work out where software is installed. So all I had to do here was

grep /usr/openv contents > contents.openv

to save the entries for later, and

grep -v /usr/openv contents > contents.new
mv  contents.new contents

to create a contents file without any references to /usr/openv.

After this, Live Upgrade worked a treat and didn't try and stick its nose where it wasn't wanted.

Then, after the upgrade, I just had to merge the saved contents.openv I saved above into the new contents file. There's a bit of a trick here - the contents file is sorted on filename, so you can't just cat them together. I opened up the new contents file, went to the right place, inserted the saved entries, and was good to go.

Recursive zfs send and receive

I normally keep my ZFS filesystem hierarchy simple, so that the filesystem boundary is the boundary of administrative activity. So migrating a filesystem from one place to another is usually as simple as:

zfs snapshot tank/a@copy
zfs send tank/a@copy | zfs recv cistern/a

However, if you have child filesystems, and clones in particular, that you want to move, then it's slightly more involved. Suppose you have the following


where myfiles-clone is a clone of the @clone snapshot. I often create these temporarily if someone wants a copy of some data in a slightly different layout. In today's case, it had taken some time to shuffle the files around in the clone and I didn't want to have to do that all over again.

So, ZFS has recursive send and receive. The first thing I learnt is that myfiles-clone isn't really a descendant of myfiles - think of it as more of a sibling. So in this case you start from tank/a and send everything under that. First create a recursive snapshot:

zfs snapshot -r tank/a@copy

Then to send the whole lot

zfs send -R tank/a@copy

I was rather naive and thought that

zfs send -R tank/a@copy | zfs recv cistern/a

would do what I wanted - simply drop everything into cistern/a, and was somewhat surprised that this doesn't work. Particularly as ZFS almost always just works and does what you expect.

What went wrong? What I think happens here is that the recursive copy effectively does:

zfs send tank/a@copy | zfs recv cistern/a
zfs send tank/a/myfiles@copy | zfs recv cistern/a
zfs send tank/a/myfiles-clone@copy | zfs recv cistern/a

and attempts to put all the child filesystems in the same place, which fails rather badly. (You can see the hierarchy that would be created on the receiving side by using 'zfs recv -vn'.)

The way to solve this is to use the -e or -d options of zfs recv, like so:

zfs send -R tank/a@copy | zfs recv -d cistern/a


zfs send -R tank/a@copy | zfs recv -e cistern/a

In both cases it uses the name of the source dataset to construct the name at the destination, so it will lay it out properly.

The difference (read the man page) is that -e simply uses the last part of the source name at the destination. In this example, this was fine, but if you start off with a hierarchy it will get flattened (and you could potentially have naming collisions). With -d, it will just strip off the beginning (in this case, tank), so that the structure of the hierarchy is preserved, although you may end up with extra levels at the destination. If it's not quite right, though, zfs rename can sort it all out.

Monday, September 10, 2012


Just over a month or two ago, I wasn't an Apple customer. Sure, my daughter had an iPad (which I had used for a little development), but I didn't own or use any Apple devices myself.

Then my company mobile phone gave up the ghost. I had a Galaxy SII, and was assuming that I would get an SIII when the renewal came due. However, my old phone simply died a few days before the SIII became available in the UK, and I got an iPhone instead as it was actually available there and then.

The iPhone is OK, I guess. I'm not really a heavy smartphone user, it's handy to have some of the features but I wouldn't say that they're really crucial to me. My own personal phone is many years old now, and is one of those increasingly rare device that's actually useful for making phone calls. Generally, though, I wouldn't say that the iPhone is dramatically better than the Galaxy SII I had before; it's a bit more polished, but that's about all.

When it comes to spending my own cash, I then got myself a new iPad. The one with the retina display. I had been meaning to for a while, but was often too busy.

I had used several iPads before, and they've always just felt right. The touch, weight, balance, quality, all combine to generate an excellent experience. I wanted something around the home with reasonable battery life, instant on, and something that doesn't need a magnifying glass or too precise finger location. And the iPad delivers.

The main thing I use it for, a lot, is Sky Go. Generally in catchup mode, rather than live. (It's one of those odd things. Of all the programming that's available, a small fraction is what I want to watch. Invariably there's a multiway conflict, followed by hours or days of total wasteland.)

Generally, I find the Sky Go player to work extremely well. Better than iPlayer, anyway (whether that's the player or the delivery mechanism, though, I'm not quite sure). But then I notice that the latest release of the iPlayer app can download content to the iPad for viewing later, which may come in extremely useful.

I got a little iPod shuffle along with it. It's just great. I had an mp3 player from a brand that I had never heard of, and it wasn't reliable, nor did it have reasonable battery life. The whole thing put me off. But I wanted a little distraction at the gym, so the shuffle was perfect - you don't want to look at it or fiddle with the controls, ever, beyond on and off, and I wanted the smallest and lightest model available. I find myself using it so much now that I could actually do with a larger capacity model to get more tracks in the mix.

The latest toy is a new MacBook Pro. A new company laptop was due, I'm known to be a unix guy, so got offered a choice. My only constraint was that the resolution be adequate. (Seriously, 1366x768 is so 1990s.) So the retina display was called for again.

Frankly, I love it. So the keyboard is different, the trackpad is different, the user interface is different. But I quickly became accustomed to it, especially when things actually work. Like the iPad, though, the user experience is dramatically superior. And that old Windows thing from a mainstream supplier I had before is utter garbage in comparison.

Unfortunately I'm having a bit of trouble persuading the company to go for the dual thunderbolt 27-inch display setup.

So, largely by accident, and certainly without deliberate planning, most of my devices happen to have an Apple logo on.

Sunday, September 02, 2012

Cargo Cult IT

In a Cargo Cult, practitioners slavishly imitate the superficial behaviours of a more advanced culture in the hope that they will receive the benefits of that more advanced culture.

I'm seeing signs that Cargo Cult behaviour is becoming prevalent in IT. Some examples that come to mind are agile, cloud, and devops.

This isn't to say that these technologies are inherently flawed. Rather, just as in the true Cargo Cults, adherents completely miss the point and hope to reap the benefits of a technology by blindly applying its superficial manifestations in formulaic fashion.

Let's be absolutely clear - many advanced organizations are using agile, cloud, devops, and other technologies to great effect. The problem comes when more primitive societies merely emulate the formalism without any clear understanding of the reasons behind it - or even an acceptance that there are underlying reasons.

Slavishly imitating the behavioural patterns of a more successful organization is unlikely to lead to a successful outcome. Rather, understanding your own organization's problems and then understanding how other organizations have attacked theirs, and why they have adopted the solutions they have, will allow progress.

Of course, there are other cults that are simply false. I'm tempted to drop ITIL and ISO9000 straight into that bucket.