Tuesday, April 29, 2008

Controlling the Install Footprint

Solaris 10 introduced two technologies that, both individually and in concert, change the way we think about installing the OS.

Zones provide very lightweight virtualization. Especially sparse-root zones, which share most of the system files (binaries such as /usr) but have their own copy of configuration files and data (/etc and /var).

SMF, the Service Management Facility, replaces the old rc system and inetd, and allows you to configure exactly what services and daemons are running on a system.

The point about SMF is that historically, the only real way to guarantee that something was disabled was to not install it at all. With SMF, it is feasible to install something and rely on SMF to make sure it won't be active. So while minimization can be done the old way, you can use SMF to minimize what's running on a much larger install.

Then think about zones, and especially sparse-root zones. Because /usr is shared, you have to install everything that's going to be used throughout all the zones. So if you need a zone that runs tomcat, you have to install tomcat globally and it's there in all the zones. (Unless you install a private copy outside the OS in the zone that needs it.) And this is where SMF comes in - you can have all the applications everywhere, but only enable them in the zones that need them.

And it's not just the applications you have now that you need to worry about - to allow for future zones, you really need to make sure that you've installed everything that you might need in the future.

And if you wish to migrate zones between machines, then you have to install everything that might be needed by any zone on your network.

The old days of "install what you need, and no more" on each system don't apply. Consolidation using zones tends to drive you towards fat installs, using SMF to manage what's active in each zone.

That's fine - up to a point. And that point is where each zone gets overloaded with the services that it might run. One part of this is manifest import, which can be the dominant factor in sparse-root zone installation times. (Thankfully work to speed this up has recently been done!)

And as time goes on, the number of services is steadily proliferating. My desktop has over 200 lines of output from svcs. I know it doesn't do 200 different things.

So I have recently been looking again at my install profiles, to see what level of additional tweaking can be done. And the emphasis here is not just on minimizing what software is installed, but minimizing what services are installed, to reduce the cost of manifest import and reduce the available number of services that have to be managed.

On my (largely untweaked) desktop, there are currently 158 SMF manifests. Broken down into packages by number, that is:

1 SUNWaccr
1 SUNWadmr
1 SUNWapmsc
1 SUNWatfsr
1 SUNWbindr
1 SUNWbrgr
1 SUNWcfplr
1 SUNWcsd
1 SUNWdhcsr
1 SUNWfontconfig-root
1 SUNWftpr
1 SUNWgnome-display-mgr-root
1 SUNWgssc
1 SUNWinstall-patch-utils-root
1 SUNWipmir
1 SUNWipplr
1 SUNWiscsir
1 SUNWiscsitgtr
1 SUNWjwnsr
1 SUNWkdcr
1 SUNWmconr
1 SUNWntpr
1 SUNWocfr
1 SUNWos86r
1 SUNWpcr
1 SUNWpiclr
1 SUNWpmr
1 SUNWppror
1 SUNWrcapr
1 SUNWslpr
1 SUNWsndmr
1 SUNWsshdr
1 SUNWstosreg
1 SUNWstsfr
1 SUNWtnamr
1 SUNWtnetr
1 SUNWtsr
1 SUNWwbcor
1 SUNWxwssu
2 SUNWbsr
2 SUNWnfssr
2 SUNWpoolr
2 SUNWpsr
2 SUNWsacom
2 SUNWservicetagr
2 SUNWsmmgr
2 SUNWvbox
2 SUNWvolr
2 SUNWzoner
3 SUNWckr
3 SUNWkrbr
3 SUNWnisr
3 SUNWtsg
4 SUNWsmbar
4 SUNWxwplr
4 SUNWypr
5 SUNWcnsr
5 SUNWnfscr
6 SUNWmdr
10 SUNWrcmdr
49 SUNWcsr

Now I can't uninstall SUNWcsr, but there are a lot of packages there that my servers aren't going to need. So I end up removing all the WBEM, smc, service tags, telnetd, tnamd, samba, core network services, and the rcmds help a bit too.

Sunday, April 27, 2008

JKstat 0.21

It's been a while since I updated JKstat. In one sense, this slowing down is indicative that the project has reached a level of comparative maturity. Alternatively, I haven't found as much time to work on it as I would like. The reality is that both are true - the first phase of producing functional software is largely finished, and I haven't had time to address the more ambitious plans I have for the next phase.

Anyway, I'm putting out an updated version of JKstat now that has just a couple of fixes. One (an excellent suggestion from Tony Curtis) is that the network accessory shows the rates in humanized units (and I ought to implement this feature more widely). The second is a bunch of NullPointerExceptions raised by empty kstats. (Yes, I eventually stumbled across some kstats that had no data at all.)

Tuesday, April 22, 2008

NDIS Wrapper on Indiana

I've been setting up a laptop (a Dell D600, as it happens) so I thought I would try the Indiana preview on it.

Things have moved on a bit since I did this last - the onboard ethernet is recognised out of the box. But not the wireless. Off to try the NDIS Wrapper trick.

Now, I've got the Windows driver, which gives me the .inf and .sys files, and I downloaded the latest ndis package. Now to build it.

Unlike Solaris 10, where you type make and it works first time, Indiana has been cut down to fit onto the live CD and so you need some extra stuff. Several iterations of trial and error later, and the required steps are:

pfexec pkg install SUNWgcc
pfexec pkg install SUNWhea
pfexec pkg install SUNWflexlex
pfexec pkg install SUNWgm4

(Clearly, the dependency information for these packages is incomplete. Without gm4, flex gives an error message than couldn't ever be described as useful.)

With that, the "make ndiscvt" step works. Then you need to copy the .sys and .inf files into place and run

./ndiscvt -i ndis.inf -s ndis.sys -o ndis.h

if that fails with the error

ndiscvt: line 13: e: syntax error.

then the .inf file is in utf-16 format, so you need to

iconv -f utf-16 -t ascii path/to/bcmwl5.inf > ndis.inf

and run the ndiscvt step again.

Then it really is as simple as

make ndis
pfexec cp bcmndis /kernel/drv/bcmndis
make ndisapi
pfexec cp ndisapi /kernel/misc

Looking at "/usr/X11/bin/scanpci -v", we see

pci bus 0x0002 cardnum 0x03 function 0x00: vendor 0x14e4 device 0x4320
Broadcom Corporation BCM4306 802.11b/g Wireless LAN Controller
CardVendor 0x1028 card 0x0001 (Dell TrueMobile 1300 WLAN Mini-PCI Card)
STATUS 0x0010 COMMAND 0x0106
CLASS 0x02 0x80 0x00 REVISION 0x02
BIST 0x00 HEADER 0x00 LATENCY 0x20 CACHE 0x00
BASE0 0xfafee000 addr 0xfafee000 MEM
MAX_LAT 0x00 MIN_GNT 0x00 INT_PIN 0x01 INT_LINE 0x0b
BYTE_0 0x01 BYTE_1 0x00 BYTE_2 0xc2 BYTE_3 0xff

So we need to (see the numbers for vendor and device in the first line above)

add_drv -i '"pci14e4,4320"' bcmndis

and Yay!, it attaches.

Monday, April 21, 2008

Where's my Primary Administrator gone?

One neat aspect of Solaris is RBAC, which allows you to control which actions users can perform.

A particularly blunt instrument is the 'Primary Administrator' profile. If you're a Primary Administrator, then you are effectively root - in that you can use pfexec to assume the privileges of the root account (or role).

In Indiana, for example, root is (normally) a role and the account you create at install is set up as a Primary Administrator. It's very convenient.

So I decided to implement the same mechanism on my home machines (and use RBAC to let my children do extra things without having to pester dad).

Which failed, big time. It's really easy, you just use usermod to add a profile to an account:

usermod -P 'Primary Administrator' user_name

at which point Solaris thumbed its nose at me.

UX: usermod: ERROR: Primary Administrator is not a valid profile name. Choose another.

I decided to dig a little deeper, and then you discover that the way these profiles find their way onto the system is (ahem) strange.

The profiles are defined in some files that live in /etc/security - auth_attr, exec_attr, and prof_attr - and then /etc/user_attr controls what is assigned to users. So where do these files come from?

It turns out that different packages stick their own entries in. If you start looking around the Solaris media, then go into Solaris_XX/Product and look for */reloc/etc/security/exec_attr (and the same for prof_attr and auth_attr). These are the files that get merged into the master copy by some funky class action script. (There are things about IPS that I don't agree with, but its plan of getting rid of all these way out scripts is something that has to be good.)

OK, so looking in those files, and the Primary Administrator is delivered by the SUNWwbcor package. What's that? "Solaris WBEM Services (root)".

No wonder I hadn't got the profile. I never install WBEM or anything to do with it. Systems are much better off without it (and I couldn't ever see myself installing it on something like a home system, or indeed any system where Primary Administrator might be used). But, if you don't install that package then you're going to have to install the profile yourself. Something like the following in the Product directory on the media should do it:

cat SUNWwbcor/reloc/etc/security/auth_attr >> /etc/security/auth_attr
cat SUNWwbcor/reloc/etc/security/exec_attr >> /etc/security/exec_attr
cat SUNWwbcor/reloc/etc/security/prof_attr >> /etc/security/prof_attr

Friday, April 18, 2008

From scary to useful

Much as I was taken aback by the native cifs client that is now in OpenSolaris, there is a serious side to this.

Integration with the Microsoft world is a huge deal. You just can't ignore it. (OK, Sun did try that approach for a while.)

As the "backup guy" I was just given a simple task. Copy some old files off to tape, archiving them so they can be deleted. Simple. Only they're on a Windows server.

But now it's real easy. Just mount the share, and tar to tape.

Normally I install each SXCE build onto an old Ultra 60 that's dedicated to testing. But for this task I could justify using something just a little more modern (and something that has a faster network interface so I don't have to wait forever). OK, so a Sun Blade 1500 isn't going to set the world on fire, but it's more than capable of doing the job. And it really was just a case of mounting the share and issuing a tar command.

Job done.

Tuesday, April 15, 2008

X4140 - X4440

I haven't seen the announcement from Sun, but just noticed the X4140 and X4440 appear.

They look just like Opteron versions of the X4150 and X4450.

(And yes, I think it's a shame you don't have the 16-drive option for the X4440.)

The case of the unloved files

I've been investigating a strange problem on one of my servers. It started out as a simple case of NetBackup failing to back the filesystem up.

Now that's not entirely unusual - NetBackup often goes off and sulks in a corner. But this was rather different, as it didn't disappear as mysteriously as it came. Rather, it stayed put and the filesystem repeatedly refused to complete a backup. And the diagnostics are pretty much non-existent.

OK, so after a week or so of this I decide to try an alternative approach. To my surprise, I couldn't blame NetBackup.

First attempt was to try ufsdump. It started off in a promising manner, then froze completely on me.

OK, that's not good. So I try various attempts at tar. So various tar commands, writing to a remote tape or filesystem. That would work, right? Wrong!

Each attempt freezes completely on me. That's local tar piped to tar that writes over nfs; tar piped to an rsh; tar on an nfs client; tar using rmt to write to a tape.

That's odd. Now, at least I can look at the output and see how far it's got. I'm starting to make headway, as it looks like it gets to the same point each time.

OK, so I start to build a copy by tarring up various bits of the filesystem, avoiding the place where I know it breaks. Until I get into the suspect area. And yes, it still fails in the same place (but at least I've got a copy of most of the filesystem now, so can breathe easier).

The bad area is in an area that has various versions (views of the data) of the index files of a proprietary search engine. Now it looks like tar always traverses the hierarchy in the same order. OK, so if I manually list the subdirectories in an order that puts the failed one last, I can copy off the files I'm missing, right?

That was a fine theory until it froze on me again. And this is where it gets really strange. Each subdirectory has the same structure. So in each subdirectory there's a bunch of files with different suffixes. And it always fails on one particular suffix. Furthermore, it fails at about the same distance (about 38 of 40 megabytes). That's about as weird as it gets, in my experience. What on earth is there about these files that causes anything that tries to back these files up to lock up completely?

And it gets worse. I can cp this file locally. Try cp to any nfs-mounted location and the cp wedges. An rcp to any remote system wedges. And it's the same again - it wedges at the same distance into the file.

It must be something in the data stream that's contained in these files. At least I did find one way to copy them across - if I gzip them first then they go across fine.

But where is the bug that's being tickled? Is this something in the network stack?

Monday, April 14, 2008

In my next job...

Now, I'm not looking for a job, as I'm very happy where I am, but if and when I do move on there's one hard question I'm going to ask any potential employer. And it's this:

How reliable is your air conditioning?

Because my servers were all sweating away in a sauna this morning when the A/C failed. Again.

We sorted it out and nothing went down as a result. And nothing has failed, yet. (I'm expecting a rash of disk failures over the next couple of weeks.)

I don't know what it is but wherever I've worked has had air conditioning problems. This should be mature technology, but seems ever so hard to get right.

Friday, April 11, 2008


Enough to send a shiver down my spine.

So, you download a current OpenSolaris.

Then, as root:

svcadm enable smb/client

And you can - as yourself - mount a windows share, from a server called windows.server in domain DOMAIN accessed as user user_name

mkdir /tmp/G
mount -F smbfs '//DOMAIN;user_name@windows.server/G' /tmp/G

and it just works

df -k /tmp/G
Filesystem kbytes used avail capacity Mounted on
1563815260 421453348 1142361912 27% /tmp/G

If you just want to see what shares are available:

smbutil view '//DOMAIN;user_name@windows.server'

Thursday, April 10, 2008

T5240 - disks and more disks

One of the things I've complained about in the past is the lack of internal drives in Sun servers.

The X4150 was interesting, in that it had 8 drives in a 1U chassis.

So I'm impressed with the T5240. 16 drives in 2U. Wow! I like it.

Now why didn't the X4450 do that?