An obvious thought is: there are many time-series databases being used for monitoring now, with a variety of user-configurable dashboards that can be used to query and display the data. Why not use one of those?
First the database. For this test I'm using InfluxDB, which is written, like many applications are these days, in Go. Fortunately, Go works fine on illumos and I package it for Tribblix, so it's fairly easy to follow the instructions, so make a working directory, cd there, and:
export GOPATH=`pwd` go get github.com/influxdb/influxdb
cd $GOPATH/src/github.com/influxdb/influxdb go get -u -f -t ./...
go clean ./... go install ./...
(Note that it's influxdb/influxdb, not influxdata/influxdb. The name was changed, but the source and the build still use the old name.)
That should just work, leaving you with binaries in $GOPATH/bin.
So then you'll want a visualization front end. Now, there is Chronograf. Unfortunately it's closed source (that's fine, companies can make whatever choices they like) which means I can't build it for Tribblix. The other obvious path is Grafana.
Building Grafana requires Go, which we've already got, and Node.js. Again, Tribblix has Node.js, so we're (almost) good to go.
Again, it's mostly a case of following the build instructions. For Grafana, this comes in 2 parts. The back-end is Go, so make a working directory, cd there, and:
export GOPATH=`pwd` go get github.com/grafana/grafana
cd $GOPATH/src/github.com/grafana/grafana go run build.go setup $GOPATH/bin/godep restore go run build.go build
You'll find the Grafana server in $GOPATH/src/github.com/grafana/grafana/bin/grafana-server
The front-end involves a little variation to get it to work properly. The problem here is that a basic 'npm install' will install both production and development dependencies. We don't actually want to do development of Grafana, which ultimately requires webkit and won't work anyway. So we really just want the production pieces, and we don't want to install anything globally. But we still need to run 'npm install' to start with, as otherwise the dependencies get messed up. Just ignore the errors and warnings around PhantomJS.
npm install --production npm install grunt-cli ./node_modules/.bin/grunt --force
With that, you can fire up influxd and grafana-server, and get them to talk to each other.
For the general aspects of getting Grafana and Influxdb to talk to each other, here's a tutorial I found useful.
Now, with all this in place, I can go back to playing with kstats.