Setting up an etcd cluster on Tribblix

Using etcd to store configuration data is a common pattern, so how might you set up an etcd cluster on Tribblix?

Updated: With current etcd, you may need to add the --enable-v2=true flag, as shown below. For example, Patroni requires v2.

I'll start by creating 3 zones to create a 3-node cluster. For testing these could all be on the same physical system, for production you would obviously want them on separate machines.

As root:

zap refresh

zap create-zone -z node1 -t whole -o base -x 192.168.0.231

zap create-zone -z node2 -t whole -o base -x 192.168.0.232

zap create-zone -z node3 -t whole -o base -x 192.168.0.233

If you add the -U flag with your own username then you'll be able to use zlogin via pfexec from your own account, rather than always running it as root (in other words, subsequent invocations of zlogin could be pfexec zlogin.)

Then zlogin to node1 (and node2 and node3) to install etcd, and create
a user to run the service.

zlogin node1

zap install TRIBblix-etcd
useradd -u 11798 -g staff -s /bin/bash -d /export/home/etcd etcd
passwd -N etcd
mkdir -p /export/home/etcd
chown -hR etcd /export/home/etcd

I'm going to use static initialization to create the cluster. See the
clustering documentation.

You need to give each node a name (I'm going to use the zone name) and the cluster a name, here I'll use pg-cluster-1 as I'm going to use it for some PostgreSQL clustering tests. Then you need to specify the URLs that will be used by this node, and the list of URLs used by the cluster as a whole - which means all 3 machines. For this testing I'm going to use unencrypted connections between the nodes, in practice you would want to run everything over ssl.

zlogin -l etcd node1

/opt/tribblix/etcd/bin/etcd \
  --name node1 \
  --initial-advertise-peer-urls http://192.168.0.231:2380 \
  --listen-peer-urls http://192.168.0.231:2380 \
  --listen-client-urls http://192.168.0.231:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.0.231:2379 \
  --initial-cluster-token pg-cluster-1 \
  --initial-cluster node1=http://192.168.0.231:2380,node2=http://192.168.0.232:2380,node3=http://192.168.0.233:2380 \
  --initial-cluster-state new \
  --enable-v2=true

The same again for node2, with the same cluster list, but its own
URLs.

zlogin -l etcd node2

/opt/tribblix/etcd/bin/etcd \
  --name node2 \
  --initial-advertise-peer-urls http://192.168.0.232:2380 \
  --listen-peer-urls http://192.168.0.232:2380 \
  --listen-client-urls http://192.168.0.232:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.0.232:2379 \
  --initial-cluster-token pg-cluster-1 \
  --initial-cluster node1=http://192.168.0.231:2380,node2=http://192.168.0.232:2380,node3=http://192.168.0.233:2380 \
  --initial-cluster-state new \
  --enable-v2=true

And for node3:

zlogin -l etcd node3

/opt/tribblix/etcd/bin/etcd \
  --name node3 \
  --initial-advertise-peer-urls http://192.168.0.233:2380 \
  --listen-peer-urls http://192.168.0.233:2380 \
  --listen-client-urls http://192.168.0.233:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://192.168.0.233:2379 \
  --initial-cluster-token pg-cluster-1 \
  --initial-cluster node1=http://192.168.0.231:2380,node2=http://192.168.0.232:2380,node3=http://192.168.0.233:2380 \
  --initial-cluster-state new \
  --enable-v2=true

OK, that gives you a 3-node cluster. Initially you'll see complaints about being unable to connect to the other nodes, but it will settle down once they've all started.

And that's basically it. I think in an ideal world this would be an SMF service, with svccfg properties defining the cluster. Something I ought to implement for Tribblix at some point.

One useful tip, while discussing etcd. How do you see what's been stored in etcd? Obviously if you know what the keys in use are, you can just look them up, but if you just want to poke around you don't know what to look for. Also, etcdctl ls has been removed, which is how we used to do it. So to simply list all the keys:

etcdctl get "" --prefix --keys-only

There you have it.