Skip to Content
Skip Breadcrumb

Some Tips and Tricks for etcd

Posted on Modified on by Aleksandar Lazic

You will find here some all day work tips for etcd

cluster-health

This is a example from a OpenShift Cluster

etcd v2

/usr/bin/etcdctl --peers \
  $( awk -F= '/ETCD_LISTEN_CLIENT_URLS=/ {print $2}' < /etc/etcd/etcd.conf ) \
  --debug \
  --ca-file /etc/origin/master/master.etcd-ca.crt \
  --key-file /etc/origin/master/master.etcd-client.key \
  --cert-file /etc/origin/master/master.etcd-client.crt \
  cluster-health

cluster is healthy
member 4cc83942f30cdc13 is healthy
member 4d9d43ce9d81c2e1 is healthy
member 97f18a07f755fe77 is healthy

etcd v3

create endpoint line

export ENDPOINTS=$( ETCDCTL_API=3 etcdctl \
  --endpoints $( awk -F= '/ETCD_LISTEN_CLIENT_URLS=/ {print $2}' < /etc/etcd/etcd.conf ) \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  -w fields  member list |awk -F'"' '/ClientURL/ { printf "%s,",$4}'|sed -e 's/,$//'
  )

member list

ETCDCTL_API=3 etcdctl \
  --debug \
  --endpoints $( awk -F= '/ETCD_LISTEN_CLIENT_URLS=/ {print $2}' < /etc/etcd/etcd.conf ) \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  member list
  
INFO: 2018/08/09 10:25:24 ccBalancerWrapper: updating state and picker called by balancer: IDLE, 0xc42023c240
INFO: 2018/08/09 10:25:24 dialing to target with scheme: ""
INFO: 2018/08/09 10:25:24 could not get resolver for scheme: ""
INFO: 2018/08/09 10:25:24 balancerWrapper: is pickfirst: false
INFO: 2018/08/09 10:25:24 balancerWrapper: got update addr from Notify: [{<IP>:2379 <nil>}]
INFO: 2018/08/09 10:25:24 ccBalancerWrapper: new subconn: [{<IP>:2379 0  <nil>}]
INFO: 2018/08/09 10:25:24 balancerWrapper: handle subconn state change: 0xc420231240, CONNECTING
INFO: 2018/08/09 10:25:24 ccBalancerWrapper: updating state and picker called by balancer: CONNECTING, 0xc42023c240
INFO: 2018/08/09 10:25:24 balancerWrapper: handle subconn state change: 0xc420231240, READY
INFO: 2018/08/09 10:25:24 clientv3/balancer: pin "<IP>:2379"
INFO: 2018/08/09 10:25:24 ccBalancerWrapper: updating state and picker called by balancer: READY, 0xc42023c240
INFO: 2018/08/09 10:25:24 balancerWrapper: got update addr from Notify: [{<IP>:2379 <nil>}]

50e0ae8192bfb9a5, started, Master002.<DOMAIN>, https://<IP>:2380, https://<IP>:2379
9763834a055b8696, started, Master001.<DOMAIN>, https://<IP>:2380, https://<IP>:2379
c61053783c2db878, started, Master003.<DOMAIN>, https://<IP>:2380, https://<IP>:2379

endpoint health

ETCDCTL_API=3 etcdctl \
  --endpoints ${ENDPOINTS} \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  endpoint health
  
https://<IP>:2379 is healthy: successfully committed proposal: took = 2.568932ms
https://<IP>:2379 is healthy: successfully committed proposal: took = 1.632072ms
https://<IP>:2379 is healthy: successfully committed proposal: took = 2.56582ms

endpoint status

ETCDCTL_API=3 etcdctl \
  --endpoints ${ENDPOINTS} \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  -w table \
  endpoint status

Output

+--------------------------+------------------+---------+---------+-----------+-----------+------------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+
| https://<IP>:2379        | 50e0ae8192bfb9a5 |  3.2.11 |   88 MB |     false |      9210 |   42064684 |
| https://<IP>:2379        | 9763834a055b8696 |  3.2.11 |   88 MB |     false |      9210 |   42064684 |
| https://<IP>:2379        | c61053783c2db878 |  3.2.11 |   88 MB |      true |      9210 |   42064684 |
+--------------------------+------------------+---------+---------+-----------+-----------+------------+

list entries of a v2 cluster

You can get a inventory tree with the ls command.

/usr/bin/etcdctl --peers \
  $( awk -F= '/ETCD_LISTEN_CLIENT_URLS=/ {print $2}' < /etc/etcd/etcd.conf ) \
  --debug \
  --ca-file /etc/etcd/ca.crt \
  --key-file /etc/etcd/peer.key \
  --cert-file /etc/etcd/peer.crt \
  ls

list entries (keys) of a v3 cluster

ETCDCTL_API=3 etcdctl \
  --endpoints ${ENDPOINTS} \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  get / --prefix --keys-only

get info for a minion v2

You can get information from a key like this.

/usr/bin/etcdctl --peers \
  $( awk -F= '/ETCD_LISTEN_CLIENT_URLS=/ {print $2}' < /etc/etcd/etcd.conf ) \
  --debug \
  --ca-file /etc/etcd/ca.crt \
  --key-file /etc/etcd/peer.key \
  --cert-file /etc/etcd/peer.crt \
  get /kubernetes.io/minions/<THE_HOSTNAME> | python -m json.tool

get info for a minion v3

You can get information from a key like this.

ETCDCTL_API=3 etcdctl \
  --endpoints ${ENDPOINTS} \
  --cacert /etc/origin/master/master.etcd-ca.crt \
  --key /etc/origin/master/master.etcd-client.key \
  --cert /etc/origin/master/master.etcd-client.crt \
  -w json \
  get /kubernetes.io/minions/<THE_HOSTNAME> | jq -r '.kvs[].value'|base64 -d|less

You can contact me for any further questions and orders