Gcache and IST

The Gcache is a memory-based cache of recent Galera transactions that is local to each node in a cluster. If a node leaves and rejoins the cluster, it can use the gcache from another node that stayed in the cluster (i.e., its donor node) to fetch the transactions it missed (IST) as opposed to doing a full state snapshot transfer (SST). However, there are a few nuances that are not obvious to the beginner:

The Gcache is lost when a node restarts

The Gcache is fixed size and implemented as a LRU. Once it is full, older transactions roll off.

Donor selection is made irregardless of the gcache state

If the given donor for a restarting node doesn’t have all transactions needed, a full SST (read: full backup) is done instead

Until recent developments, there was no way to tell what, precisely, was in the Gcache.

So, with (somewhat) arbitrary donor selection, it was hard to be certain that a node restart would not trigger a SST. For example:

A node crashed over night or was otherwise down for some length of time. How do you know if the gcache on any node is big enough to contain all the transactions necessary for IST?

If you brought two nodes in your cluster simultaneously, the second one you restart might select the first one as its donor and be forced to SST.

Along comes PXC 5.6.15 RC1

Astute readers of the PXC 5.6.15 release notes will have noticed this little tidbit:

New wsrep_local_cached_downto status variable has been introduced. This variable shows the lowest sequence number in gcache . This information can be helpful with determining IST and/or SST.

Until this release there was no visibility into any node’s Gcache and what was likely to happen when restarting a node. You could make some assumptions, but now it its a bit easier to:

Tell if a given node would be a suitable donor And hence select a donor manually using wsrep_sst_donor instead of leaving it to chance.

What it looks like

Suppose I have a 3 node cluster where load is hitting node1. I execute the following in sequence:

Shut down node2 Shut down node3 Restart node2

At step 3, node1 is the only viable donor for node2. Because our restart was quick, we can have some reasonable assurance that node2 will IST correctly (and it does).

However, before we restart node3, let’s check the oldest transaction in the gcache on nodes 1 and 2:

[root@node1 ~]# mysql -e "show global status like 'wsrep_local_cached_downto';" +---------------------------+--------+ | Variable_name | Value | +---------------------------+--------+ | wsrep_local_cached_downto | 889703 | +---------------------------+--------+ [root@node2 mysql]# mysql -e "show global status like 'wsrep_local_cached_downto';" +---------------------------+---------+ | Variable_name | Value | +---------------------------+---------+ | wsrep_local_cached_downto | 1050151 | +---------------------------+---------+ 1 2 3 4 5 6 7 8 9 10 11 12 13 [ root @ node1 ~ ] # mysql -e "show global status like 'wsrep_local_cached_downto';" + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- + | Variable_name | Value | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- + | wsrep_local_cached_downto | 889703 | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- + [ root @ node2 mysql ] # mysql -e "show global status like 'wsrep_local_cached_downto';" + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- - + | Variable_name | Value | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- - + | wsrep_local_cached_downto | 1050151 | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- - +

So we can see that node1 has a much more “complete” gcache than node2 does (i.e., a much smaller seqno). Node2’s gcache was wiped when it restarted, so it only has transactions from after its restart.

To check node3’s GTID, we can either check the grastate.dat, or (if it has crashed and the grastate is zeroed) use –wsrep_recover:

[root@node3 ~]# cat /var/lib/mysql/grastate.dat # GALERA saved state version: 2.1 uuid: 7206c8e4-7705-11e3-b175-922feecc92a0 seqno: 1039191 cert_index: [root@node3 ~]# mysqld_safe --wsrep-recover 140107 16:18:37 mysqld_safe Logging to '/var/lib/mysql/error.log'. 140107 16:18:37 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 140107 16:18:37 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.pIVkT4' --pid-file='/var/lib/mysql/node3-recover.pid' 140107 16:18:39 mysqld_safe WSREP: Recovered position 7206c8e4-7705-11e3-b175-922feecc92a0:1039191 140107 16:18:41 mysqld_safe mysqld from pid file /var/lib/mysql/node3.pid ended 1 2 3 4 5 6 7 8 9 10 11 12 13 [ root @ node3 ~ ] # cat /var/lib/mysql/grastate.dat # GALERA saved state version : 2.1 uuid : 7206c8e4 - 7705 - 11e3 - b175 - 922feecc92a0 seqno : 1039191 cert_index : [ root @ node3 ~ ] # mysqld_safe --wsrep-recover 140107 16 : 18 : 37 mysqld_safe Logging to '/var/lib/mysql/error.log' . 140107 16 : 18 : 37 mysqld_safe Starting mysqld daemon with databases from / var / lib / mysql 140107 16 : 18 : 37 mysqld_safe WSREP : Running position recovery with -- log_error = '/var/lib/mysql/wsrep_recovery.pIVkT4' -- pid - file = '/var/lib/mysql/node3-recover.pid' 140107 16 : 18 : 39 mysqld_safe WSREP : Recovered position 7206c8e4 - 7705 - 11e3 - b175 - 922feecc92a0 : 1039191 140107 16 : 18 : 41 mysqld_safe mysqld from pid file / var / lib / mysql / node3 .pid ended

So, armed with this information, we can tell what would happen to node3, depending on which donor was selected:

Donor selected Donor’s gcache oldest seqno Node3’s seqno Result for node3 node2 1050151 1039191 SST node1 889703 1039191 IST

So, we can instruct node3 to use node1 as its donor on restart with wsrep_sst_donor:

[root@node3 ~]# service mysql start --wsrep_sst_donor=node1 1 [ root @ node3 ~ ] # service mysql start --wsrep_sst_donor=node1

Note that passing mysqld options on the command line is only supported in RPM packages, Debian requires you put that setting in your my.cnf. We can see from node3’s log that it does properly IST:

2014-01-07 16:23:26 19834 [Note] WSREP: Prepared IST receiver, listening at: tcp://192.168.70.4:4568 2014-01-07 16:23:26 19834 [Note] WSREP: Node 0.0 (node3) requested state transfer from 'node1'. Selected 2.0 (node1)(SYNCED) as donor. ... 2014-01-07 16:23:27 19834 [Note] WSREP: Receiving IST: 39359 writesets, seqnos 1039191-1078550 2014-01-07 16:23:27 19834 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.6.15-56' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release 25.2, Revision 645, wsrep_25.2.r4027 2014-01-07 16:23:41 19834 [Note] WSREP: IST received: 7206c8e4-7705-11e3-b175-922feecc92a0:1078550 1 2 3 4 5 6 7 2014 - 01 - 07 16 : 23 : 26 19834 [ Note ] WSREP : Prepared IST receiver , listening at : tcp : / / 192.168.70.4 : 4568 2014 - 01 - 07 16 : 23 : 26 19834 [ Note ] WSREP : Node 0.0 ( node3 ) requested state transfer from 'node1' . Selected 2.0 ( node1 ) ( SYNCED ) as donor . . . . 2014 - 01 - 07 16 : 23 : 27 19834 [ Note ] WSREP : Receiving IST : 39359 writesets , seqnos 1039191 - 1078550 2014 - 01 - 07 16 : 23 : 27 19834 [ Note ] / usr / sbin / mysqld : ready for connections . Version : '5.6.15-56' socket : '/var/lib/mysql/mysql.sock' port : 3306 Percona XtraDB Cluster ( GPL ) , Release 25.2 , Revision 645 , wsrep_25 . 2.r4027 2014 - 01 - 07 16 : 23 : 41 19834 [ Note ] WSREP : IST received : 7206c8e4 - 7705 - 11e3 - b175 - 922feecc92a0 : 1078550

Sometime in the future, this may be handled automatically on donor selection, but for now it is very useful that we can at least see the status of the gcache.