Prevent watchdog split-brain scenario in some corner cases.

Watchdog uses two methods for communication: beacon message and
heartbeat.  If the heartbeat is not working yet, it is possible that
split-brain occurs if beacon messages are not received by standby
nodes more than 30 seconds (see watchdog_state_machine_standby). In
this case other watchdog node becomes new leader node. Problem is, the
former leader node remains leader too, thus there are two leader
nodes: split-brain.

Suppose we have 3 watchdog nodes (node 0, 1, 2) and node 2 was not
started due a maintenace work. In this case life check is not started.

$ pcp_watchdog_info -p 50005
3 3 NO localhost:50000 Linux tishii-CFSV9-2 localhost

localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 7 STANDBY 0 MEMBER
localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER
Not_Set localhost 50008 50010 0 DEAD 0 MEMBER

Then to stop the beacon signal, send STOP signal to node0 watchdg
process. Node 1 is promoted and now we have two leader nodes 0 and 1.

$ pcp_watchdog_info -p 50005
3 3 NO localhost:50004 Linux tishii-CFSV9-2 localhost

localhost:50004 Linux tishii-CFSV9-2 localhost 50004 50006 4 LEADER 0 MEMBER
localhost:50000 Linux tishii-CFSV9-2 localhost 50000 50002 4 LEADER 0 MEMBER
Not_Set localhost 50008 50010 0 DEAD 0 MEMBER

This commit tries to prevent having two leaders by revoking the former
leader. If a node detects beacon message from leader node being lost
more than 2 times (3 * BEACON_MESSAGE_INTERVAL_SECONDS), leader node
is set to LOST state and new leader node election process is triggered
by calling set_state(WD_JOINING).

Backpatch-through: v4.2

diff --git a/src/watchdog/watchdog.c b/src/watchdog/watchdog.c
index 6f6d2d2cd56875fd9691fe0b87f06373d3e15b5f..c37984331541d61aeb051ab81341bb06e32ce772 100644 (file)
--- a/src/watchdog/watchdog.c
+++ b/src/watchdog/watchdog.c
@@ -7305,6 +7305,35 @@ watchdog_state_machine_standby(WD_EVENTS event, WatchdogNode *wdNode, WDPacketDa
                                        (errmsg("we have not received a beacon message from leader node \"%s\" and it has not replied to our info request",
                                                        WD_LEADER_NODE->nodeName),
                                         errdetail("re-initializing the cluster")));
+                       /*
+                        * Starting to set LOST status to the leader node.  This needs to
+                        * be done because there could be two leader nodes.
+                        */
+                       WD_LEADER_NODE->state = WD_LOST;
+                       ereport(LOG,
+                                       (errmsg("remote node \"%s\" is lost due to missed beacons", WD_LEADER_NODE->nodeName)));
+
+                       /* Inform the node, that it is lost for us */
+                       send_cluster_service_message(WD_LEADER_NODE, pkt, CLUSTER_NODE_APPEARING_LOST);
+                       ereport(LOG,
+                                       (errmsg("watchdog cluster has lost the coordinator node")));
+
+                       /* close all socket connections to the node */
+                       close_socket_connection(&WD_LEADER_NODE->client_socket);
+                       close_socket_connection(&WD_LEADER_NODE->server_socket);
+
+                       /* clear the wait timer on the node */
+                       WD_LEADER_NODE->last_sent_time.tv_sec = 0;
+                       WD_LEADER_NODE->last_sent_time.tv_usec = 0;
+                       WD_LEADER_NODE->sending_failures_count = 0;
+
+                       node_lost_while_ipc_command(WD_LEADER_NODE);
+                       set_cluster_leader_node(NULL);
+                       /*
+                        * Finish setting LOST status to the leader node.
+                        */
+
+                       /* re-initialize the cluster */
                        set_state(WD_JOINING);
                }
                else if (last_rcv_sec >= (2 * BEACON_MESSAGE_INTERVAL_SECONDS))