multimaster provides several functions to check the current cluster state.
To check node-specific information, use mtm.status():
SELECT * FROM mtm.status();
To get the list of all nodes in the cluster together with their status, use mtm.nodes():
SELECT * FROM mtm.nodes();
For details on all the returned information, see the section called "Functions".
If a cluster node is disabled, any attempt to read or write data on this node raises an error by default. If you need to access the data on a disabled node, you can override this behavior at connection time by setting the application_name parameter to mtm_admin. In this case, you can run read and write queries on this node without multimaster supervision.
With the multimaster extension, you can add or drop cluster nodes. Before adding node, stop the load and ensure (with mtm.status() that all nodes (except the ones to be dropped) are online. When adding a new node, you need to load all the data to this node using pg_basebackup from any cluster node, and then start this node.
Suppose we have a working cluster of three nodes, with node1, node2, and node3 host names. To add node4, follow these steps:
Figure out the required connection string to access the new node. For example, for the database mydb, user mtmuser, and the new node node4, the connection string can be "dbname=mydb user=mtmuser host=node4".
In psql connected to any alive node, run:
SELECT mtm.add_node('dbname=mydb user=mtmuser host=node4');
This command changes the cluster configuration on all nodes and creates replication slots for the new node. It also returns node_id of the new node, which will be required to complete the setup.
Go to the new node and clone all the data from one of the alive nodes to this node:
pg_basebackup -D datadir -h node1 -U mtmuser -c fast -v
pg_basebackup copies the entire data directory from node1, together with configuration settings, and prints the last LSN replayed from WAL, such as '0/12D357F0'. This value will be required to complete the setup.
Configure the new node to boot with recovery_target=immediate to prevent redo past the point where replication will begin. Add to postgresql.conf
restore_command = 'false'
recovery_target = 'immediate'
recovery_target_action = 'promote'
And create recovery.signal file in the data directory.
Start PgES on the new node.
In psql connected to the node used to take the base backup, run:
SELECT mtm.join_node(4, '0/12D357F0');
where 4 is the node_id returned by the mtm.add_node() function call and '0/12D357F0' is the LSN value returned by pg_basebackup.
Before removing node, stop the load and ensure (with mtm.status() that all nodes (except the ones to be dropped) are online. Shut down the nodes you are going to remove. To remove the node from the cluster:
Run the mtm.nodes() function to learn the ID of the node to be removed:
SELECT * FROM mtm.nodes();
Run the mtm.drop_node()> function with this node ID as a parameter:
SELECT mtm.drop_node(3);
This will delete replication slots for node 3 on all cluster nodes and stop replication to this node.
If you would like to return the node to the cluster later, you will have to add it as a new node, as explained in the section called Adding New Nodes to the Cluster.