Monitoring

Kumofs bunldes some management tools. kumotop enables you to monitor status of kumo-servers like UNIX's 'top' command.

$ kumotop -m mgr

This screenshot shows that a server running on 192.168.10.255:19800 is processing 89,877 Get requests and 15,137 Set requests per second. It has processed 10,854,681 Get requests and 2,062,999 Set requests before. And it stores 522,748 items.

As I added them up, the 6-node cluster is processing 646,067 requests per second and stores 3,145,344 items.

Adding, removing and recovering servers

To add servers to the cluster, run new kumo-servers and run kumoctl mgr attach.

To recover crasehd servers, restart the server and run kumoctl mgr attach.

To remove crashed servers and put back number of replicated data, run kumoctl mgr detach.

kumo-managers does attach/detach automatically when --auto-replace is specified on the command line.

Note that attach/detach starts rebalancing. It will cause large network traffic depending on total amount of stored data and number of kumo-servers.

If one of two kumo-managers is down, just restart it on the same address and port, and then run kumoctl mgr replace.

Even if both of the kumo-managers are down, it is no problem. restart them on the same address and port, and run kumoctl mgr attach.