redis集群节点重启后恢复
服务器重启后,集群报错:
[root@SHH-HQ-NHS11S nhsuser]# redis-cli -c -h ip -p 7000
ip:7000> set cc dd
(error) CLUSTERDOWN Hash slot not served
检查集群节点情况
> cluster nodes
c0767666bee76e5e0dc67f24031a3e1b574235cc :7000@17000 myself,master - 0 0 0 connected
发现集群的节点只剩一个,其余节点都不见了。
将集群节点加入:
# redis-cli --cluster add-node ip:7005 ip:7000 >>> Adding node ip:7005 to cluster ip:7000 >>> Performing Cluster Check (using node ip:7000) M: c0767666bee76e5e0dc67f24031a3e1b574235cc ip:7000 slots: (0 slots) master [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [ERR] Not all 16384 slots are covered by nodes.
这个往往是由于主node移除了,但是并没有移除node上面的slot,从而导致了slot总数没有达到16384,其实也就是slots分布不正确。所以在删除节点的时候一定要注意删除的是否是Master主节点。现在这样情况,我们先修复集群:
# redis-cli --cluster fix IP:7000
>>> Covering slot 9675 with ip:7000
有点久,稍等下
然后检查修复情况:
# redis-cli --cluster check ip:7000 ip:7000 (c0767666...) -> 0 keys | 16384 slots | 0 slaves. [OK] 0 keys in 1 masters. 0.00 keys per slot on average. >>> Performing Cluster Check (using node ip:7000) M: c0767666bee76e5e0dc67f24031a3e1b574235cc IP:7000 slots:[0-16383] (16384 slots) master [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered.
将节点加入集群:
[root@SHH-HQ-NHS11S nhsuser]# redis-cli --cluster add-node ip:7001 ip:7000 >>> Adding node ip:7001 to cluster ip:7000 >>> Performing Cluster Check (using node ip:7000) M: c0767666bee76e5e0dc67f24031a3e1b574235cc ip:7000 slots:[0-16383] (16384 slots) master M: a98432e520af41117f5d8a81e2c0e2a430940c7e ip:7003 slots: (0 slots) master M: 2ff0c7ee051e26f484b6826d6aa8a5a3fbc93d17 ip:7002 slots: (0 slots) master M: 26bf8ce786d7c9d0bbb9d9d78978497f86fbbe0a ip:7005 slots: (0 slots) master M: 0a7718d2c53a16debdbfb391a7acd23a50a05faa ip:7004 slots: (0 slots) master [OK] All nodes agree about slots configuration. >>> Check for open slots... >>> Check slots coverage... [OK] All 16384 slots covered. >>> Send CLUSTER MEET to node ip:7001 to make it join the cluster. [OK] New node added correctly.
测试一把:
[root@SHH-HQ-NHS11S nhsuser]# redis-cli -c -h IP -p 7000 IP:7000> set aa dd OK IP:7000> get aa "dd" IP:7000>