ruby-2.4.6和redis-4.1.3.gem環境下的redis-cluster集群中的bug

bug場景說明：
ruby-2.4.6和redis-4.1.3.gem環境下的redis-cluster集群中，在移除redis-cluster集群中含有數據的節點時出現的bug。下面詳細說明下bug出現的場景。

redis-cluster移除節點：
和節點添加一樣，移除節點也有移除主節點，從節點。

 1、移除主節點
   移除節點使用redis-trib的del-node命令，

redis-trib del-node 127.0.0.1:7002 ${node-id}

127.0.0.1:7002是redis-cluster 集群中的任意節點，node-id為要刪除的主節點。 和添加節點不同，移除節點node-id是必需的，測試刪除7002主節點：
實例：刪除redis-cluster集群節點f7a95238e3de39b616b93949ec7c9f86d3867d63 對應的實例：192.168.1.39:1986
這個節點正好有數據

[root@mysql-redis39 log]# /data/soft/redis-4.0.12/src/redis-trib.rb del-node 192.168.1.39:1986 f7a95238e3de39b616b93949ec7c9f86d3867d63
>>> Removing node f7a95238e3de39b616b93949ec7c9f86d3867d63 from cluster 192.168.1.39:1986
[ERR] Node 192.168.1.39:1986 is not empty! Reshard data away and try again.
[root@mysql-redis39 log]# 

刪除節點失敗，提示節點中存在數據，不能從redis-cluster中刪除,需要將他的數據轉移出去，也就是和新增主節點一樣需重新分片.

需要重新分片:(移除192.168.1.39:1986的4096 個hash slots 到be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986 這master節點)

 [root@mysql-redis39 log]# /data/soft/redis-4.0.12/src/redis-trib.rb  reshard  192.168.1.39:1986
>>> Performing Cluster Check (using node 192.168.1.39:1986)
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   2 additional replica(s)
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: 0b3963a0be38ea500013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751

需要移動到全部主節點上還是單個主節點：

將4096個槽點移動到192.168.1.54:1986上，填寫192.168.1.39:1986的node id ：f7a95238e3de39b616b93949ec7c9f86d3867d63

How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:f7a95238e3de39b616b93949ec7c9f86d3867d63      ###192.168.1.39:1986的node id 
Source node #2:done
...................
...................
 Moving slot 10920 from f7a95238e3de39b616b93949ec7c9f86d3867d63
 Moving slot 10921 from f7a95238e3de39b616b93949ec7c9f86d3867d63
 Moving slot 10922 from f7a95238e3de39b616b93949ec7c9f86d3867d63
Do you want to proceed with the proposed reshard plan (yes/no)?yes

確認之后會一個一個將192.168.1.39:1986的卡槽移到到192.168.1.54:1986上

[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  reshard  192.168.1.182:1986
>>> Performing Cluster Check (using node 192.168.1.182:1986)
M: 0b3963a0be38e
0013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:8530-10922 (2393 slots) master
   1 additional replica(s)
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-8529,10923-12287 (5799 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.1.39:1986 has slots in migrating state (8530).
[WARNING] Node 192.168.1.54:1986 has slots in importing state (8530).
[WARNING] The following slots are open: 8530
>>> Check slots coverage...
[OK] All 16384 slots covered.

如果在遷移過程遇到下面這樣的錯誤：
[WARNING] Node 192.168.1.39:1986 has slots in migrating state (8530).
[WARNING] Node 192.168.1.54:1986 has slots in importing state (8530).
[WARNING] The following slots are open: 8530

[root@mysql-redis39 ~]# /usr/local/redis/bin/redis-cli -h 192.168.1.39 -p 1986
192.168.1.39:1986> cluster setslot 8530 stable
OK

[root@mysql-redis39 ~]# /usr/local/redis/bin/redis-cli -h 192.168.1.54 -p 1986
192.168.1.54:1986> cluster setslot 8530 stable
OK
可以考慮使用命令“redis-trib.rb fix 192.168.0.3:1986”嘗試修復。需要顯示有節點處于migrating或importing狀態，可以登錄到相應的節點，使用命令“cluster setslot (8530 stable”修改，參數8530為問題顯示的slot的ID。

再次執行下面的命令：

[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb reshard  192.168.1.39:1986
>>> Performing Cluster Check (using node 192.168.1.39:1986)
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:8530-10922 (2393 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-8529,10923-12287 (5799 slots) master
   2 additional replica(s)
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: 0b3963a0be38ea100013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea100013ddcaa5a5524801421dd5
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:f7a95238e3de39b616b93949ec7c9f86d3867d63
Source node #2:done

Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 8530 from 192.168.1.39:1986 to 192.168.1.54:1986: 
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

報錯：
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

參考網上資料：
https://blog.csdn.net/m0_37128231/article/details/80755478
說這個是redis-cluster集群管理命令：redis-trib.rb的一個bug
按照網上說的方法，但是結果還是在報錯，沒得到解決。

于是重新安裝ruby和redis-gem版本：

 [root@mysql-redis39 redis]# rvm install 2.6.3
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/7/x86_64/ruby-2.6.3.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.6.3, this may take a while depending on your cpu(s)...
ruby-2.6.3 - #downloading ruby-2.6.3, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.8M  100 13.8M    0     0  27236      0  0:08:52  0:08:52 --:--:-- 32881
ruby-2.6.3 - #extracting ruby-2.6.3 to /usr/local/rvm/src/ruby-2.6.3.....
ruby-2.6.3 - #configuring......................................................................
ruby-2.6.3 - #post-configuration..
ruby-2.6.3 - #compiling...............................................................................................
ruby-2.6.3 - #installing................................
ruby-2.6.3 - #making binaries executable..
ruby-2.6.3 - #downloading rubygems-3.0.6
ruby-2.6.3 - #extracting rubygems-3.0.6......
ruby-2.6.3 - #removing old rubygems........
ruby-2.6.3 - #installing rubygems-3.0.6...............................................
ruby-2.6.3 - #gemset created /usr/local/rvm/gems/ruby-2.6.3@global
ruby-2.6.3 - #importing gemset /usr/local/rvm/gemsets/global.gems................................................................
ruby-2.6.3 - #generating global wrappers.......
ruby-2.6.3 - #gemset created /usr/local/rvm/gems/ruby-2.6.3
ruby-2.6.3 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.6.3 - #generating default wrappers.......
ruby-2.6.3 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.6.3 - #complete 
Ruby was built without documentation, to build it run: rvm docs generate-ri
[root@mysql-redis39 redis]#  ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]

 [root@mysql-redis39 redis]# gem install redis --version 3.2.1 

Fetching redis-3.2.1.gem

Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
Done installing documentation for redis after 0 seconds
1 gem installed

然后再測試刪除節點：
刪除節點注意：

redis-trib.rb del-node host:port node_id
在刪除節點之前，其對應的槽必須為空，所以，在進行節點刪除動作之前，必須使用redis-trib.rb reshard將其遷移出去。
需要注意的是，如果某個節點的槽被完全遷移出去，其對應的slave也會隨著更新，指向遷移的目標節點。

[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  del-node 192.168.1.54:1986 a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b
>>> Removing node a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b from cluster 192.168.1.54:1986
/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning: constant ::Fixnum is deprecated
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  check 192.168.1.54:1986 
/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning: constant ::Fixnum is deprecated
[ERR] Sorry, can't connect to node 192.168.1.54:1986
[root@mysql-redis39 ~]#

到此處已經成功的刪除掉了先還有數據的redis-cluster中的節點實例。解決一開始出現的bug問題