Codis Dashboard：[error]: dashboard already exists: {“addr”:
有时候，Codis的Dashboard可能起不来，如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@codis-server1 ~]# cd /data/go_me/src/github.com/CodisLabs/codis/ [root@codis-server1 codis]# bin/codis-config dashboard 2016/08/20 13:45:28 dashboard.go:160: [INFO] dashboard listening on addr: :18087 2016/08/20 13:45:30 dashboard.go:234: [PANIC] create zk node failed [error]: dashboard already exists: {"addr": "10.158.1.94:18087", "pid": 23586} [stack]: 3 /data/go_me/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:234 main.runDashboard 2 /data/go_me/src/github.com/CodisLabs/codis/cmd/cconfig/dashboard.go:54 main.cmdDashboard 1 /data/go_me/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:85 main.runCommand 0 /data/go_me/src/github.com/CodisLabs/codis/cmd/cconfig/main.go:152 main.main ... ... [root@codis-server1 codis]# |
解决方法：
在之前，Codis Dashboard启动成功的时候，是这样的：
|
1 2 3 4 5 6 7 |
[root@codis-server1 codis]# bin/codis-config dashboard 2016/08/19 14:00:22 dashboard.go:160: [INFO] dashboard listening on addr: :18087 2016/08/19 14:00:23 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "10.158.1.94:18087", "pid": 23586} 2016/08/19 14:00:23 dashboard.go:144: [WARN] ********** Attention ********** 2016/08/19 14:00:23 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me, 2016/08/19 14:00:23 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually 2016/08/19 14:00:23 dashboard.go:147: [WARN] ******************************* |
在zookeeper所在的服务器上做清理：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
[root@zookeeper1 zookeeper]# ps -ef | grep --color zookeeper root 321 32725 0 13:50 pts/0 00:00:00 grep --color zookeeper avahi 1387 1 0 Aug19 ? 00:00:01 avahi-daemon: running [zookeeper1.local] root 2680 1 0 Aug19 ? 00:03:27 java -Dzookeeper.log.dir=/opt/zookeeper/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-zookeeper1.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper/bin/../lib/servlet-api-2.5-20081211.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-6.1.26.jar:/opt/zookeeper/bin/../lib/jetty-6.1.26.jar:/opt/zookeeper/bin/../lib/javacc.jar:/opt/zookeeper/bin/../lib/jackson-mapper-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../zookeeper-3.5.2-alpha.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/zookeeper/bin/../conf/zoo.cfg [root@zookeeper1 zookeeper]# [root@zookeeper1 zookeeper]# netstat -tupln | grep java tcp 0 0 :::2181 :::* LISTEN 2680/java tcp 0 0 :::8080 :::* LISTEN 2680/java tcp 0 0 :::50714 :::* LISTEN 2680/java tcp 0 0 ::ffff:10.158.1.98:3387 :::* LISTEN 2680/java [root@zookeeper1 zookeeper]# [root@zookeeper1 zookeeper]# bin/zkCli.sh -server 127.0.0.1:2181 /usr/bin/java Connecting to 127.0.0.1:2181 2016-08-20 14:09:39,907 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.2-alpha-1750793, built on 06/30/2016 13:15 GMT 2016-08-20 14:09:39,911 [myid:] - INFO [main:Environment@109] - Client environment:host.name=zookeeper1 2016-08-20 14:09:39,911 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_101 2016-08-20 14:09:39,915 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation 2016-08-20 14:09:39,915 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre 2016-08-20 14:09:39,915 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper/bin/../lib/servlet-api-2.5-20081211.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-6.1.26.jar:/opt/zookeeper/bin/../lib/jetty-6.1.26.jar:/opt/zookeeper/bin/../lib/javacc.jar:/opt/zookeeper/bin/../lib/jackson-mapper-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../zookeeper-3.5.2-alpha.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA> 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:os.version=2.6.32-279.el6.x86_64 2016-08-20 14:09:39,916 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root 2016-08-20 14:09:39,917 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root 2016-08-20 14:09:39,917 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/opt/zookeeper 2016-08-20 14:09:39,917 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=236MB 2016-08-20 14:09:39,922 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=247MB 2016-08-20 14:09:39,923 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=241MB 2016-08-20 14:09:39,927 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@51521cc1 Welcome to ZooKeeper! 2016-08-20 14:09:39,983 [myid:127.0.0.1:2181] - INFO [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2016-08-20 14:09:40,192 [myid:127.0.0.1:2181] - INFO [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:36376, server: 127.0.0.1/127.0.0.1:2181 [zk: 127.0.0.1:2181(CONNECTING) 0] 2016-08-20 14:09:40,296 [myid:127.0.0.1:2181] - INFO [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x10007afa5ca0006, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: 127.0.0.1:2181(CONNECTED) 0] [zk: 127.0.0.1:2181(CONNECTED) 0] ls /zk [codis] [zk: 127.0.0.1:2181(CONNECTED) 1] ls /zk/codis [db_test] [zk: 127.0.0.1:2181(CONNECTED) 2] ls /zk/codis/db_test [ActionResponse, actions, dashboard, migrate_tasks, slots] [zk: 127.0.0.1:2181(CONNECTED) 3] ls /zk/codis/db_test/dashboard [] [zk: 127.0.0.1:2181(CONNECTED) 4] [zk: 127.0.0.1:2181(CONNECTED) 4] help ZooKeeper -server host:port cmd args addauth scheme auth close config [-c] [-w] [-s] connect host:port create [-s] [-e] [-c] path [data] [acl] delete [-v version] path deleteall path delquota [-n|-b] path get [-s] [-w] path getAcl [-s] path history listquota path ls [-s] [-w] path ls2 path [watch] printwatches on|off quit reconfig [-s] [-v version] [[-file path] | [-members serverID=host:port1:port2;port3[,...]*]] | [-add serverId=host:port1:port2;port3[,...]]* [-remove serverId[,...]*] redo cmdno removewatches path [-c|-d|-a] [-l] rmr path set [-s] [-v version] path data setAcl [-s] [-v version] path acl setquota -n|-b val path stat [-w] path sync path [zk: 127.0.0.1:2181(CONNECTED) 5] [zk: 127.0.0.1:2181(CONNECTED) 5] rmr /zk/codis/db_test/dashboard The command 'rmr' has been deprecated. Please use 'deleteall' instead. [zk: 127.0.0.1:2181(CONNECTED) 6] |
可以看到，推荐我们使用deleteall去删除路径。
由于前面codis的配置文件里，我们将两个zookeeper的路径都写了：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
[root@codis-server1 codis]# cat config.ini | grep -v '#' | strings coordinator=zookeeper zk=10.158.1.98:2181,10.158.1.99:2182 product=test dashboard_addr=10.158.1.94:18087 password=coids backend_ping_period=5 session_max_timeout=1800 session_max_bufsize=131072 session_max_pipeline=1024 zk_session_timeout=30000 proxy_id=proxy_1 [root@codis-server1 codis]# |
所以，从【zookeeper2】上操作也是一样的。
下面的操作从【zookeeper2】上操作，通过deleteall去删除路径：/zk/codis/db_test。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
[root@zookeeper2 zookeeper]# hostname zookeeper2 [root@zookeeper2 zookeeper]# [root@zookeeper2 zookeeper]# pwd /opt/zookeeper [root@zookeeper2 zookeeper]# [root@zookeeper2 zookeeper]# ps -ef | grep --color zookeeper avahi 1377 1 0 Aug19 ? 00:00:02 avahi-daemon: running [zookeeper2.local] root 2505 1 0 Aug19 ? 00:04:20 java -Dzookeeper.log.dir=/opt/zookeeper/bin/../logs -Dzookeeper.log.file=zookeeper-root-server-zookeeper2.log -Dzookeeper.root.logger=INFO,CONSOLE -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError=kill -9 %p -cp /opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper/bin/../lib/servlet-api-2.5-20081211.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-6.1.26.jar:/opt/zookeeper/bin/../lib/jetty-6.1.26.jar:/opt/zookeeper/bin/../lib/javacc.jar:/opt/zookeeper/bin/../lib/jackson-mapper-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../zookeeper-3.5.2-alpha.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: -Xmx1000m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/zookeeper/bin/../conf/zoo.cfg root 32673 32462 0 14:15 pts/0 00:00:00 grep --color zookeeper [root@zookeeper2 zookeeper]# [root@zookeeper2 zookeeper]# netstat -tupln | grep --color java tcp 0 0 :::2182 :::* LISTEN 2505/java tcp 0 0 :::33582 :::* LISTEN 2505/java tcp 0 0 ::ffff:10.158.1.99:2288 :::* LISTEN 2505/java tcp 0 0 :::8080 :::* LISTEN 2505/java tcp 0 0 ::ffff:10.158.1.99:3388 :::* LISTEN 2505/java [root@zookeeper2 zookeeper]# [root@zookeeper2 zookeeper]# bin/zkCli.sh -server 127.0.0.1:2182 /usr/bin/java Connecting to 127.0.0.1:2182 2016-08-20 14:15:20,069 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.2-alpha-1750793, built on 06/30/2016 13:15 GMT 2016-08-20 14:15:20,077 [myid:] - INFO [main:Environment@109] - Client environment:host.name=zookeeper2 2016-08-20 14:15:20,077 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_101 2016-08-20 14:15:20,081 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation 2016-08-20 14:15:20,081 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre 2016-08-20 14:15:20,081 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/bin/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper/bin/../lib/servlet-api-2.5-20081211.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper/bin/../lib/jline-2.11.jar:/opt/zookeeper/bin/../lib/jetty-util-6.1.26.jar:/opt/zookeeper/bin/../lib/jetty-6.1.26.jar:/opt/zookeeper/bin/../lib/javacc.jar:/opt/zookeeper/bin/../lib/jackson-mapper-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper/bin/../zookeeper-3.5.2-alpha.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf: 2016-08-20 14:15:20,081 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-08-20 14:15:20,081 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA> 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:os.version=2.6.32-279.el6.x86_64 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root 2016-08-20 14:15:20,082 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root 2016-08-20 14:15:20,083 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/opt/zookeeper 2016-08-20 14:15:20,083 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=236MB 2016-08-20 14:15:20,088 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=247MB 2016-08-20 14:15:20,089 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=241MB 2016-08-20 14:15:20,093 [myid:] - INFO [main:ZooKeeper@855] - Initiating client connection, connectString=127.0.0.1:2182 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@51521cc1 Welcome to ZooKeeper! 2016-08-20 14:15:20,150 [myid:127.0.0.1:2182] - INFO [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2016-08-20 14:15:20,360 [myid:127.0.0.1:2182] - INFO [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:41941, server: 127.0.0.1/127.0.0.1:2182 [zk: 127.0.0.1:2182(CONNECTING) 0] 2016-08-20 14:15:20,457 [myid:127.0.0.1:2182] - INFO [main-SendThread(127.0.0.1:2182):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x20007af9fa00004, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: 127.0.0.1:2182(CONNECTED) 0] [zk: 127.0.0.1:2182(CONNECTED) 0] ls /zk [codis] [zk: 127.0.0.1:2182(CONNECTED) 1] ls /zk/codis [db_test] [zk: 127.0.0.1:2182(CONNECTED) 2] ls /zk/codis/db_test [ActionResponse, actions, migrate_tasks, slots] [zk: 127.0.0.1:2182(CONNECTED) 3] |
可以看到，dashboard已经不在了。
执行删除，也是看不到东西的：
|
1 2 3 |
[zk: 127.0.0.1:2182(CONNECTED) 0] deleteall /zk/codis/db_test/dashboard Node does not exist: /zk/codis/db_test/dashboard [zk: 127.0.0.1:2182(CONNECTED) 1] |
codis，重新启动dashboard。
|
1 2 3 4 5 6 7 |
[root@codis-server1 codis]# bin/codis-config -c config.ini dashboard 2016/08/20 14:20:15 dashboard.go:160: [INFO] dashboard listening on addr: :18087 2016/08/20 14:20:16 dashboard.go:143: [INFO] dashboard node created: /zk/codis/db_test/dashboard, {"addr": "10.158.1.94:18087", "pid": 4480} 2016/08/20 14:20:16 dashboard.go:144: [WARN] ********** Attention ********** 2016/08/20 14:20:16 dashboard.go:145: [WARN] You should use `kill {pid}` rather than `kill -9 {pid}` to stop me, 2016/08/20 14:20:16 dashboard.go:146: [WARN] or the node resisted on zk will not be cleaned when I'm quiting and you must remove it manually 2016/08/20 14:20:16 dashboard.go:147: [WARN] ******************************* |
可以看到，启动成功。
————————————————————
Done。
