Adamhuan's Data Center - 【逻辑驱动数据】
数据玩物、代码屋、1/0游戏：（零和博弈）/ 禅宗意志 / 规则战争 / 解放数据力量 / 技术的飞速发展并没有改变这个世界，因为，这个世界从没有变，它只是越来越趋近于它本来的模样。
参考文档：
安装前
https://www.cloudera.com/documentation/enterprise/6/6.1/topics/installation_reqts.html
安装时
https://www.cloudera.com/documentation/en[……]
Adamhuan
命令行如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
[root@cdh2 ~]# impala-shell Starting Impala Shell without Kerberos authentication Connected to cdh2:21000 Server version: impalad version 2.7.0-cdh5.10.0 RELEASE (build 785a073cd07e2540d521ecebb8b38161ccbd2aa2) *********************************************************************************** Welcome to the Impala shell. (Impala Shell v2.7.0-cdh5.10.0 (785a073) built on Fri Jan 20 12:03:56 PST 2017) Want to know what version of Impala you're connected to? Run the VERSION command to find out! *********************************************************************************** [cdh2:21000] > connect cdh2; Connected to cdh2:21000 Server version: impalad version 2.7.0-cdh5.10.0 RELEASE (build 785a073cd07e2540d521ecebb8b38161ccbd2aa2) [cdh2:21000] > [cdh2:21000] > invalidate metadata; Query: invalidate metadata Query submitted at: 2018-08-31 23:28:25 (Coordinator: http://cdh2:25000) Query progress can be monitored at: http://cdh2:25000/query_plan?query_id=724a13ec400f3864:ceb32a4400000000 Fetched 0 row(s) in 6.92s [cdh2:21000] > [cdh2:21000] > show tables; Query: show tables +-------------+ | name | +-------------+ | movie | | movierating | +-------------+ Fetched 2 row(s) in 0.54s [cdh2:21000] > [cdh2:21000] > select count(*) from movie; Query: select count(*) from movie Query submitted at: 2018-08-31 23:29:33 (Coordinator: http://cdh2:25000) Query progress can be monitored at: http://cdh2:25000/query_plan?query_id=824a3ba5793a8adf:4cd33f1700000000 +----------+ | count(*) | +----------+ | 3881 | +----------+ Fetched 1 row(s) in 18.26s [cdh2:21000] > [cdh2:21000] > select count(*) from movierating; Query: select count(*) from movierating Query submitted at: 2018-08-31 23:30:03 (Coordinator: http://cdh2:25000) Query progress can be monitored at: http://cdh2:25000/query_plan?query_id=304df122f17f2a8a:8592d4b00000000 +----------+ | count(*) | +----------+ | 1000205 | +----------+ Fetched 1 row(s) in 4.25s [cdh2:21000] > [cdh2:21000] > quit; Goodbye root [root@cdh2 ~]# |
CM的图形界面中，可以看到上述命令的历史：
——————————————————
Done。[……]
在使用HIVE的Beeline访问数据表的时候，有可能出现如题所示的NULL的问题。
具体如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
[hdfs@cdh1 ~]$ beeline -u jdbc:hive2://cdh1:10000 -n hdfs Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 scan complete in 3ms Connecting to jdbc:hive2://cdh1:10000 Connected to: Apache Hive (version 1.1.0-cdh5.10.0) Driver: Hive JDBC (version 1.1.0-cdh5.10.0) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.10.0 by Apache Hive 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> show tables; INFO : Compiling command(queryId=hive_20180831151616_72e0d1c3-14f0-4410-879c-c50adf08ea5e): show tables INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20180831151616_72e0d1c3-14f0-4410-879c-c50adf08ea5e); Time taken: 0.009 seconds INFO : Executing command(queryId=hive_20180831151616_72e0d1c3-14f0-4410-879c-c50adf08ea5e): show tables INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20180831151616_72e0d1c3-14f0-4410-879c-c50adf08ea5e); Time taken: 0.202 seconds INFO : OK +--------------+--+ | tab_name | +--------------+--+ | movie | | movierating | +--------------+--+ 2 rows selected (0.375 seconds) 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> select * from movie limit 5; INFO : Compiling command(queryId=hive_20180831151717_b0ee2167-b317-4812-9f34-0e868d45463c): select * from movie limit 5 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:movie.id, type:int, comment:null), FieldSchema(name:movie.name, type:string, comment:null), FieldSchema(name:movie.year, type:smallint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180831151717_b0ee2167-b317-4812-9f34-0e868d45463c); Time taken: 0.06 seconds INFO : Executing command(queryId=hive_20180831151717_b0ee2167-b317-4812-9f34-0e868d45463c): select * from movie limit 5 INFO : Completed executing command(queryId=hive_20180831151717_b0ee2167-b317-4812-9f34-0e868d45463c); Time taken: 0.017 seconds INFO : OK +-----------+-------------+-------------+--+ | movie.id | movie.name | movie.year | +-----------+-------------+-------------+--+ | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | +-----------+-------------+-------------+--+ 5 rows selected (0.167 seconds) 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> select * from movierating limit 5; INFO : Compiling command(queryId=hive_20180831151717_9c227dca-d789-4fa4-a247-b18daa66058d): select * from movierating limit 5 INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:movierating.userid, type:int, comment:null), FieldSchema(name:movierating.movieid, type:string, comment:null), FieldSchema(name:movierating.rating, type:tinyint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180831151717_9c227dca-d789-4fa4-a247-b18daa66058d); Time taken: 0.058 seconds INFO : Executing command(queryId=hive_20180831151717_9c227dca-d789-4fa4-a247-b18daa66058d): select * from movierating limit 5 INFO : Completed executing command(queryId=hive_20180831151717_9c227dca-d789-4fa4-a247-b18daa66058d); Time taken: 0.01 seconds INFO : OK +---------------------+----------------------+---------------------+--+ | movierating.userid | movierating.movieid | movierating.rating | +---------------------+----------------------+---------------------+--+ | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | | NULL | NULL | NULL | +---------------------+----------------------+---------------------+--+ 5 rows selected (0.17 seconds) 0: jdbc:hive2://cdh1:10000> |
其实，它们对应的HDFS上的文件是有数据的：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
[hdfs@cdh3 ~]$ hdfs dfs -cat mysql_data/movielens/movie/part-m-00000 | head -n 11 1,Toy Story,1995 2,Jumanji,1995 3,Grumpier Old Men,1995 4,Waiting to Exhale,1995 5,Father of the Bride Part II,1995 6,Heat,1995 7,Sabrina,1995 8,Tom and Huck,1995 9,Sudden Death,1995 10,GoldenEye,1995 11,American President, The,1995 [hdfs@cdh3 ~]$ [hdfs@cdh3 ~]$ hdfs dfs -cat mysql_data/movielens/movierating/part-m-00000 | head -n 11 1,1193,5 1,661,3 1,914,3 1,3408,4 1,2355,5 1,1197,3 1,1287,5 1,2804,5 1,594,4 1,919,4 1,595,5 cat: Unable to write to output stream. [hdfs@cdh3 ~]$ |
[……]
在使用HIVE的beeline的时候你可能遇到如题所示的问题。
该问题发生在你希望退出HIVE的Beeline命令行的时候，具体如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[hdfs@cdh1 ~]$ beeline -u jdbc:hive2://cdh1:10000 -n hdfs Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 scan complete in 3ms Connecting to jdbc:hive2://cdh1:10000 Connected to: Apache Hive (version 1.1.0-cdh5.10.0) Driver: Hive JDBC (version 1.1.0-cdh5.10.0) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.10.0 by Apache Hive 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> exit; Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'exit' '<EOF>' '<EOF>' (state=42000,code=40000) 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> quit; Error: Error while compiling statement: FAILED: ParseException line 1:0 cannot recognize input near 'quit' '<EOF>' '<EOF>' (state=42000,code=40000) 0: jdbc:hive2://cdh1:10000> |
其实，正确的退出方法应该是这样的：
1：!quit
[crayon-5cb9e24[……]
Adamhuan
在Cloudera中，默认情况下所有节点都会被分配到机架【/default】。
如果你修改了Cloudera集群中机架的分布，如下：
那么，在你执行过期重启之前，CDH的实际运行时的机架配置依旧会和以前的一样：
|
1 2 3 4 5 6 7 8 |
[cert@cdh1 ~]$ hdfs dfsadmin -printTopology Rack: /default 10.158.1.112:50010 (cdh2) 10.158.1.113:50010 (cdh3) 10.158.1.116:50010 (cdh4) 10.158.1.122:50010 (cdh5) [cert@cdh1 ~]$ |
[……]