Cloudera Hive error：org.apache.thrift.TException: Error in calling method FetchResults
在Cloudera的Hive中操纵一个较大的源，希望返回一个数据集的时候，我遇到了如题所示的错误。
具体如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
0: jdbc:hive2://cdh1:10000> select * from import_data_20180929_700m order by time; INFO : Compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from import_data_20180929_700m order by time INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 0.051 seconds INFO : Executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from import_data_20180929_700m order by time INFO : Query ID = hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:3 INFO : Submitting tokens for job: job_1538147215784_0011 INFO : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0011/ INFO : Starting Job = job_1538147215784_0011, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0011/ INFO : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job -kill job_1538147215784_0011 INFO : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1 INFO : 2018-09-29 17:01:18,092 Stage-1 map = 0%, reduce = 0% INFO : 2018-09-29 17:01:34,732 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 12.28 sec INFO : 2018-09-29 17:01:36,806 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 30.95 sec INFO : 2018-09-29 17:01:37,853 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 48.2 sec INFO : 2018-09-29 17:01:42,005 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 54.04 sec INFO : 2018-09-29 17:01:55,507 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 73.65 sec INFO : 2018-09-29 17:02:24,562 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 105.26 sec INFO : 2018-09-29 17:02:30,793 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 109.15 sec INFO : 2018-09-29 17:02:37,002 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 112.24 sec INFO : 2018-09-29 17:02:43,215 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 115.72 sec INFO : 2018-09-29 17:02:48,415 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 116.89 sec INFO : 2018-09-29 17:02:54,655 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 118.16 sec INFO : 2018-09-29 17:02:58,801 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 119.27 sec INFO : MapReduce Total cumulative CPU time: 1 minutes 59 seconds 270 msec INFO : Ended Job = job_1538147215784_0011 INFO : MapReduce Jobs Launched: INFO : Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 119.27 sec HDFS Read: 667885712 HDFS Write: 656952922 SUCCESS INFO : Total MapReduce CPU Time Spent: 1 minutes 59 seconds 270 msec INFO : Completed executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 123.111 seconds INFO : OK org.apache.thrift.TException: Error in calling method FetchResults at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1307) at com.sun.proxy.$Proxy0.FetchResults(Unknown Source) at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:363) at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43) at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2054) at org.apache.hive.beeline.Commands.executeInternal(Commands.java:998) at org.apache.hive.beeline.Commands.execute(Commands.java:1158) [hdfs@cdh1 ~]$ |
有时候，也可能是这样的报错：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
[hdfs@cdh1 ~]$ beeline -u jdbc:hive2://cdh1:10000 -n hdfs Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 scan complete in 4ms Connecting to jdbc:hive2://cdh1:10000 Connected to: Apache Hive (version 1.1.0-cdh5.10.0) Driver: Hive JDBC (version 1.1.0-cdh5.10.0) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.10.0 by Apache Hive 0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> select * from import_data_20180929_700m order by time; INFO : Compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from import_data_20180929_700m order by time INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 0.235 seconds INFO : Executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from import_data_20180929_700m order by time INFO : Query ID = hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:3 INFO : Submitting tokens for job: job_1538147215784_0012 INFO : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0012/ INFO : Starting Job = job_1538147215784_0012, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0012/ INFO : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job -kill job_1538147215784_0012 INFO : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1 INFO : 2018-09-29 17:14:05,897 Stage-1 map = 0%, reduce = 0% INFO : 2018-09-29 17:14:22,665 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 12.75 sec INFO : 2018-09-29 17:14:24,771 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 48.55 sec INFO : 2018-09-29 17:14:29,967 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 54.0 sec INFO : 2018-09-29 17:14:36,197 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 66.7 sec INFO : 2018-09-29 17:14:40,349 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 71.43 sec INFO : 2018-09-29 17:14:58,169 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 92.98 sec INFO : 2018-09-29 17:15:04,475 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 99.25 sec INFO : 2018-09-29 17:15:11,088 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 103.47 sec INFO : 2018-09-29 17:15:13,243 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 104.57 sec INFO : MapReduce Total cumulative CPU time: 1 minutes 44 seconds 570 msec INFO : Ended Job = job_1538147215784_0012 INFO : MapReduce Jobs Launched: INFO : Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 104.57 sec HDFS Read: 667885717 HDFS Write: 656952922 SUCCESS INFO : Total MapReduce CPU Time Spent: 1 minutes 44 seconds 570 msec INFO : Completed executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 76.224 seconds INFO : OK GC overhead limit exceeded Closing: 0: jdbc:hive2://cdh1:10000 [hdfs@cdh1 ~]$ |
也可能是这样的：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
0: jdbc:hive2://cdh1:10000> 0: jdbc:hive2://cdh1:10000> select * from import_data_20180929_700m order by time; INFO : Compiling command(queryId=hive_20180929173535_587c6297-8192-462d-9e6a-7b8166ec33b0): select * from import_data_20180929_700m order by time INFO : Semantic Analysis Completed INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20180929173535_587c6297-8192-462d-9e6a-7b8166ec33b0); Time taken: 0.954 seconds INFO : Executing command(queryId=hive_20180929173535_587c6297-8192-462d-9e6a-7b8166ec33b0): select * from import_data_20180929_700m order by time INFO : Query ID = hive_20180929173535_587c6297-8192-462d-9e6a-7b8166ec33b0 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Number of reduce tasks determined at compile time: 1 INFO : In order to change the average load for a reducer (in bytes): INFO : set hive.exec.reducers.bytes.per.reducer=<number> INFO : In order to limit the maximum number of reducers: INFO : set hive.exec.reducers.max=<number> INFO : In order to set a constant number of reducers: INFO : set mapreduce.job.reduces=<number> INFO : number of splits:3 INFO : Submitting tokens for job: job_1538213248666_0001 INFO : The url to track the job: http://cdh3:8088/proxy/application_1538213248666_0001/ INFO : Starting Job = job_1538213248666_0001, Tracking URL = http://cdh3:8088/proxy/application_1538213248666_0001/ INFO : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job -kill job_1538213248666_0001 INFO : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1 INFO : 2018-09-29 17:35:58,877 Stage-1 map = 0%, reduce = 0% INFO : 2018-09-29 17:36:13,649 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 10.34 sec INFO : 2018-09-29 17:36:15,740 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 29.0 sec INFO : 2018-09-29 17:36:17,857 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 47.65 sec INFO : 2018-09-29 17:36:21,003 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 52.31 sec INFO : 2018-09-29 17:36:33,716 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 71.34 sec INFO : 2018-09-29 17:36:52,196 Stage-1 map = 100%, reduce = 67%, Cumulative CPU 88.71 sec INFO : 2018-09-29 17:36:57,998 Stage-1 map = 100%, reduce = 73%, Cumulative CPU 96.63 sec INFO : 2018-09-29 17:37:03,515 Stage-1 map = 100%, reduce = 81%, Cumulative CPU 105.38 sec INFO : 2018-09-29 17:37:09,863 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 111.78 sec INFO : 2018-09-29 17:37:15,108 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 116.58 sec INFO : MapReduce Total cumulative CPU time: 1 minutes 56 seconds 580 msec INFO : Ended Job = job_1538213248666_0001 INFO : MapReduce Jobs Launched: INFO : Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 116.58 sec HDFS Read: 667885698 HDFS Write: 656952922 SUCCESS INFO : Total MapReduce CPU Time Spent: 1 minutes 56 seconds 580 msec INFO : Completed executing command(queryId=hive_20180929173535_587c6297-8192-462d-9e6a-7b8166ec33b0); Time taken: 96.882 seconds INFO : OK org.apache.thrift.TException: Error in calling method FetchResults at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1307) at com.sun.proxy.$Proxy0.FetchResults(Unknown Source) at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:363) at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43) at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2054) at org.apache.hive.beeline.Commands.executeInternal(Commands.java:998) at org.apache.hive.beeline.Commands.execute(Commands.java:1158) at org.apache.hive.beeline.Commands.sql(Commands.java:1072) at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1172) at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1003) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:915) （过多的输出。） |
出现以上问题的原因是：
Beeline的客户端在返回的结果集很大的时候，会出现【out-of-memory】
对该情况：
beeline获取结果集其实是有两种方式的：
1. Buffer
2. 增量模式
默认是【Buffer】模式。
所[……]