ERR – Adamhuan's Data Center – 【逻辑驱动数据】

Cloudera Hive error：org.apache.thrift.TException: Error in calling method FetchResults

在Cloudera的Hive中操纵一个较大的源，希望返回一个数据集的时候，我遇到了如题所示的错误。

具体如下：

0: jdbc:hive2://cdh1:10000> select * from  import_data_20180929_700m order by time;
INFO  : Compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from  import_data_20180929_700m order by time
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 0.051 seconds
INFO  : Executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from  import_data_20180929_700m order by time
INFO  : Query ID = hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:3
INFO  : Submitting tokens for job: job_1538147215784_0011
INFO  : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0011/
INFO  : Starting Job = job_1538147215784_0011, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0011/
INFO  : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job  -kill job_1538147215784_0011
INFO  : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
INFO  : 2018-09-29 17:01:18,092 Stage-1 map = 0%,  reduce = 0%
INFO  : 2018-09-29 17:01:34,732 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 12.28 sec
INFO  : 2018-09-29 17:01:36,806 Stage-1 map = 44%,  reduce = 0%, Cumulative CPU 30.95 sec
INFO  : 2018-09-29 17:01:37,853 Stage-1 map = 56%,  reduce = 0%, Cumulative CPU 48.2 sec
INFO  : 2018-09-29 17:01:42,005 Stage-1 map = 78%,  reduce = 0%, Cumulative CPU 54.04 sec
INFO  : 2018-09-29 17:01:55,507 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 73.65 sec
INFO  : 2018-09-29 17:02:24,562 Stage-1 map = 100%,  reduce = 78%, Cumulative CPU 105.26 sec
INFO  : 2018-09-29 17:02:30,793 Stage-1 map = 100%,  reduce = 84%, Cumulative CPU 109.15 sec
INFO  : 2018-09-29 17:02:37,002 Stage-1 map = 100%,  reduce = 89%, Cumulative CPU 112.24 sec
INFO  : 2018-09-29 17:02:43,215 Stage-1 map = 100%,  reduce = 94%, Cumulative CPU 115.72 sec
INFO  : 2018-09-29 17:02:48,415 Stage-1 map = 100%,  reduce = 96%, Cumulative CPU 116.89 sec
INFO  : 2018-09-29 17:02:54,655 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 118.16 sec
INFO  : 2018-09-29 17:02:58,801 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 119.27 sec
INFO  : MapReduce Total cumulative CPU time: 1 minutes 59 seconds 270 msec
INFO  : Ended Job = job_1538147215784_0011
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-1: Map: 3  Reduce: 1   Cumulative CPU: 119.27 sec   HDFS Read: 667885712 HDFS Write: 656952922 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 1 minutes 59 seconds 270 msec
INFO  : Completed executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 123.111 seconds
INFO  : OK
org.apache.thrift.TException: Error in calling method FetchResults
        at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1307)
        at com.sun.proxy.$Proxy0.FetchResults(Unknown Source)
        at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:363)
        at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43)
        at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2054)
        at org.apache.hive.beeline.Commands.executeInternal(Commands.java:998)
        at org.apache.hive.beeline.Commands.execute(Commands.java:1158)
[hdfs@cdh1 ~]$

0: jdbc:hive2://cdh1:10000> select * from import_data_20180929_700m order by time;

INFO : Compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from import_data_20180929_700m order by time

INFO : Semantic Analysis Completed

INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null)

INFO : Completed compiling command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 0.051 seconds

INFO : Executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7): select * from import_data_20180929_700m order by time

INFO : Query ID = hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7

INFO : Total jobs = 1

INFO : Launching Job 1 out of 1

INFO : Starting task [Stage-1:MAPRED] in serial mode

INFO : Number of reduce tasks determined at compile time: 1

INFO : In order to change the average load for a reducer (in bytes):

INFO : set hive.exec.reducers.bytes.per.reducer=<number>

INFO : In order to limit the maximum number of reducers:

INFO : set hive.exec.reducers.max=<number>

INFO : In order to set a constant number of reducers:

INFO : set mapreduce.job.reduces=<number>

INFO : number of splits:3

INFO : Submitting tokens for job: job_1538147215784_0011

INFO : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0011/

INFO : Starting Job = job_1538147215784_0011, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0011/

INFO : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job -kill job_1538147215784_0011

INFO : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1

INFO : 2018-09-29 17:01:18,092 Stage-1 map = 0%, reduce = 0%

INFO : 2018-09-29 17:01:34,732 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 12.28 sec

INFO : 2018-09-29 17:01:36,806 Stage-1 map = 44%, reduce = 0%, Cumulative CPU 30.95 sec

INFO : 2018-09-29 17:01:37,853 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 48.2 sec

INFO : 2018-09-29 17:01:42,005 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 54.04 sec

INFO : 2018-09-29 17:01:55,507 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 73.65 sec

INFO : 2018-09-29 17:02:24,562 Stage-1 map = 100%, reduce = 78%, Cumulative CPU 105.26 sec

INFO : 2018-09-29 17:02:30,793 Stage-1 map = 100%, reduce = 84%, Cumulative CPU 109.15 sec

INFO : 2018-09-29 17:02:37,002 Stage-1 map = 100%, reduce = 89%, Cumulative CPU 112.24 sec

INFO : 2018-09-29 17:02:43,215 Stage-1 map = 100%, reduce = 94%, Cumulative CPU 115.72 sec

INFO : 2018-09-29 17:02:48,415 Stage-1 map = 100%, reduce = 96%, Cumulative CPU 116.89 sec

INFO : 2018-09-29 17:02:54,655 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 118.16 sec

INFO : 2018-09-29 17:02:58,801 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 119.27 sec

INFO : MapReduce Total cumulative CPU time: 1 minutes 59 seconds 270 msec

INFO : Ended Job = job_1538147215784_0011

INFO : MapReduce Jobs Launched:

INFO : Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 119.27 sec HDFS Read: 667885712 HDFS Write: 656952922 SUCCESS

INFO : Total MapReduce CPU Time Spent: 1 minutes 59 seconds 270 msec

INFO : Completed executing command(queryId=hive_20180929170000_c83501ac-0fa2-4ad1-a57f-6475438fc6e7); Time taken: 123.111 seconds

INFO : OK

org.apache.thrift.TException: Error in calling method FetchResults

at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1307)

at com.sun.proxy.$Proxy0.FetchResults(Unknown Source)

at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:363)

at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:43)

at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2054)

at org.apache.hive.beeline.Commands.executeInternal(Commands.java:998)

at org.apache.hive.beeline.Commands.execute(Commands.java:1158)

[hdfs@cdh1 ~]$

有时候，也可能是这样的报错：

[hdfs@cdh1 ~]$ beeline -u jdbc:hive2://cdh1:10000 -n hdfs
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 4ms
Connecting to jdbc:hive2://cdh1:10000
Connected to: Apache Hive (version 1.1.0-cdh5.10.0)
Driver: Hive JDBC (version 1.1.0-cdh5.10.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.1.0-cdh5.10.0 by Apache Hive
0: jdbc:hive2://cdh1:10000> 
0: jdbc:hive2://cdh1:10000> select * from  import_data_20180929_700m order by time;
INFO  : Compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from  import_data_20180929_700m order by time
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:import_data_20180929_700m.no, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.time, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.source, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.target, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.protocol, type:string, comment:null), FieldSchema(name:import_data_20180929_700m.info, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 0.235 seconds
INFO  : Executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from  import_data_20180929_700m order by time
INFO  : Query ID = hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Number of reduce tasks determined at compile time: 1
INFO  : In order to change the average load for a reducer (in bytes):
INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
INFO  : In order to limit the maximum number of reducers:
INFO  :   set hive.exec.reducers.max=<number>
INFO  : In order to set a constant number of reducers:
INFO  :   set mapreduce.job.reduces=<number>
INFO  : number of splits:3
INFO  : Submitting tokens for job: job_1538147215784_0012
INFO  : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0012/
INFO  : Starting Job = job_1538147215784_0012, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0012/
INFO  : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job  -kill job_1538147215784_0012
INFO  : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
INFO  : 2018-09-29 17:14:05,897 Stage-1 map = 0%,  reduce = 0%
INFO  : 2018-09-29 17:14:22,665 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 12.75 sec
INFO  : 2018-09-29 17:14:24,771 Stage-1 map = 56%,  reduce = 0%, Cumulative CPU 48.55 sec
INFO  : 2018-09-29 17:14:29,967 Stage-1 map = 78%,  reduce = 0%, Cumulative CPU 54.0 sec
INFO  : 2018-09-29 17:14:36,197 Stage-1 map = 89%,  reduce = 0%, Cumulative CPU 66.7 sec
INFO  : 2018-09-29 17:14:40,349 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 71.43 sec
INFO  : 2018-09-29 17:14:58,169 Stage-1 map = 100%,  reduce = 82%, Cumulative CPU 92.98 sec
INFO  : 2018-09-29 17:15:04,475 Stage-1 map = 100%,  reduce = 92%, Cumulative CPU 99.25 sec
INFO  : 2018-09-29 17:15:11,088 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 103.47 sec
INFO  : 2018-09-29 17:15:13,243 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 104.57 sec
INFO  : MapReduce Total cumulative CPU time: 1 minutes 44 seconds 570 msec
INFO  : Ended Job = job_1538147215784_0012
INFO  : MapReduce Jobs Launched: 
INFO  : Stage-Stage-1: Map: 3  Reduce: 1   Cumulative CPU: 104.57 sec   HDFS Read: 667885717 HDFS Write: 656952922 SUCCESS
INFO  : Total MapReduce CPU Time Spent: 1 minutes 44 seconds 570 msec
INFO  : Completed executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 76.224 seconds
INFO  : OK
GC overhead limit exceeded
Closing: 0: jdbc:hive2://cdh1:10000
[hdfs@cdh1 ~]$

[hdfs@cdh1 ~]$ beeline -u jdbc:hive2://cdh1:10000 -n hdfs

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0

Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0

scan complete in 4ms

Connecting to jdbc:hive2://cdh1:10000

Connected to: Apache Hive (version 1.1.0-cdh5.10.0)

Driver: Hive JDBC (version 1.1.0-cdh5.10.0)

Transaction isolation: TRANSACTION_REPEATABLE_READ

Beeline version 1.1.0-cdh5.10.0 by Apache Hive

0: jdbc:hive2://cdh1:10000>

0: jdbc:hive2://cdh1:10000> select * from import_data_20180929_700m order by time;

INFO : Compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from import_data_20180929_700m order by time

INFO : Semantic Analysis Completed

INFO : Completed compiling command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 0.235 seconds

INFO : Executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671): select * from import_data_20180929_700m order by time

INFO : Query ID = hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671

INFO : Total jobs = 1

INFO : Launching Job 1 out of 1

INFO : Starting task [Stage-1:MAPRED] in serial mode

INFO : Number of reduce tasks determined at compile time: 1

INFO : In order to change the average load for a reducer (in bytes):

INFO : set hive.exec.reducers.bytes.per.reducer=<number>

INFO : In order to limit the maximum number of reducers:

INFO : set hive.exec.reducers.max=<number>

INFO : In order to set a constant number of reducers:

INFO : set mapreduce.job.reduces=<number>

INFO : number of splits:3

INFO : Submitting tokens for job: job_1538147215784_0012

INFO : The url to track the job: http://cdh3:8088/proxy/application_1538147215784_0012/

INFO : Starting Job = job_1538147215784_0012, Tracking URL = http://cdh3:8088/proxy/application_1538147215784_0012/

INFO : Kill Command = /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop/bin/hadoop job -kill job_1538147215784_0012

INFO : Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1

INFO : 2018-09-29 17:14:05,897 Stage-1 map = 0%, reduce = 0%

INFO : 2018-09-29 17:14:22,665 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 12.75 sec

INFO : 2018-09-29 17:14:24,771 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 48.55 sec

INFO : 2018-09-29 17:14:29,967 Stage-1 map = 78%, reduce = 0%, Cumulative CPU 54.0 sec

INFO : 2018-09-29 17:14:36,197 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 66.7 sec

INFO : 2018-09-29 17:14:40,349 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 71.43 sec

INFO : 2018-09-29 17:14:58,169 Stage-1 map = 100%, reduce = 82%, Cumulative CPU 92.98 sec

INFO : 2018-09-29 17:15:04,475 Stage-1 map = 100%, reduce = 92%, Cumulative CPU 99.25 sec

INFO : 2018-09-29 17:15:11,088 Stage-1 map = 100%, reduce = 98%, Cumulative CPU 103.47 sec

INFO : 2018-09-29 17:15:13,243 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 104.57 sec

INFO : MapReduce Total cumulative CPU time: 1 minutes 44 seconds 570 msec

INFO : Ended Job = job_1538147215784_0012

INFO : MapReduce Jobs Launched:

INFO : Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 104.57 sec HDFS Read: 667885717 HDFS Write: 656952922 SUCCESS

INFO : Total MapReduce CPU Time Spent: 1 minutes 44 seconds 570 msec

INFO : Completed executing command(queryId=hive_20180929171313_6c04b786-1109-4000-ada0-3512e6f7b671); Time taken: 76.224 seconds

INFO : OK

GC overhead limit exceeded

Closing: 0: jdbc:hive2://cdh1:10000

[hdfs@cdh1 ~]$

[……]

发表于： 2018年9月29日 2018年9月29日
作者： Adamhuan
分类： Cloudera
标签： --incremental=true, BEELINE, bigdata, Buffer模式, Cloudera, ERR, error, GC overhead limit exceeded, HIVE, org.apache.thrift.TException: Error in calling method FetchResults, out-of-memory, 增量模式, 大数据, 结果集, 过大, 错误
发表评论: Cloudera Hive error：org.apache.thrift.TException: Error in calling method FetchResults

ERR标签存档订阅

Cloudera Hive error：org.apache.thrift.TException: Error in calling method FetchResults

阿里云 – DataV（快速入门）：空气质量的全国分布变化【not well】