设为首页收藏本站 开启辅助访问
 找回密码
 立即注册

QQ登录

只需一步,快速开始

用新浪微博登录

只需一步,快速搞定

查看: 390|回复: 2

[未解决] kettle 从hive导数据到mysql报错,请帮忙看一下是什么原因

[复制链接]

3

主题

4

帖子

28

积分

新手上路

Rank: 1

积分
28
发表于 2018-4-4 22:58:55 | 显示全部楼层 |阅读模式
Kettle从hive到mysql导数据时(数据量有些大,大概6000万+),导到700多万时报错,利用kettle的错误信息如下:
2018/04/04 12:53:29 - 表输出.0 - linenr 7400000
2018/04/04 12:54:07 - 表输入.0 - linenr 7450000
2018/04/04 12:54:09 - 表输出.0 - linenr 7450000
org.Pentaho.di.core.exception.KettleDatabaseException:
Couldn't get row from result set
java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
        at org.pentaho.di.core.database.Database.getRow(Database.java:2374)
        at org.pentaho.di.core.database.Database.getRow(Database.java:2344)
        at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:145)
        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
        at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
        at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.pentaho.hadoop.shim.common.DriverProxyInvocationChain$ResultSetInvocationHandler.invoke(DriverProxyInvocationChain.java:756)
        at com.sun.proxy.$Proxy41.next(Unknown Source)
        at org.pentaho.di.core.database.Database.getRow(Database.java:2362)
        ... 4 more

2018/04/04 12:54:32 - 表输入.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Unexpected error
2018/04/04 12:54:32 - 表输入.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2018/04/04 12:54:32 - 表输入.0 - Couldn't get row from result set
2018/04/04 12:54:32 - 表输入.0 - java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
2018/04/04 12:54:32 - 表输入.0 -
2018/04/04 12:54:32 - 表输入.0 -         at org.pentaho.di.core.database.Database.getRow(Database.java:2374)
2018/04/04 12:54:32 - 表输入.0 -         at org.pentaho.di.core.database.Database.getRow(Database.java:2344)
2018/04/04 12:54:32 - 表输入.0 -         at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:145)
2018/04/04 12:54:32 - 表输入.0 -         at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/04/04 12:54:32 - 表输入.0 -         at java.lang.Thread.run(Thread.java:748)

点评

海!外直播 t.cn/RxlBL8s 禁闻视频 t.cn/RJvO789 禁止言论自由只有三个解释:1.它过去做了坏事,怕人们提起.2.它正在干坏事,怕人们批评.3.它准备继续干坏事,怕人们揭露.总之,禁止言论自由一定与坏事相关,绝对不是好   发表于 2018-4-7 11:19
楼主热帖

0

主题

13

帖子

33

积分

新手上路

Rank: 1

积分
33
发表于 2018-5-6 19:41:50 | 显示全部楼层
你的数据量太大了 hive jdbc接口不适合
您hive底层怎么存储的?text csv parquet orc?
在pentaho企业版中支持直接读写hadoop文件直接输出到mysql是不是会更快?

www.hitachivantara.com
help.penthao.com
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

手机版|小黑屋| μKettle ( 沪ICP备14044064号 )     

GMT+8, 2018-9-23 05:18 , Processed in 0.151838 second(s), 19 queries , File On.

Powered by Discuz!  uKettle

Copyright © 2014 Comsenz Inc.

快速回复 返回顶部 返回列表