用Kettle从hive到mysql导数据时(数据量有些大,大概6000万+),导到700多万时报错,利用kettle的错误信息如下:
2018/04/04 12:53:29 - 表输出.0 - linenr 7400000
2018/04/04 12:54:07 - 表输入.0 - linenr 7450000
2018/04/04 12:54:09 - 表输出.0 - linenr 7450000
org.Pentaho.di.core.exception.KettleDatabaseException:
Couldn't get row from result set
java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at org.pentaho.di.core.database.Database.getRow(Database.java:2374)
at org.pentaho.di.core.database.Database.getRow(Database.java:2344)
at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:145)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:256)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:242)
at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:365)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.hadoop.shim.common.DriverProxyInvocationChain$ResultSetInvocationHandler.invoke(DriverProxyInvocationChain.java:756)
at com.sun.proxy.$Proxy41.next(Unknown Source)
at org.pentaho.di.core.database.Database.getRow(Database.java:2362)
... 4 more
2018/04/04 12:54:32 - 表输入.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Unexpected error
2018/04/04 12:54:32 - 表输入.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : org.pentaho.di.core.exception.KettleDatabaseException:
2018/04/04 12:54:32 - 表输入.0 - Couldn't get row from result set
2018/04/04 12:54:32 - 表输入.0 - java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@41d71f56 rejected from java.util.concurrent.ThreadPoolExecutor@45ab8304[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
2018/04/04 12:54:32 - 表输入.0 -
2018/04/04 12:54:32 - 表输入.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2374)
2018/04/04 12:54:32 - 表输入.0 - at org.pentaho.di.core.database.Database.getRow(Database.java:2344)
2018/04/04 12:54:32 - 表输入.0 - at org.pentaho.di.trans.steps.tableinput.TableInput.processRow(TableInput.java:145)
2018/04/04 12:54:32 - 表输入.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/04/04 12:54:32 - 表输入.0 - at java.lang.Thread.run(Thread.java:748)