Sqoop從Oracle導出數據出錯：The Network

Sqoop執行命令：

    ./sqoop  import  --connect jdbc:oracle:thin:@10.112.101.251:1621:crmadbmr --username bass_etl --password 75!n!u6J --table DBPMSADM.PD_USERSVC_INFO_00 -m 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906

執行日志：

crmd3n:/d2_data0/user/ocdc/bin/sqoop-1.4.2-cdh5.2.1/bin> 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906 <

14/07/19 15:31:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 

14/07/19 15:31:30 INFO manager.SqlManager: Using default fetchSize of 1000

14/07/19 15:31:30 INFO tool.CodeGenTool: Beginning code generation

14/07/19 15:31:31 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DBPMSADM.PD_USERSVC_INFO_00 t WHERE 1=0

14/07/19 15:31:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /d2_data0/user/ocdc/bin/hadoop-2.0.0-mr1-cdh5.2.1

注意：/tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM_PD_USERSVC_INFO_00.java 使用或覆蓋了已過時的 API。

注意：要了解詳細信息，請使用 -Xlint:deprecation 重新編譯。

14/07/19 15:31:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM.PD_USERSVC_INFO_00.jar

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.

14/07/19 15:31:33 INFO mapreduce.ImportJobBase: Beginning import of DBPMSADM.PD_USERSVC_INFO_00

14/07/19 15:31:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

14/07/19 15:31:36 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(SVC_ID), MAX(SVC_ID) FROM DBPMSADM.PD_USERSVC_INFO_00

14/07/19 15:31:37 WARN db.TextSplitter: Generating splits for a textual index column.

14/07/19 15:31:37 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.

14/07/19 15:31:37 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.

14/07/19 15:31:38 INFO mapred.JobClient: Running job: job_201407151514_3489

14/07/19 15:31:39 INFO mapred.JobClient: map 0% reduce 0%

14/07/19 15:31:50 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:53 INFO mapred.JobClient: map 25% reduce 0% 

14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:59 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:05 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_2, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:06 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_2, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:08 INFO mapred.JobClient: map 50% reduce 0% 

14/07/19 15:32:17 INFO mapred.JobClient: Job complete: job_201407151514_3489

14/07/19 15:32:17 INFO mapred.JobClient: Counters: 24

14/07/19 15:32:17 INFO mapred.JobClient: File System Counters

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes read=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes written=369072

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of large read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of write operations=0

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes read=256

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes written=8

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of read operations=2

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of large read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of write operations=2

14/07/19 15:32:17 INFO mapred.JobClient: Job Counters

14/07/19 15:32:17 INFO mapred.JobClient: Failed map tasks=1

14/07/19 15:32:17 INFO mapred.JobClient: Launched map tasks=12

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=91182

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Map-Reduce Framework

14/07/19 15:32:17 INFO mapred.JobClient: Map input records=0

14/07/19 15:32:17 INFO mapred.JobClient: Map output records=0

14/07/19 15:32:17 INFO mapred.JobClient: Input split bytes=256

14/07/19 15:32:17 INFO mapred.JobClient: Spilled Records=0

14/07/19 15:32:17 INFO mapred.JobClient: CPU time spent (ms)=8910

14/07/19 15:32:17 INFO mapred.JobClient: Physical memory (bytes) snapshot=698327040

14/07/19 15:32:17 INFO mapred.JobClient: Virtual memory (bytes) snapshot=7732314112

14/07/19 15:32:17 INFO mapred.JobClient: Total committed heap usage (bytes)=2022703104

14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Transferred 8 bytes in 43.3403 seconds (0.1846 bytes/sec)

14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Retrieved 0 records.

14/07/19 15:32:17 ERROR tool.ImportTool: Error during import: Import job failed!

    昨天發了從DB2中導入數據的錯誤日志，后來定位為JDBC驅動包的問題，但是替換之后還是沒用，然后又定位為是由于JDBC線程未關閉，然后換成了從Oracle中導入數據，又來了另外一個問題：連接有時能通，有時不能通，對于-m 4是基本通不了，原因在于Mapreduce是自動分發匹配的，所以有些結點可能是不可見的，這樣對我們連接又會出現網絡不通的問題，初步解決方案：自行修改源代碼，找到任務分發代碼，直接指定一個專門用來做數據導入的結點。正在實踐ing。。。。