怎么使用eclipse在遠程hadoop集群上在線運行和調試mapreduce程序

本篇內容主要講解“怎么使用eclipse在遠程hadoop集群上在線運行和調試mapreduce程序”，感興趣的朋友不妨來看看。本文介紹的方法操作簡單快捷，實用性強。下面就讓小編來帶大家學習“怎么使用eclipse在遠程hadoop集群上在線運行和調試mapreduce程序”吧!

前提：

1.我使用的hadoop是hadoop-2.3.0-cdh6.1.0.tar

2.以下是我的hadoop核心配置文件的配置：

core-site.xml

<configuration>

                <property>

                                <name>fs.defaultFS</name>

                                <value>hdfs://master:9000</value>

                </property>

       <property>

                                <name>io.file.buffer.size</name>

                                <value>131072</value>

       </property>

       <property>

                                <name>hadoop.tmp.dir</name>

                                <value>file:/home/yinkaipeng/tmp</value>

                                <description>Abase for other temporary directories.</description>

                </property>

        <property>

               <name>hadoop.proxyuser.hduser.hosts</name>

               <value>*</value>

       </property>

                 <property>

               <name>hadoop.proxyuser.hduser.groups</name>

               <value>*</value>

       </property>

</configuration>

hdfs-site.xml

<configuration>

       <property>

                <name>dfs.namenode.secondary.http-address</name>

               <value>master:9001</value>

        </property>

         <property>

                  <name>dfs.namenode.name.dir</name>

                 <value>file:/usr/local/data/dfs/name</value>

            </property>

           <property>

                    <name>dfs.datanode.data.dir</name>

                    <value>file:/usr/local/data/dfs/data</value>

            </property>

            <property>

                     <name>dfs.replication</name>

                     <value>3</value>

             </property>

             <property>

                     <name>dfs.webhdfs.enabled</name>

                     <value>true</value>

         </property>

</configuration>

mapred-site.xml

<configuration>

                <property>

                                <name>mapreduce.framework.name</name>

                                <value>yarn</value>

                </property>

                <property>

                                <name>mapreduce.jobhistory.address</name>

                                <value>master:10020</value>

                </property>

                <property>

               <name>mapreduce.jobhistory.webapp.address</name>

               <value>master:19888</value>

       </property>

</configuration>

yarn-site.xml

<configuration>

                 <property>

               <name>yarn.nodemanager.aux-services</name>

               <value>mapreduce_shuffle</value>

        </property>

                 <property>

               <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

               <value>org.apache.hadoop.mapred.ShuffleHandler</value>

        </property>

        <property>

               <name>yarn.resourcemanager.address</name>

               <value>master:8032</value>

       </property>

                <property>

               <name>yarn.resourcemanager.scheduler.address</name>

               <value>master:8030</value>

               </property>

               <property>

                       <name>yarn.resourcemanager.resource-tracker.address</name>

                        <value>master:8031</value>

               </property>

               <property>

                       <name>yarn.resourcemanager.admin.address</name>

                        <value>master:8033</value>

               </property>

                <property>

               <name>yarn.resourcemanager.webapp.address</name>

               <value>master:8088</value>

       </property>

</configuration>

在hadoop2.0 中如果不配置zookeeper，只需將你將你的datanode加入slaves文件中即可。

hadoop集群配置好后，然后就是使用eclipse進行連接啦！

好的，我使用的是：hadoop-eclipse-plugin-2.2.0，從網上下載的。

下面開始我們開始工作。

1. 啟動hadoop集群

2. 將hadoop-eclipse-plugin-2.2.0拷貝到eclipse的plugins 目錄下，啟動eclipse。

3. 

4. 

5. 

如現在就進行連接的話不會成功的。因為我們用的是Windows所以還需要進行以下步驟：

1. 將電腦當前用戶名改為hadoop的啟動用戶名

2. eclipse連接hadoop源碼目錄，將hadoop-common-2.2.0-bin-master的bin目錄考到eclipse的workspace

   
   

   
   

3. 

4. 

5. 

6. 

   注：上面hadoop目錄為我從Linux系統上下載的解壓后的hadoop。

   到這里操作hdfs就沒問題了，如果要運行mapreduce還會報錯，進行以下兩步：

   1.將hadoop源碼中的org.apache.hadoop.io.nativeio加到項目中，并進行以下修改：

   
   

   
   

2.在我們的mapreduce的main函數中注入hadoop本地目錄地址環境變量。

ok！這下就可以在Windows上進行hadoop集群的mapreduce在線調試了！

到此，相信大家對“怎么使用eclipse在遠程hadoop集群上在線運行和調試mapreduce程序”有了更深的了解，不妨來實際操作一番吧！這里是億速云網站，更多相關內容可以進入相關頻道進行查詢，關注我們，繼續學習！