Hadoop偽模式配置

練習環境：

操作系統：Ubuntu 16.04 LTS

Hadoop版本：Hadoop 2.7.1

1.配置core-site.xml

hadoop@dblab:/usr/local/hadoop/etc/hadoop$ vim core-site.xml

<configuration>

??????? <property>

??????????????? <name>hadoop.tmp.dir</name>

??????????????? <value>file:/usr/local/hadoop/tmp</value>

??????? </property>

??????? <property>

??????????????? <name>fs.defaultFS</name>

??????????????? <value>hdfs://localhost:9000</value>

??????? </property>

</configuration>

2.配置hdfs-site.xml

?hadoop@dblab:/usr/local/hadoop/etc/hadoop$ vim hdfs-site.xml

<configuration>

??????? <property>

??????????????? <name>dfs.replication</name>

??????????????? <value>1</value>

??????? </property>

??????? <property>

??????????????? <name>dfs.namenode.name.dir</name>

??????????????? <value>file:/usr/local/hadoop/tmp/dfs/name</value>

??????? </property>

??????? <property>

??????????????? <name>dfs.datanode.data.dir</name>

??????????????? <value>file:/usr/local/hadoop/tmp/dfs/data</value>

??????? </property>

</configuration>

3.執行名稱節點格式化

hadoop@dblab:/usr/local/hadoop$

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs namenode –format

?Re-format filesystem in Storage Directory /usr/local/hadoop/tmp/dfs/name ? (Y or N)

?19/05/16 14:23:44 INFO namenode.FSImage: Allocated new BlockPoolId: BP-748770776-127.0.0.1-1557987824492

19/05/16 14:23:44 INFO common.Storage: Storage directory /usr/local/hadoop/tmp/dfs/name has been successfully formatted.

19/05/16 14:23:45 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

19/05/16 14:23:45 INFO util.ExitUtil: Exiting with status 0

19/05/16 14:23:45 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at java.net.UnknownHostException: dblab: dblab: 未知的名稱或服務

************************************************************/

解釋上述問題:

hadoop@dblab:/usr/local/hadoop$ hostname

dblab

hadoop@dblab:/usr/local/hadoop$ cat /etc/hosts

127.0.0.1?????? localhost

127.0.1.1?????? dblab-VirtualBox

# The following lines are desirable for IPv6 capable hosts

::1???? ip6-localhost ip6-loopback

fe00::0 ip6-localnet

ff00::0 ip6-mcastprefix

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

hadoop@dblab:/usr/local/hadoop$ sudo vim /etc/hosts

127.0.1.1?????? dblab

hadoop@dblab:/usr/local/hadoop$ sudo /etc/init.d/networking restart

4.啟動Hadoop

hadoop@dblab:/usr/local/hadoop$ ./sbin/start-dfs.sh

Starting namenodes on [localhost]

localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-dblab.out

localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-dblab.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-dblab.out

#以上是啟動失敗

hadoop@dblab:/usr/local/hadoop$ ./sbin/start-dfs.sh

Starting namenodes on [localhost]

localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-dblab.out

localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-dblab.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-dblab.out

HDFS的Web管理界面

5.運行Hadoop偽分布式實例

hadoop@dblab:/usr/local/hadoop$ cd /usr/local/hadoop

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir -p /user/hadoop

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs dfs -mkdir input

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs dfs -put ./etc/hadoop/*.xml input

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs dfs -ls input

Found 8 items

-rw-r--r--? ?1 hadoop supergroup? ? ? ?4436 2019-05-16 14:52 input/capacity-scheduler.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ? 965 2019-05-16 14:52 input/core-site.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ?9683 2019-05-16 14:52 input/hadoop-policy.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ?1080 2019-05-16 14:52 input/hdfs-site.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ? 620 2019-05-16 14:52 input/httpfs-site.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ?3518 2019-05-16 14:52 input/kms-acls.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ?5511 2019-05-16 14:52 input/kms-site.xml

-rw-r--r--? ?1 hadoop supergroup? ? ? ? 690 2019-05-16 14:52 input/yarn-site.xml

hadoop@dblab:/usr/local/hadoop$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar? grep input output 'dfs[a-z.]+'

hadoop@dblab:/usr/local/hadoop$ ./bin/hdfs dfs -cat output/*

1? ? ? ?dfsadmin

1? ? ? ?dfs.replication

1? ? ? ?dfs.namenode.name.dir

1? ? ? ?dfs.datanode.data.dir

6.關閉Hadoop

hadoop@dblab:/usr/local/hadoop$ ./sbin/stop-dfs.sh? ?#關閉Hadoop

7.啟動YARN

$./sbin/start-dfs.sh

hadoop@dblab:/usr/local/hadoop$ vim? /usr/local/hadoop/etc/hadoop/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

? <name>yarn.nodemanager.aux-services</name>

? <value>mapreduce_shuffle</value>

? </property>

</configuration>

hadoop@dblab:/usr/local/hadoop$ ./sbin/start-dfs.sh

Starting namenodes on [localhost]

localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-dblab.out

localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-dblab.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-dblab.out

hadoop@dblab:/usr/local/hadoop$ ./sbin/start-dfs.sh?

Starting namenodes on [localhost]

localhost: namenode running as process 28914. Stop it first.

localhost: datanode running as process 29067. Stop it first.

Starting secondary namenodes [0.0.0.0]

0.0.0.0: secondarynamenode running as process 29261. Stop it first.

hadoop@dblab:/usr/local/hadoop$ vim ./etc/hadoop/mapred-site.xml?

<property>

?<name>mapreduce.framework.name</name>

?<value>yarn</value>

?</property>

</configuration>

hadoop@dblab:/usr/local/hadoop$ vim ./etc/hadoop/yarn-site.xml?

<property>

? <name>yarn.nodemanager.aux-services</name>

? <value>mapreduce_shuffle</value>

? </property>

</configuration>

hadoop@dblab:/usr/local/hadoop$ ./sbin/start-yarn.sh?? #啟動YARN

starting yarn daemons

starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-dblab.out

localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-dblab.out

hadoop@dblab:/usr/local/hadoop$ ./sbin/mr-jobhistory-daemon.sh start historyserver? #開啟歷史服務器

starting historyserver, logging to /usr/local/hadoop/logs/mapred-hadoop-historyserver-dblab.out

hadoop@dblab:/usr/local/hadoop$ jps? ?#查看進程

29809 ResourceManager

28034 RunJar

28914 NameNode

30424 Jps

30248 JobHistoryServer

29067 DataNode

29933 NodeManager

29261 SecondaryNameNode

通過Web界面查看任務的運行情況

關閉YARN和Hadoop的腳本如下：

hadoop@dblab:/usr/local/hadoop$ ./sbin/stop-yarn.sh

stopping yarn daemons

stopping resourcemanager

localhost: stopping nodemanager

no proxyserver to stop

hadoop@dblab:/usr/local/hadoop$ ./sbin/mr-jobhistory-daemon.sh stop historyserver

stopping historyserver

hadoop@dblab:/usr/local/hadoop$ ./sbin/stop-dfs.sh

Stopping namenodes on [localhost]

localhost: stopping namenode

localhost: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode