大數據：spark集群搭建

創建spark用戶組，組ID1000

groupadd -g 1000 spark

在spark用戶組下創建用戶ID 2000的spark用戶 獲取視頻中文檔資料及完整視頻的伙伴請加QQ群：947967114
useradd -u 2000 -g spark spark

設置密碼

passwd spark

修改sudo權限

chmod u+w /etc/sudoers

vi /etc/sudoers

找到

root ALL=(ALL) ALL

添加

spark ALL=(ALL) ALL

創建一個app目錄用來存放spark的軟件環境（jdk、scala、spark）

mkdir /app

修改這個文件的屬組和屬主

chown -R spark:spark /app

創建soft

mkdir /app/soft

創建spark

mkdir /app/spark

創建/spark/work

mkdir -p /home/spark/work

改變/spark/work屬組和屬主

chown -R spark:spark /home/spark/work

切換用戶

su root

解壓JDK

cd /tmp/

tar zxvf jdk-8u192-linux-x64.tar.gz -C /app/soft/

如果沒有權限首先使用chmod 777 -R /tmp修改權限

cd /app/soft/

ll -a

配置/etc/profile

sudo vi /etc/profile，所有需要的配置都添加了

JAVA_HOME=/app/soft/jdk1.8.0_192

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin

export PATH

讓配置生效 獲取視頻中文檔資料及完整視頻的伙伴請加QQ群：947967114
source /etc/profile

安裝scala：

tar zxvf /tmp/scala-2.11.12.tgz -C /app/soft/

配置環境變量

sudo vi /etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

PATH=$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

export PATH

配置ssh無秘登錄

ssh-keygen -t rsa

cd ~/

cd .ssh/

修改公鑰的名字

master節點：mv id_rsa.pub authorized_keys_master.pub

slave1節點：mv id_rsa.pub authorized_keys_slave1.pub

slave2節點：mv id_rsa.pub authorized_keys_slave2.pub

把slave1和slave2的公鑰給master

slave1節點：scp authorized_keys_slave1.pub spark@master:/home/spark/.ssh/

slave2節點：scp authorized_keys_slave2.pub spark@master:/home/spark/.ssh/

把三個節點的公鑰都寫在一個文件中

cat authorized_keys_master.pub >> authorized_keys

cat authorized_keys_slave1.pub >> authorized_keys

cat authorized_keys_slave2.pub >> authorized_keys

查看一下總的公鑰文件

vi authorized_keys

把總的公鑰文件authorized_keys給到slave1和slave2節點

scp authorized_keys spark@slave1:/home/spark/.ssh

scp authorized_keys spark@slave2:/home/spark/.ssh

修改authorized_keys的操作權限，三個節點都需要修改

chmod 400 authorized_keys

驗證免密登錄是否成功

ssh master

ssh slave1

ssh slave2

ssh master

安裝spark：

tar -zxf /tmp/spark-2.1.0-bin-hadoop2.6.gz -C /app/spark/

cd /app/spark/

ls

cd spark-2.1.0-bin-hadoop2.6/

配置環境變量：

vi /etc/profile

JAVA_HOME=/app/soft/jdk1.8.0_192

SCALA_HOME=/app/soft/scala-2.11.12/

SPARK_HOME=/app/spark/spark-2.1.0-bin-hadoop2.6

PATH=$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin:$PATH:$HOME/bin:$SCALA_HOME/bin

export PATH

配置spark的核心文件：

cd spark-2.1.0-bin-hadoop2.6/

cd conf/

配置slaves

mv slaves.template slaves

vi slaves 添加三個節點

master

slave1

slave2

配置spark-env.sh

cp spark-env.sh.template spark-env.sh

vi spark-env.sh

export JAVA_HOME=/app/soft/jdk1.8.0_192

export SCALA_HOME=/app/soft/scala-2.11.12

export SPARK_MASTER_IP=master

export SPARK_MASTER_PORT=7077

export SPARK_EXECUTOR_INSTANCES=1

export SPARK_WORKER_INSTANCES=1

export SPARK_WORKER_CORES=1

export SPARK_WORKER_MEMORY=1024M

export SPARK_MASTER_WEBUI=8080

export SPARK_CONF_DIR=/app/spark/spark-2.1.0-bin-hadoop2.6/conf/

把所有的節點的app的work和soft權限都改成777：在所有的節點上執行 chmod 777 -R /app/soft 和chmod 777 -R /app/spark

scp -r /app/spark/ spark@slave1:/app/

scp -r /app/soft/ spark@slave1:/app/

到此spark集群已經搭建完成：

開啟：start-all.sh獲取視頻中文檔資料及完整視頻的伙伴請加QQ群：947967114
jps可以看到如下進程：

master節點：

3617 Worker

3507 Master

4156 Jps

slave1節點：

3361 Worker

3702 Jps

slave2節點：

3319 Worker

3647 Jps

開啟spark-shell驗證：

spark-shell --master spark://master:7077 --executor-memory 1024m --driver-memory 1024m

啟動之后會顯示如下內容：

18/11/29 16:13:46 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

18/11/29 16:13:47 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as 'sc' (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as 'spark'.

Welcome to

  ____              __

 / __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  '_/

// ./_,// //_\ version 2.1.0

  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_192)

Type in expressions to have them evaluated.

Type :help for more information.

scala>

就可以在>后面書寫spark代碼了：

g NoSuchObjectException

Spark context Web UI available at http://192.168.0.10:4040

Spark context available as 'sc' (master = spark://master:7077, app id = app-20181129161336-0000).

Spark session available as 'spark'.

Welcome to

  ____              __

 / __/__  ___ _____/ /__

_\ \/ _ \/ _ `/ __/  '_/

// ./_,// //_\ version 2.1.0

  /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_192)

Type in expressions to have them evaluated.

Type :help for more information.

scala> sc.textFile("/app/spark/spark-2.1.0-bin-hadoop2.6/README.md").flatMap(.split(" ")).map(x=>(x,1)).reduceByKey(+_).map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1)).take(10)

res0: Array[(String, Int)] = Array(("",71), (the,24), (to,17), (Spark,16), (for,12), (and,9), (##,8), (a,8), (can,7), (run,7))

scala>獲取視頻中文檔資料及完整視頻的伙伴請加QQ群：947967114