91超碰碰碰碰久久久久久综合_超碰av人澡人澡人澡人澡人掠_国产黄大片在线观看画质优化_txt小说免费全本

溫馨提示×

溫馨提示×

您好,登錄后才能下訂單哦!

密碼登錄×
登錄注冊×
其他方式登錄
點擊 登錄注冊 即表示同意《億速云用戶服務條款》

Spark 集群部署(MasterHA)

發布時間:2020-06-24 23:39:46 來源:網絡 閱讀:1643 作者:Lynn_Yuan 欄目:大數據

一. 前提條件

引用

Spark Standalone集群是Master-Slaves架構的集群模式,和大部分的Master-Slaves結構集群一樣,存在著Master單點故障的問題。如何解決這個單點故障的問題,Spark提供了兩種方案:

基于文件系統的單點恢復(Single-Node Recovery with Local File System)
基于zookeeper的Standby Masters(Standby Masters with ZooKeeper)
ZooKeeper提供了一個Leader Election機制,利用這個機制可以保證雖然集群存在多個Master,但是只有一個是Active的,其他的都是Standby。當Active的Master出現故障時,另外的一個Standby Master會被選舉出來。由于集群的信息,包括Worker, Driver和Application的信息都已經持久化到文件系統,因此在切換的過程中只會影響新Job的提交,對于正在進行的Job沒有任何的影響。加入ZooKeeper的集群整體架構

Zookeeper集群正常運行

二. 部署步驟

  1. 下載Spark程序壓縮包
    wget http://mirrors.shu.edu.cn/apache/spark/spark-2.4.0/spark-2.4.0-bin-hadoop2.7.tgz
  2. 解壓縮并重命名
    tar -zxvf spark-2.4.0-bin-hadoop2.7.tgz -C /opt
    mv spark-2.4.0-bin-hadoop2.7 spark-2.4.0
  3. 配置環境變量
    /etc/profile
    export JAVA_HOME=/usr/lib/jdk1.8.0_172
    export CLASSPATH=${JAVA_HOME}/jre/lib:${JAVA_HOME}/lib
    export HADOOP_HOME=/opt/hadoop-2.7.6
    export SPARK_HOME=/opt/spark-2.4.0
    export PATH=${JAVA_HOME}/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$PATH

    修改機器名稱

    hostnamectl set-hostname res-spark-0001

    執行命令使得環境變量生效

    source /etc/profile
  4. 修改配置文件
cd /opt/spark-2.4.0/conf
cp log4j.properties.template log4j.properties
cp slaves.template slaves
cp spark-env.sh.template spark-env.sh
cp spark-defaults.conf.template spark-defaults.conf

4.1 slaves

res-spark-0003
res-spark-0004
res-spark-0005

4.2 spark-defaults.conf

spark.deploy.recoveryMode          ZOOKEEPER
spark.deploy.zookeeper.url         res-spark-0001:2181,res-spark-0002:2181,res-spark-0003:2181

spark.master                       spark://res-spark-0001:7077
spark.eventLog.enabled             true
spark.eventLog.dir                 hdfs://cluster1/spark/eventLog

spark.shuffle.service.enabled  true

4.3 spark-env.sh

export JAVA_HOME=/usr/lib/jdk1.8.0_172
export HADOOP_HOME=/opt/hadoop-2.7.6
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

export SPARK_HOME=/opt/spark-2.4.0

export SPARK_WORKER_CORES=6
export SPARK_WORKER_MEMORY=24g

4.4 log4j.properties

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Set everything to be logged to the console
log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
  1. 分發spark程序以及配置文件到其他節點

    scp -r /opt/spark-2.4.0 res-spark-0002:/opt
    scp -r /opt/spark-2.4.0 res-spark-0003:/opt
    scp -r /opt/spark-2.4.0 res-spark-0004:/opt
    scp -r /opt/spark-2.4.0 res-spark-0005:/opt
  2. 修改 res-spark-0002節點的配置文件
    6.1 spark-defaults.conf

    spark.master                       spark://res-spark-0002:7077
  3. 啟動集群
cd sbin
./start-all.sh

res-spark-0002節點

cd sbin
./start-master.sh
  1. 測試
    res-spark-0001節點執行
    ./stop-master.sh

    得到如下結果
    Spark 集群部署(MasterHA)

Spark 集群部署(MasterHA)

  1. submit app
    spark-submit --master spark://res-spark-0001:7077  --driver-cores 4 --driver-memory 6g --conf spark.dynamicAllocation.enabled=true --conf spark.shuffle.service.enabled=true --class com.cloud.RuleEngine rule-engine-1.0-SNAPSHOT-jar-with-dependencies.jar

    報錯信息

    18/12/30 08:47:41 ERROR TaskSchedulerImpl: Lost executor 3 on 172.16.0.24: Unable to create executor due to Unable to register with external shuffle server due to : Failed to connect to /172.16.0.24:7337

    官網:

    In standalone mode, simply start your workers with spark.shuffle.service.enabled set to true.

向AI問一下細節

免責聲明:本站發布的內容(圖片、視頻和文字)以原創、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。

AI

马鞍山市| 黔西县| 昌平区| 衡山县| 武胜县| 罗田县| 通江县| 黔西县| 江陵县| 龙泉市| 宣化县| 竹山县| 寿宁县| 定兴县| 新兴县| 葫芦岛市| 禄丰县| 肥城市| 新疆| 宝清县| 宣城市| 镇原县| 昌邑市| 东丽区| 双辽市| 太康县| 广州市| 寻甸| 洛南县| 布尔津县| 宜丰县| 清流县| 会理县| 岑溪市| 茌平县| 无锡市| 万盛区| 勃利县| 临城县| 巴中市| 新巴尔虎右旗|