ol7.7安裝部署4節(jié)點(diǎn)spark3.0.0分布式集群的詳細(xì)教程

為學(xué)習(xí)spark，虛擬機(jī)中開4臺(tái)虛擬機(jī)安裝spark3.0.0

底層hadoop集群已經(jīng)安裝好，見ol7.7安裝部署4節(jié)點(diǎn)hadoop 3.2.1分布式集群學(xué)習(xí)環(huán)境

首先，去http://spark.apache.org/downloads.html下載對(duì)應(yīng)安裝包

解壓

[hadoop@master ~]$ sudo tar -zxf spark-3.0.0-bin-without-hadoop.tgz -C /usr/local
[hadoop@master ~]$ cd /usr/local
[hadoop@master /usr/local]$ sudo mv ./spark-3.0.0-bin-without-hadoop/ spark
[hadoop@master /usr/local]$ sudo chown -R hadoop: ./spark

四個(gè)節(jié)點(diǎn)都添加環(huán)境變量

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

配置spark

spark目錄中的conf目錄下cp ./conf/spark-env.sh.template ./conf/spark-env.sh后面添加

export SPARK_MASTER_IP=192.168.168.11
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_LOCAL_DIRS=/usr/local/hadoop
export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)

然后配置work節(jié)點(diǎn)，cp ./conf/slaves.template ./conf/slaves修改為

master
slave1
slave2
slave3

寫死JAVA_HOME，sbin/spark-config.sh最后添加

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_191

復(fù)制spark目錄到其他節(jié)點(diǎn)

sudo scp -r /usr/local/spark/ slave1:/usr/local/
sudo scp -r /usr/local/spark/ slave2:/usr/local/
sudo scp -r /usr/local/spark/ slave3:/usr/local/
sudo chown -R hadoop ./spark/

...

啟動(dòng)集群

先啟動(dòng)hadoop集群/usr/local/hadoop/sbin/start-all.sh

然后啟動(dòng)spark集群

通過master8080端口監(jiān)控

完成安裝

到此這篇關(guān)于ol7.7安裝部署4節(jié)點(diǎn)spark3.0.0分布式集群的詳細(xì)教程的文章就介紹到這了,更多相關(guān)ol7.7安裝部署spark集群內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！