Hadoop 3

3.关闭防火墙

sudo systemctl stop firewalld

sudo systemctl disable firewalld

4 . 在/opt 目录下创建文件夹

cd /opt

sudo mkdir software

5 .安装 JDK

（1）卸载现有 JDK

rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps

（2）安装 JDK (所有 tar 包上传至/opt/software 目录)

cd /opt/software

sudo tar -zxvf jdk-8u201-linux-x64.tar.gz

sudo vi /etc/profile

文件末尾添加

export JAVA_HOME=/opt/software/jdk1.8.0_201

export PATH= J A V A _ H O M E / b i n : JAVA\_HOME/bin: JAVA_HOME/bin:PATH

使环境变量生效

sudo source /etc/profile

java -version

6 . 安装 hadoop

(1) 下载 Hadoop 安装包

(2）解压 hadoop-3.1.3.tar.gz

tar -zxvf hadoop-3.1.3.tar.gz

(3) 配置环境变量

sudo vi /etc/profile

文件末尾添加

export HADOOP_HOME=/opt/software/hadoop-3.1.3

export PATH= P A T H : PATH: PATH:HADOOP_HOME/bin

export PATH= P A T H : PATH: PATH:HADOOP_HOME/sbin

使环境变量生效

sud 《一线大厂 Java 面试题解析+后端开发学习笔记+最新架构讲解视频+实战项目源码讲义》无偿开源 威信搜索公众号【编程进阶路】 o source /etc/profile

hadoop version

[](()配置集群（三台虚拟机都要执行）

1 . 配置 ssh 免密登录

（1）生成公钥和私钥

然后敲（三个回车），就会生成两个文件 id_rsa（私钥）、id_rsa.pub（公钥）

ssh-keygen -t rsa

（2）将公钥拷贝到要免密登录的目标机器上

ssh-copy-id hadoop001

ssh-copy-id hadoop002

ssh-copy-id hadoop003

2 . 集群配置

(1) 集群部署规划

组件

hadoop001

hadoop002

hadoop003

HDFS

NameNode,dataNode

dataNode

dataNode,SecondaryNameNode

YARN

NodeManager

ResourceManager,NodeManager

NodeManager

(2)配置集群

配置 core-site.xml

cd $HADOOP_HOME/etc/hadoop

vim core-site.xml

文件内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<property>

<name>fs.defaultFS</name>

<value>hdfs://hadoop001:9870</value>

</property>

<property>

<name>hadoop.tmp.dir</name>

<value>/opt/software/hadoop-3.1.3/data</value>

</property>

<property>

<name>hadoop.proxyuser.atguigu.hosts</name>

<value>*</value>

</property>

<property>

<name>hadoop.proxyuser.root.groups</name>

<value>*</value>

</property>

<property>

<name>hadoop.http.staticuser.user</name>

<value>root</value>

</property>

</configuration>

配置 hdfs-site.xml

vim hdfs-site.xml

文件内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>hadoop003:9868</value>

</property>

</configuration>

配置 yarn-site.xml

vim yarn-site.xml

文件内容如下：

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.hostname</name>

<value>hadoop002</value>

</property>

<property>

<name>yarn.nodemanager.env-whitelist</name>

<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>

</property>

<property>

<name>yarn.scheduler.minimum-allocation-mb</name>

<value>512</value>

</property>

<property>

<name>yarn.scheduler.maximum-allocation-mb</name>

<value>4096</value>

</property>

<property>

<name>yarn.nodemanager.resource.memory-mb</name>

<value>4096</value>

</property>

<property>

<name>yarn.nodemanager.pmem-check-enabled</name>

<value>false</value>

</property>

<property>

<name>yarn.nodemanager.vmem-check-enabled</name>