写点什么

Hadoop 3

  • 2022 年 5 月 09 日
  • 本文字数:1618 字

    阅读完需:约 5 分钟

3.关闭防火墙


sudo systemctl stop firewalld


sudo systemctl disable firewalld


4 . 在/opt 目录下创建文件夹


cd /opt


sudo mkdir software


5 .安装 JDK


(1)卸载现有 JDK


rpm -qa | grep -i java | xargs -n1 sudo rpm -e --nodeps


(2)安装 JDK (所有 tar 包上传至/opt/software 目录)


cd /opt/software


sudo tar -zxvf jdk-8u201-linux-x64.tar.gz


sudo vi /etc/profile


文件末尾添加


export JAVA_HOME=/opt/software/jdk1.8.0_201


export PATH= J A V A _ H O M E / b i n : JAVA\_HOME/bin: JAVA_HOME/bin:PATH


使环境变量生效


sudo source /etc/profile


java -version


6 . 安装 hadoop


(1) 下载 Hadoop 安装包


(2)解压 hadoop-3.1.3.tar.gz


tar -zxvf hadoop-3.1.3.tar.gz


(3) 配置环境变量


sudo vi /etc/profile


文件末尾添加


export HADOOP_HOME=/opt/software/hadoop-3.1.3


export PATH= P A T H : PATH: PATH:HADOOP_HOME/bin


export PATH= P A T H : PATH: PATH:HADOOP_HOME/sbin


使环境变量生效


sud 《一线大厂 Java 面试题解析+后端开发学习笔记+最新架构讲解视频+实战项目源码讲义》无偿开源 威信搜索公众号【编程进阶路】 o source /etc/profile


hadoop version


[](()配置集群(三台虚拟机都要执行)




1 . 配置 ssh 免密登录


(1)生成公钥和私钥


然后敲(三个回车),就会生成两个文件 id_rsa(私钥)、id_rsa.pub(公钥)


ssh-keygen -t rsa


(2)将公钥拷贝到要免密登录的目标机器上


ssh-copy-id hadoop001


ssh-copy-id hadoop002


ssh-copy-id hadoop003


2 . 集群配置


(1) 集群部署规划


组件


hadoop001


hadoop002


hadoop003


HDFS


NameNode,dataNode


dataNode


dataNode,SecondaryNameNode


YARN


NodeManager


ResourceManager,NodeManager


NodeManager


(2)配置集群


配置 core-site.xml


cd $HADOOP_HOME/etc/hadoop


vim core-site.xml


文件内容如下:


<?xml version="1.0" encoding="UTF-8"?>


<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<property>


<name>fs.defaultFS</name>


<value>hdfs://hadoop001:9870</value>


</property>


<property>


<name>hadoop.tmp.dir</name>


<value>/opt/software/hadoop-3.1.3/data</value>


</property>


<property>


<name>hadoop.proxyuser.atguigu.hosts</name>


<value>*</value>


</property>


<property>


<name>hadoop.proxyuser.root.groups</name>


<value>*</value>


</property>


<property>


<name>hadoop.http.staticuser.user</name>


<value>root</value>


</property>


</configuration>


配置 hdfs-site.xml


vim hdfs-site.xml


文件内容如下:


<?xml version="1.0" encoding="UTF-8"?>


<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>


<property>


<name>dfs.namenode.secondary.http-address</name>


<value>hadoop003:9868</value>


</property>


</configuration>


配置 yarn-site.xml


vim yarn-site.xml


文件内容如下:


<?xml version="1.0" encoding="UTF-8"?>


<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>


<configuration>


<property>


<name>yarn.nodemanager.aux-services</name>


<value>mapreduce_shuffle</value>


</property>


<property>


<name>yarn.resourcemanager.hostname</name>


<value>hadoop002</value>


</property>


<property>


<name>yarn.nodemanager.env-whitelist</name>


<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>


</property>


<property>


<name>yarn.scheduler.minimum-allocation-mb</name>


<value>512</value>


</property>


<property>


<name>yarn.scheduler.maximum-allocation-mb</name>


<value>4096</value>


</property>


<property>


<name>yarn.nodemanager.resource.memory-mb</name>


<value>4096</value>


</property>


<property>


<name>yarn.nodemanager.pmem-check-enabled</name>


<value>false</value>


</property>


<property>


<name>yarn.nodemanager.vmem-check-enabled</name>

用户头像

还未添加个人签名 2022.04.13 加入

还未添加个人简介

评论

发布
暂无评论
Hadoop 3_Java_爱好编程进阶_InfoQ写作社区