IMPALA2.12 环境安装

大家好，我是怀瑾握瑜，一只大数据萌新，家有两只吞金兽，嘉与嘉，上能 code 下能 teach 的全能奶爸

如果您喜欢我的文章，可以[关注⭐]+[点赞👍]+[评论📃]，您的三连是我前进的动力，期待与您共同成长~

1. IMPALA 环境准备

1、上传 IMPALA.tar.gz 至/var/lib/ambari-server/resources/stacks/HDP/3.1/services 并解压

2、上传 IMPALA_CDH.tar.gz 至/var/www/html 并解压

3、上传 impala.repo 镜像至/etc/yum.repos.d/

网页段可以打开相应页面

http://172.29.30.61/IMPALA_CDH/

2. 重启 ambari-server

# service ambari-server restart

复制代码

3. 安装 ampala

1、登陆 ambari 页面管理

2、首页左下角点击 add service，勾选未安装的 impala 进行安装

3、选择需要安装到的主机即可

安装可能会遇到一些报错，目前有人如下解决方案：

1）

Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>

BeforeAnyHook().execute()

File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute

method(env)

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook

setup_users()

File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users

groups = params.user_to_groups_dict[user],

KeyError: u'impala'

Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-863.json', '/

ambari 主节点执行命令行，需要有 python 运行环境，可以尝试使用默认的，报错就需要手动安装

$cluster_name 为 ambari 的 cluster 名称，这里为 Sea

$ambari_server 为 ambari 地址，这里为 172.29.30.61

# cd /var/lib/ambari-server/resources/scripts

# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create

"ignore_groupsusers_create": "false",

# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

复制代码

2）所有安装 impala 节点修改 python 文件

# vim /var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA/package/scripts/impala-catalog.py

复制代码

把涉及地址的那三行代码删除，最终结果如下：

执行后重新安装

4. 修改 HDFS 相关配置文件

在 ambari 上操作，找到 hdfs 的 config

找到 Custom core-site，点击 Add property

添加如下内容

dfs.client.read.shortcircuit=true

dfs.client.read.shortcircuit.skip.checksum=false

dfs.datanode.hdfs-blocks-metadata.enabled=true

复制代码

找到 Custom hdfs-site,点击 Add property

dfs.datanode.hdfs-blocks-metadata.enabled=true

dfs.block.local-path-access.user=impala 如果报错的话就改成 hadoop

dfs.client.file-block-storage-locations.timeout.millis=60000

复制代码

重启 hdfs

5. 修改 impala 配置（所有机器）

# vim /etc/default/impala

复制代码

在 MEM_LIMIT=20gb 配置下添加 kudu 地址（前提为 kudu 已安装完成，但该步骤不影响 impala 整体安装）：

KUDU_MASTER_HOST=xxx3.hadoop.com:7051

复制代码

在 IMPALA_SERVER_ARGS 配置项最后添加如下配置：

-kudu_master_hosts=${KUDU_MASTER_HOST} \

复制代码

6. 启动 impala 前准备工作（所有节点）

1、查看/etc/impala/conf/下是否有 core-site.xml hdfs-site.xml hive-site.xml

如果没有需要从 hive 和 hdfs 的 conf 中同步以上三个 xml 文件

2、创建 impala 目录并添加相关授权

各节点将 impala 用户添加到组中

# usermod -G hive,hdfs,hadoop impala

复制代码

创建 impala 相关目录并授权

# su - hdfs -c "hadoop fs -mkdir /user/impala"

# su - hdfs -c "hadoop fs -chown -R impala /user/impala"

复制代码

3、配置 bigtop-utils 的 JAVA_HOME

# vim /etc/default/bigtop-utils

export JAVA_HOME=/app/tools/java/jdk1.8.0_201

复制代码

4、把 hadoop 低版本的 jar 包上传并创建软连接到/usr/lib/impala/lib

上传 hdp-2.6.4.0-91_hadoop_lib.zip 文件到/usr/hdp/并解压缩

执行如下软连接命令

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-annotations.jar /usr/lib/impala/lib/hadoop-annotations.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-archives-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-archives.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-auth.jar /usr/lib/impala/lib/hadoop-auth.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-aws.jar /usr/lib/impala/lib/hadoop-aws.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-common.jar /usr/lib/impala/lib/hadoop-common.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-hdfs-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-hdfs.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-mapreduce-client-common.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-mapreduce-client-core.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-mapreduce-client-jobclient.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-mapreduce-client-shuffle.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-yarn-api.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-yarn-client.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-yarn-common.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-yarn-server-applicationhistoryservice-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/hadoop-yarn-server-applicationhistoryservice.jar

ln -s -f /usr/hdp/hdp-2.6.4.0-91_hadoop_lib/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar /usr/lib/impala/lib/