作者： 边城元元原文来源：https://tidb.net/blog/b54581d1
一、背景
在安装 TiDB v6.0 的时候，使用 Tiup 扩容的方式安装 TiSpark 集群，最高的版本是 TiSpark v2.4.1，没有最新的 Release TiSpark v2.5.1 。另外，TiSpark v2.5.0 及以上版本实现了部分鉴权与授权功能。
本次主要是体验
二、准备环境
2.1 安装 Cluster111 (V6.0.0)
2.1.1 Cluster111 拓扑
# cluster111.ymlserver_configs: tidb: log.slow-threshold: 300 binlog.enable: false binlog.ignore-error: false tikv: readpool.storage.use-unified-pool: false readpool.coprocessor.use-unified-pool: true pd: schedule.leader-schedule-limit: 4 schedule.region-schedule-limit: 2048 schedule.replica-schedule-limit: 64 replication.location-labels: - hostpd_servers: - host: 10.0.2.15 # ssh_port: 22 # name: "pd-1" client_port: 2379 # peer_port: 2380tidb_servers: - host: 10.0.2.15tikv_servers: - host: 10.0.2.15 # ssh_port: 22 port: 20160 status_port: 20180 config: server.grpc-concurrency: 4monitoring_servers: - host: 10.0.2.15grafana_servers: - host: 10.0.2.15alertmanager_servers: - host: 10.0.2.15
复制代码
2.1.2 安装 Cluster111
# 安装tiupcurl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | shsource /root/.bash_profiletiup update clustertiup cluster list# 检测环境配置并尝试修正tiup cluster check ./cluster111.yml --user root -p --apply# 安装cluster111tiup cluster deploy cluster111 v6.0.0 ./cluster111.yml --user root -p# 启动集群tiup cluster start cluster111tiup cluster display cluster111
复制代码
2.2 TiSpark v2.4.1
2.2.1 拓扑
# cluster111-v6.0.0-tispark.ymltispark_masters: - host: 10.0.2.15 ssh_port: 22 port: 7077# NOTE: multiple worker nodes on the same host is not supported by Sparktispark_workers: - host: 10.0.2.15
复制代码
2.2.2 安装 TiSpark
安装 openjdk8 (略)
扩容的方式安装 TiSpark
2.3 测试 Spark v2.4.3 Standalone
/tidb-deploy/tispark-master-7077/sbin/start-all.sh
三、升级 TiSpark
3.1 下载升级软件
# 下载 Spark V3.1.3curl -L "https://dlcdn.apache.org/spark/spark-3.1.3/spark-3.1.3-bin-hadoop3.2.tgz" -O spark-3.1.3-bin-hadoop3.2.tgz# 下载 TiSpark V2.5.1curl -L "https://github.com/pingcap/tispark/releases/download/v2.5.1/tispark-assembly-3.1-2.5.1.jar" -O tispark-assembly-3.1-2.5.1.jar
复制代码
3.2 备份
\cp -rf /tidb-deploy/tispark-master-7077 /tidb-deploy/tispark-master-7077-bak2.4.1
复制代码
3.3 升级
# 替换 Sparkmkdir -p /usr/local0/webserver/tispark && tar -zxvf spark-3.1.3-bin-hadoop3.2.tgz -C /usr/local0/webserver/tispark/mv /usr/local0/webserver/tispark/spark-3.1.3-bin-hadoop3.2 /tidb-deploy/tispark-master-7077chown tidb.tidb -R /tidb-deploy/tispark-master-7077# 替换 TiSpark 包cp -rf tispark-assembly-3.1-2.5.1.jar /tidb-deploy/tispark-master-7077/jars/# 配置文件cp -rf /tidb-deploy/tispark-master-7077-bak2.4.1/conf/* /tidb-deploy/tispark-master-7077/conf/
复制代码
3.4 测试
/tidb-deploy/tispark-master-7077/sbin/start-all.sh
###
四、测试 TiSpark v2.5.1 鉴权
参考：https://github.com/pingcap/tispark/blob/master/docs/authorization_userguide.md
Authorization and authentication through TiDB server
4.1 增加配置 spark-defaults.conf
spark.sql.tidb.addr 10.0.2.15spark.sql.tidb.port 4000spark.sql.tidb.user rootspark.sql.tidb.password abc# Must config in conf filespark.sql.auth.enable true# in seconds. Values range from 5 to 3600spark.sql.tidb.auth.refreshInterval 30
复制代码
4.2 配置错误密码
#这里是错误的密码spark.sql.tidb.password abc
复制代码
启动 spark-sql 后使用 执行 sql 语句将报错
4.3 修正密码
# 空密码spark.sql.tidb.password # 开启下面的 30s 将刷新一下(仅对新连接的spark-sql 使用新配置的 spark.sql.tidb.password)spark.sql.tidb.auth.refreshInterval 30
复制代码
启动 spark-sql
/tidb-deploy/tispark-master-7077/bin/spark-sql
复制代码
use tidb_catalog;show databases;
复制代码
select 'CUSTOMER' tablename , count(*) ct from tidb_catalog.TPCH_001.CUSTOMER union allselect 'NATION' tablename , count(*) ct from tidb_catalog.TPCH_001.NATION union allselect 'REGION' tablename , count(*) ct from tidb_catalog.TPCH_001.REGION union allselect 'PART' tablename , count(*) ct from tidb_catalog.TPCH_001.PART union allselect 'SUPPLIER' tablename , count(*) ct from tidb_catalog.TPCH_001.SUPPLIER union allselect 'PARTSUPP' tablename , count(*) ct from tidb_catalog.TPCH_001.PARTSUPP union allselect 'ORDERS' tablename , count(*) ct from tidb_catalog.TPCH_001.ORDERS union allselect 'LINEITEM' tablename , count(*) ct from tidb_catalog.TPCH_001.LINEITEM order by ct desc;
复制代码
4.4 SparkSession 中配置密码
spark.sqlContext.setConf("spark.sql.tidb.addr", your_tidb_server_address)spark.sqlContext.setConf("spark.sql.tidb.port", your_tidb_server_port)spark.sqlContext.setConf("spark.sql.tidb.user", your_tidb_server_user)spark.sqlContext.setConf("spark.sql.tidb.password", your_tidb_server_password)
复制代码
4.5 限制
五、总结
本篇实践了 tiup list tispark –all 没有 TiSpark v2.5.x 的情况下，升级到 TiSpark v2.5.1；
同时试用了 TiSpark v2.5.x 新支持的鉴权特性。
谢谢！
参考
https://tidb.net/blog/19eeb447#Spark Standalone 集群升级步骤https://tidb.net/blog/b8f902a9#TiSpark 2.4.1(Spark 2.4.5) 到 TiSpark 2.5.0(Spark 3.0.X/3.1.X) 迁移实践https://github.com/pingcap/tispark/blob/master/docs/authorization_userguide.md
评论