Flume 的负载均衡 load balancer
在此处我们通过三台机器来进行模拟 flume 的负载均衡
三台机器规划如下：
node01：采集数据，发送到 node02 和 node03 机器上去
node02：接收 node01 的部分数据
node03：接收 node01 的部分数据
开发 node01 服务器的 flume 配置
node01 服务器配置：
#a1 namea1.channels = c1a1.sources = r1a1.sinks = k1 k2###set gruopa1.sinkgroups = g1###set channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100#a1.sources.r1.channels = c1a1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources.r1.port = 44444
## set sink1a1.sinks.k1.channel = c1a1.sinks.k1.type = avroa1.sinks.k1.hostname = node02a1.sinks.k1.port = 52020### set sink2a1.sinks.k2.channel = c1a1.sinks.k2.type = avroa1.sinks.k2.hostname = node03a1.sinks.k2.port = 52020###set sink groupa1.sinkgroups.g1.sinks = k1 k2#load_banlancea1.sinkgroups.g1.processor.type = load_balancea1.sinkgroups.g1.processor.backoff = truea1.sinkgroups.g1.processor.selector = round_robina1.sinkgroups.g1.processor.selector.maxTimeOut=10000
复制代码
开发 node02 服务器的 flume 配置
# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1
# Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.channels = c1a1.sources.r1.bind = node02a1.sources.r1.port = 52020
# Describe the sinka1.sinks.k1.type = logger
# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1
复制代码
开发 node03 服务器 flume 配置
# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1
# Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.channels = c1a1.sources.r1.bind = node03a1.sources.r1.port = 52020
# Describe the sinka1.sinks.k1.type = logger
# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1
复制代码
准备启动 flume 服务
启动 node03 的 flume 服务
bin/flume-ngagent -n a1 -c conf -f conf/load_banlancer_server.conf-Dflume.root.logger=DEBUG,console
复制代码
启动 node02 的 flume 服务
bin/flume-ngagent -n a1 -c conf -f conf/load_banlancer_server.conf-Dflume.root.logger=DEBUG,console
复制代码
启动 node01 的 flume 服务
bin/flume-ngagent -n a1 -c conf -f conf/load_banlancer_client.conf -Dflume.root.logger=DEBUG,console
复制代码
发布于: 2021 年 05 月 19 日阅读数: 8
评论