kafka

应用售价:

0 积分

版本号:

2.11-0.10.2.1

购买企业:
备注:本应用必须在Rainbond系统上安装使用,点击免费下载安装
应用详情

发布时间:2019-06-14

最后更新时间:2019-07-01

<h1>rainbond-kafka</h1> <h2>什么是kafka</h2> <p>Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写。该项目的目标是为处理实时数据提供一个统一、高吞吐、低延迟的平台。其持久化层本质上是一个“按照分布式事务日志架构的大规模发布/订阅消息队列”,这使它作为企业级基础设施来处理流式数据非常有价值。</p> <h2>部署说明</h2> <ul> <li>该应用包含4个组件:kafka, kafka-dashboard, zookeeper, zookeeper-dashboard。</li> <li>安装该应用后,先要将zookeeper的实例数改为3,且目前只支持3个实例,否则kafka可能启动失败。</li> <li>在kafka中创建topic后,请小心使用伸缩功能,因为这可能影响您已保存到kafka中的数据,具体请参考<a href="https://kafka.apache.org/documentation" rel="nofollow">kafka分区与副本</a>相关文档。</li> </ul> <h2>配置说明</h2> <table> <thead> <tr> <th>配置项</th> <th>默认值</th> <th>描述</th> </tr> </thead> <tbody> <tr> <td>LISTENERS</td> <td>PLAINTEXT://:9093</td> <td>broker服务监听端口</td> </tr> <tr> <td>NUM_REPLICA_FETCHERS</td> <td>1</td> <td>数据副本数量</td> </tr> <tr> <td>LOG_RETENTION_HOURS</td> <td>168</td> <td>日志文件删除之前保留的小时数</td> </tr> <tr> <td>LOG_RETENTION_BYTES</td> <td>-1</td> <td>日志文件删除之前的最大大小,<code>-1</code>永不过期</td> </tr> <tr> <td>DEBUG</td> <td>true</td> <td>打印应用启动脚本的执行日志</td> </tr> </tbody> </table> <p>如需设置更多参数,请参考<a href="https://kafka.apache.org/documentation/#brokerconfigs" rel="nofollow">kafka官方文档</a>。</p> <h2>配置方式</h2> <ol> <li>在<a href="https://kafka.apache.org/documentation/#brokerconfigs" rel="nofollow">kafka官方文档</a>找到需要的配置项,假设为:<code>my.config.name</code>。</li> <li>将该配置项的名字全部大写且将<code>.</code>转为<code>_</code>,得到名字:<code>MY_CONFIG_NAME</code>。</li> <li>在云帮管理页面中找到应用,将该配置填入到应用的自定义环境变量中,如下图所示。 <img src="http://grstatic.oss-cn-shanghai.aliyuncs.com/images/docs/common/app-add-env.jpg" alt="app-add-env"/></li> <li>重启应用生效。</li> </ol> <h2>kafka-dashboard使用说明</h2> <ol> <li>点击<code>kafka-dashboard</code>应用页面的访问按钮进入kafka仪表盘页面。</li> <li>点击<code>Add Cluster</code>按钮</li> <li>输入名字和zookeeper地址(固定地址:127.0.0.1:2181) <img src="http://grstatic.oss-cn-shanghai.aliyuncs.com/images/docs/common/kafka-dashboard-add-cluster.jpg" alt="add-cluster"/></li> <li>点击页面最下方的<code>Save</code>,然后就可以查看kafka集群的各个状态了。</li> </ol> <h2>基准测试-集群内连接</h2> <h3>启动kafka</h3> <p>安装kafka,并将kafka实例数设置为3,每个实例分配1G内存,并打开内部端口。</p> <h3>启动kafka-client</h3> <p>因为kafka的生产者和消费者也会占用一部分内存,所以我们让这两个进程运行在一个单独的容器中,首先从源码安装kafka-clinet,并将内存设置为1G,并依赖kafka和zookeeper:</p> <pre><code>https://github.com/goodrain-apps/kafka.git?dir=client </code></pre> <p>以下测试命令全部在kafka-client容器中执行。</p> <h3>创建topic</h3> <pre><code>kafka-topics.sh --create \ --zookeeper $ZOOKEEPER_HOST:$ZOOKEEPER_PORT \ --replication-factor 3 \ --partitions 3 \ --topic test1 </code></pre> <h3>启动消费者</h3> <pre><code>kafka-console-consumer.sh \ --zookeeper $ZOOKEEPER_HOST:$ZOOKEEPER_PORT \ --from-beginning \ --topic test1 &gt; /dev/null </code></pre> <h3>启动生产者</h3> <pre><code>time kafka-producer-perf-test.sh \ --topic test1 \ --num-records $((10000)) \ --throughput $((100)) \ --record-size $((1024*100)) \ --producer-props \ acks=all \ bootstrap.servers=$KAFKA_HOST:$KAFKA_PORT \ buffer.memory=$((1024*1024*1024)) \ compression.type=none \ batch.size=0 </code></pre> <h3>生产者执行结果</h3> <pre><code>502 records sent, 100.3 records/sec (9.79 MB/sec), 16.6 ms avg latency, 179.0 max latency. 501 records sent, 100.1 records/sec (9.77 MB/sec), 4.5 ms avg latency, 47.0 max latency. 500 records sent, 99.9 records/sec (9.76 MB/sec), 3.6 ms avg latency, 34.0 max latency. 500 records sent, 99.9 records/sec (9.76 MB/sec), 57.1 ms avg latency, 696.0 max latency. 501 records sent, 100.0 records/sec (9.76 MB/sec), 4.0 ms avg latency, 39.0 max latency. 501 records sent, 100.0 records/sec (9.77 MB/sec), 3.0 ms avg latency, 24.0 max latency. 501 records sent, 100.1 records/sec (9.78 MB/sec), 3.0 ms avg latency, 22.0 max latency. 500 records sent, 99.9 records/sec (9.76 MB/sec), 4.0 ms avg latency, 70.0 max latency. 501 records sent, 100.1 records/sec (9.78 MB/sec), 3.0 ms avg latency, 12.0 max latency. 499 records sent, 99.6 records/sec (9.73 MB/sec), 3.5 ms avg latency, 47.0 max latency. 502 records sent, 100.3 records/sec (9.79 MB/sec), 4.0 ms avg latency, 56.0 max latency. 500 records sent, 100.0 records/sec (9.76 MB/sec), 3.8 ms avg latency, 58.0 max latency. 501 records sent, 100.1 records/sec (9.78 MB/sec), 3.3 ms avg latency, 34.0 max latency. 501 records sent, 100.1 records/sec (9.77 MB/sec), 3.4 ms avg latency, 29.0 max latency. 500 records sent, 99.9 records/sec (9.75 MB/sec), 3.3 ms avg latency, 36.0 max latency. 500 records sent, 100.0 records/sec (9.76 MB/sec), 19.2 ms avg latency, 527.0 max latency. 501 records sent, 100.1 records/sec (9.77 MB/sec), 3.2 ms avg latency, 30.0 max latency. 501 records sent, 100.0 records/sec (9.77 MB/sec), 61.4 ms avg latency, 677.0 max latency. 501 records sent, 100.0 records/sec (9.77 MB/sec), 4.3 ms avg latency, 61.0 max latency. 10000 records sent, 99.995000 records/sec (9.77 MB/sec), 10.61 ms avg latency, 696.00 ms max latency, 3 ms 50th, 14 ms 95th, 308 ms 99th, 649 ms 99.9th. real 1m41.007s user 0m9.354s sys 0m2.324s </code></pre> <h2>基准测试-集群外连接</h2> <p>目前kafka不支持从云帮外部连接使用。</p>