2 年 ago
清, 宇
6 minutes

Home brewでApache Kafkaインストール

gradleが入っていない場合gradleのインストールも始まる。
MacBookAirで10分以上時間かかったので注意

brew install kafka

==> Installing kafka
==> Downloading http://mirrors.ibiblio.org/apache/kafka/0.8.2.1/kafka-0.8.2.1-src.tgz
######################################################################## 100.0%
==> gradle
==> gradle jar


==> Caveats
To start Kafka, ensure that ZooKeeper is running and then execute:
  kafka-server-start.sh /usr/local/etc/kafka/server.properties

To have launchd start kafka at login:
    ln -sfv /usr/local/opt/kafka/*.plist ~/Library/LaunchAgents
Then to load kafka now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.kafka.plist
==> Summary
  /usr/local/Cellar/kafka/0.8.2.1: 3817 files, 38M, built in 12.9 minutes

Zookeeperもインストールする

brew intall zookeeper

Zookeeper 起動

zkServer start

kafka 起動

$ kafka-server-start.sh /usr/local/etc/kafka/server.properties
[2015-06-04 16:01:41,857] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,930] INFO Property log.dirs is overridden to /usr/local/var/lib/kafka-logs (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,931] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,932] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:41,933] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2015-06-04 16:01:42,002] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2015-06-04 16:01:42,006] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2015-06-04 16:01:42,022] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-06-04 16:01:42,036] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:host.name=192.168.107.80 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.version=1.7.0_79 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,036] INFO Client environment:java.class.path=:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/jopt-simple-3.2.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/log4j-1.2.16.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/lz4-1.2.0.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/metrics-core-2.2.0.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/scala-library-2.10.4.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/slf4j-api-1.7.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/slf4j-log4j12-1.7.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/snappy-java-1.1.1.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/zkclient-0.3.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/dependant-libs-2.10.4/zookeeper-3.4.6.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../examples/build/libs//kafka-examples-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../clients/build/libs/kafka-clients-0.8.2.1.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../libs/*.jar:/usr/local/Cellar/kafka/0.8.2.1/libexec/bin/../core/build/libs/kafka_2.10-0.8.2.1.jar (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.library.path=/Users/xxxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.io.tmpdir=/var/folders/xq/cjwpdzb90_zdsrhcfd4ckgyc0000gq/T/ (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:os.version=10.9.5 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.name=xxxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.home=/Users/xxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,037] INFO Client environment:user.dir=/Users/xxx (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,038] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@675c2785 (org.apache.zookeeper.ZooKeeper)
[2015-06-04 16:01:42,078] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,090] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,157] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14dbd5f98f80000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-06-04 16:01:42,158] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2015-06-04 16:01:42,312] INFO Log directory ‘/usr/local/var/lib/kafka-logs’ not found, creating it. (kafka.log.LogManager)
[2015-06-04 16:01:42,329] INFO Loading logs. (kafka.log.LogManager)
[2015-06-04 16:01:42,339] INFO Logs loading complete. (kafka.log.LogManager)
[2015-06-04 16:01:42,340] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2015-06-04 16:01:42,347] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2015-06-04 16:01:42,390] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2015-06-04 16:01:42,391] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2015-06-04 16:01:42,515] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-06-04 16:01:42,576] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-06-04 16:01:42,705] INFO Registered broker 0 at path /brokers/ids/0 with address 192.168.107.80:9092. (kafka.utils.ZkUtils$)
[2015-06-04 16:01:42,727] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-06-04 16:01:42,815] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
“`

トピック作成

/usr/local/bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

ログ
Created topic “test”

メッセージ送信

/usr/local/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2015-06-04 16:11:01,693] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
HELLO Kafka
second line
(^_^)
あいうえお

メッセージ受信

/usr/local/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
HELLO Kafka
second line
(^_^)
あいうえお
✖
广告
将在 10 秒后关闭
bannerAds