{"id":1276,"date":"2019-03-28T08:11:21","date_gmt":"2019-03-28T00:11:21","guid":{"rendered":"http:\/\/jrblog.pentaidea.com\/john\/2019\/03\/28\/build-3-nodes-1-cluster-kafka-zookeeper-kafka-manager-envirnment-by-docker-compose-2\/"},"modified":"2019-03-28T08:11:21","modified_gmt":"2019-03-28T00:11:21","slug":"build-3-nodes-1-cluster-kafka-zookeeper-kafka-manager-envirnment-by-docker-compose-2","status":"publish","type":"post","link":"https:\/\/john.pentaidea.com\/?p=1276","title":{"rendered":"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose"},"content":{"rendered":"<ul>\n<li>\n<p>Create folder:<\/p>\n<\/li>\n<\/ul>\n<pre><code>sudo mkdir&nbsp;ka-data\nsudo mkdir zoo-data\nsudo mkdir zoo-datalog\nsudo mkdir km-conf\n<\/code><\/pre>\n<ul>\n<li>\n<p>Create .env file (you could also config it manually)<\/p>\n<\/li>\n<\/ul>\n<pre><code>#ID should be unique in 1,2,3...\n#SERVER_ZOO_HOST the current node should use 0.0.0.0\n\nID=?\nSERVER_KAFKA=kafka1.mydomain.com\nSERVER_ZOO_HOST=server.1=0.0.0.0:2888:3888 server.2=zookeeper2.mydomain.com:2888:3888 server.3=zookeeper3.mydomain.com:2888:3888\nSERVER_ZOO_CONNECT=zookeeper1.mydomain.com:2181,zookeeper2.mydomain.com:2181,zookeeper3.mydomain.com:2181\n<\/code><\/pre>\n<ul>\n<li>\n<p>Create application.conf for Kafka-manager (Ref: <a href=\"https:\/\/github.com\/lightbend\/kafka-manager-TO-BE-DELETED\/blob\/master\/conf\/application.conf\">https:\/\/github.com\/lightbend\/kafka-manager-TO-BE-DELETED\/blob\/master\/conf\/application.conf<\/a>)<\/p>\n<p><a href=\"http:\/\/jrblog.pentaidea.com\/john\/wp-content\/uploads\/sites\/2\/2020\/06\/3df4d85febf74a43fcdee2a80828fd19.conf\">application.conf<\/a><\/p>\n<\/li>\n<\/ul>\n<pre><code># Copyright 2015 Yahoo Inc. Licensed under the Apache License, Version 2.0\n\n# See accompanying LICENSE file.\n\n# This is the main configuration file for the application.\n# ~~~~~\n\n# Secret key\n# ~~~~~\n# The secret key is used to secure cryptographics functions.\n# If you deploy your application to several instances be sure to use the same key!\nplay.crypto.secret=&quot;^&lt;csmm5Fx4d=r2HEX8pelM3iBkFVv?k[mc;IZE&lt;_Qoq8EkX_\/7@Zt6dP05Pzea3U&quot;\nplay.crypto.secret=${?APPLICATION_SECRET}\n\n# The application languages\n# ~~~~~\nplay.i18n.langs=[&quot;en&quot;]\n\nplay.http.requestHandler = &quot;play.http.DefaultHttpRequestHandler&quot;\nplay.http.context = &quot;\/&quot;\nplay.application.loader=loader.KafkaManagerLoader\n\nkafka-manager.zkhosts=&quot;kafka-manager-zookeeper:2181&quot;\nkafka-manager.zkhosts=${?ZK_HOSTS}\npinned-dispatcher.type=&quot;PinnedDispatcher&quot;\npinned-dispatcher.executor=&quot;thread-pool-executor&quot;\napplication.features=[&quot;KMClusterManagerFeature&quot;,&quot;KMTopicManagerFeature&quot;,&quot;KMPreferredReplicaElectionFeature&quot;,&quot;KMReassignPartitionsFeature&quot;]\n\nakka {\nloggers = [&quot;akka.event.slf4j.Slf4jLogger&quot;]\nloglevel = &quot;INFO&quot;\n}\n\n\nbasicAuthentication.enabled=false\nbasicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}\nbasicAuthentication.username=&quot;admin&quot;\nbasicAuthentication.username=${?KAFKA_MANAGER_USERNAME}\nbasicAuthentication.password=&quot;password&quot;\nbasicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}\nbasicAuthentication.realm=&quot;Kafka-Manager&quot;\nbasicAuthentication.excluded=[&quot;\/api\/health&quot;] # ping the health of your instance without authentification\n\nkafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}\n\n<\/code><\/pre>\n<ul>\n<li>\n<p>Create docker-compose file:<\/p>\n<\/li>\n<ul>\n<li>\n<p>docker-compose-kafka.yml<\/p>\n<\/li>\n<\/ul>\n<\/ul>\n<pre><code>version: &apos;3.1&apos;\n\nservices:\n&nbsp;&nbsp;kafka1:\n&nbsp;&nbsp;&nbsp;&nbsp;image: confluentinc\/cp-kafka:5.1.2-1.0\n&nbsp;&nbsp;&nbsp;&nbsp;container_name: ka\n&nbsp;&nbsp;&nbsp;&nbsp;restart: always\n&nbsp;&nbsp;&nbsp;&nbsp;ports:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 9092:9092\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 9999:9999\n&nbsp;&nbsp;&nbsp;&nbsp;volumes:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- .\/ka-data:\/var\/lib\/kafka\/data\n&nbsp;&nbsp;&nbsp;&nbsp;environment:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_BROKER_ID: ${ID}\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_LISTENERS: PLAINTEXT:\/\/0.0.0.0:9092\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_ADVERTISED_LISTENERS: PLAINTEXT:\/\/${SERVER_KAFKA}:9092\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_ZOOKEEPER_CONNECT: ${SERVER_ZOO_CONNECT}\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_JMX_PORT: 9999\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_JMX_HOSTNAME: ${SERVER_KAFKA}\n<\/code><\/pre>\n<ul>\n<ul>\n<li>\n<p>docker-compose-zookeeper.yml<\/p>\n<\/li>\n<\/ul>\n<\/ul>\n<pre><code>version: &apos;3.1&apos;\n\nservices:\n&nbsp;&nbsp;zookeeper-1:\n&nbsp;&nbsp;&nbsp;&nbsp;image: zookeeper:3.4.13-1.0\n&nbsp;&nbsp;&nbsp;&nbsp;container_name: zk\n&nbsp;&nbsp;&nbsp;&nbsp;restart: always\n&nbsp;&nbsp;&nbsp;&nbsp;ports:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 2181:2181\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 2888:2888\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 3888:3888\n&nbsp;&nbsp;&nbsp;&nbsp;volumes:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- .\/zoo-data:\/data\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- .\/zoo-datalog:\/datalog\n&nbsp;&nbsp;&nbsp;&nbsp;environment:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ZOO_MY_ID: ${ID}\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ZOO_SERVERS: ${SERVER_ZOO_HOST}\n<\/code><\/pre>\n<ul>\n<ul>\n<li>\n<p>docker-compose-kafka-manager.yml<\/p>\n<\/li>\n<\/ul>\n<\/ul>\n<pre><code>version: &apos;3.1&apos;\n\nservices:\n&nbsp;&nbsp;kafka-manager-1:\n&nbsp;&nbsp;&nbsp;&nbsp;image: sheepkiller\/kafka-manager:1.3.1.8-1.0\n&nbsp;&nbsp;&nbsp;&nbsp;container_name: km\n&nbsp;&nbsp;&nbsp;&nbsp;restart: always\n&nbsp;&nbsp;&nbsp;&nbsp;ports:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 80:9000\n&nbsp;&nbsp;&nbsp;&nbsp;volumes:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- .\/km-conf:\/opt\n&nbsp;&nbsp;&nbsp;&nbsp;environment:\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ZK_HOSTS: ${SERVER_ZOO_CONNECT}\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KM_CONFIGFILE: \/opt\/application.conf\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_MANAGER_AUTH_ENABLED: &quot;true&quot;\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_MANAGER_USERNAME: user\n&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;KAFKA_MANAGER_PASSWORD: xxxxx\n<\/code><\/pre>\n<ul>\n<li>\n<p>And now you could start it now:&nbsp;<\/p>\n<p>process should be Zookeeper -&gt; Kafka-manager -&gt; create a zookeeper cluster by kafka-manager -&gt; Kafka<\/p>\n<\/li>\n<\/ul>\n<pre><code>docker-compose -f docker-compose-zookeeper up -d\ndocker-compose -f docker-compose-kafka-manager up -d\n# Now create a Zookeeper cluster by Kafka-manager\ndocker-compose -f docker-compose-kafka up -d\n<\/code><\/pre><\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>Create folder:<\/p>\n<\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[15],"class_list":["post-1276","post","type-post","status-publish","format-standard","hentry","category-tech","tag-kafka"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/john.pentaidea.com\/?p=1276\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog\" \/>\n<meta property=\"og:description\" content=\"Create folder:\" \/>\n<meta property=\"og:url\" content=\"https:\/\/john.pentaidea.com\/?p=1276\" \/>\n<meta property=\"og:site_name\" content=\"John&#039;s Blog\" \/>\n<meta property=\"article:published_time\" content=\"2019-03-28T00:11:21+00:00\" \/>\n<meta name=\"author\" content=\"jj\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"jj\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/john.pentaidea.com\/?p=1276\",\"url\":\"https:\/\/john.pentaidea.com\/?p=1276\",\"name\":\"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog\",\"isPartOf\":{\"@id\":\"https:\/\/john.pentaidea.com\/#website\"},\"datePublished\":\"2019-03-28T00:11:21+00:00\",\"dateModified\":\"2019-03-28T00:11:21+00:00\",\"author\":{\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f\"},\"breadcrumb\":{\"@id\":\"https:\/\/john.pentaidea.com\/?p=1276#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/john.pentaidea.com\/?p=1276\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/john.pentaidea.com\/?p=1276#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/john.pentaidea.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/john.pentaidea.com\/#website\",\"url\":\"https:\/\/john.pentaidea.com\/\",\"name\":\"John's Blog\",\"description\":\"\u4e16\u754c\u56e0\u601d\u7dd2\u800c\u8907\u96dc, \u4eba\u56e0\u5922\u60f3\u800c\u5049\u5927\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/john.pentaidea.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f\",\"name\":\"jj\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g\",\"caption\":\"jj\"},\"sameAs\":[\"https:\/\/john.pentaidea.com\"],\"url\":\"https:\/\/john.pentaidea.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/john.pentaidea.com\/?p=1276","og_locale":"en_US","og_type":"article","og_title":"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog","og_description":"Create folder:","og_url":"https:\/\/john.pentaidea.com\/?p=1276","og_site_name":"John&#039;s Blog","article_published_time":"2019-03-28T00:11:21+00:00","author":"jj","twitter_card":"summary_large_image","twitter_misc":{"Written by":"jj","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/john.pentaidea.com\/?p=1276","url":"https:\/\/john.pentaidea.com\/?p=1276","name":"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose - John&#039;s Blog","isPartOf":{"@id":"https:\/\/john.pentaidea.com\/#website"},"datePublished":"2019-03-28T00:11:21+00:00","dateModified":"2019-03-28T00:11:21+00:00","author":{"@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f"},"breadcrumb":{"@id":"https:\/\/john.pentaidea.com\/?p=1276#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/john.pentaidea.com\/?p=1276"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/john.pentaidea.com\/?p=1276#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/john.pentaidea.com\/"},{"@type":"ListItem","position":2,"name":"Build 3 nodes 1 Cluster Kafka, Zookeeper, Kafka-manager envirnment by docker-compose"}]},{"@type":"WebSite","@id":"https:\/\/john.pentaidea.com\/#website","url":"https:\/\/john.pentaidea.com\/","name":"John's Blog","description":"\u4e16\u754c\u56e0\u601d\u7dd2\u800c\u8907\u96dc, \u4eba\u56e0\u5922\u60f3\u800c\u5049\u5927","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/john.pentaidea.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f","name":"jj","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g","caption":"jj"},"sameAs":["https:\/\/john.pentaidea.com"],"url":"https:\/\/john.pentaidea.com\/?author=1"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts\/1276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1276"}],"version-history":[{"count":0,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts\/1276\/revisions"}],"wp:attachment":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}