{"id":1619,"date":"2021-11-10T02:56:52","date_gmt":"2021-11-09T18:56:52","guid":{"rendered":"https:\/\/jrblog.pentaidea.com\/john\/2021\/11\/10\/fluentbit-output-to-azure-event-hub-using-kafka\/"},"modified":"2021-11-10T02:56:52","modified_gmt":"2021-11-09T18:56:52","slug":"fluentbit-output-to-azure-event-hub-using-kafka","status":"publish","type":"post","link":"https:\/\/john.pentaidea.com\/?p=1619","title":{"rendered":"FluentBit output to Azure Event Hub using kafka"},"content":{"rendered":"<p>Worked for me using fluentbit kafka output to Azure EventHub<\/p>\n<p>_td-agent-bit.conf_<\/p>\n<p>&#8220;`<\/p>\n<p>[INPUT]<\/p>\n<p>\u00a0 Name\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 tail<\/p>\n<p>\u00a0 Path\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 xxx.log<\/p>\n<p>\u00a0 Refresh_Interval\u00a0 \u00a0 10<\/p>\n<p>[OUTPUT]<\/p>\n<p>\u00a0 Name\u00a0 \u00a0 \u00a0 \u00a0 kafka<\/p>\n<p>\u00a0 Match\u00a0 \u00a0 \u00a0  *<\/p>\n<p>\u00a0 brokers\u00a0 \u00a0  xxx.xxx.windows.net:9093<\/p>\n<p>\u00a0 topics\u00a0 \u00a0 \u00a0 xxx<\/p>\n<p>\u00a0 rdkafka.security.protocol\u00a0  SASL_SSL<\/p>\n<p>\u00a0 rdkafka.sasl.username\u00a0 \u00a0 \u00a0  $ConnectionString<\/p>\n<p>\u00a0 rdkafka.sasl.password\u00a0 \u00a0 \u00a0  Endpoint=sb:\/\/xxx.xxx.windows.net\/;SharedAccessKeyName=xxx;SharedAccessKey=xxx<\/p>\n<p>\u00a0 rdkafka.sasl.mechanism\u00a0 \u00a0 \u00a0 PLAIN<\/p>\n<p>[OUTPUT]<\/p>\n<p>\u00a0 name\u00a0 stdout<\/p>\n<p>\u00a0 match *<\/p>\n<p>&#8220;`<\/p>\n<p>Inside docker container (MUST HAVE or broker down\/ssl fail)<\/p>\n<p>_docker-compose.yml_<\/p>\n<p>&#8220;`<\/p>\n<p>version: &#8220;3.7&#8221;<\/p>\n<p>services:<\/p>\n<p>\u00a0 fluent-bit:<\/p>\n<p>\u00a0 \u00a0 image: fluent\/fluent-bit:1.6.2<\/p>\n<p>\u00a0 \u00a0 container_name: fluentbit<\/p>\n<p>\u00a0 \u00a0 restart: always<\/p>\n<p>\u00a0 \u00a0 volumes:<\/p>\n<p>\u00a0 \u00a0 \u00a0 &#8211; .\/td-agent-bit.conf:\/fluent-bit\/etc\/fluent-bit.conf<\/p>\n<p>\u00a0 \u00a0 \u00a0 &#8211; .\/xxx.log:\/fluent-bit\/etc\/xxx.log:ro<\/p>\n<p>&#8220;`<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"mh-excerpt\"><p>Worked for me using fluentbit kafka output to Azure EventHub<\/p>\n<\/div>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[4],"tags":[],"class_list":["post-1619","post","type-post","status-publish","format-standard","hentry","category-tech"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>FluentBit output to Azure Event Hub using kafka - John&#039;s Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/john.pentaidea.com\/?p=1619\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"FluentBit output to Azure Event Hub using kafka - John&#039;s Blog\" \/>\n<meta property=\"og:description\" content=\"Worked for me using fluentbit kafka output to Azure EventHub\" \/>\n<meta property=\"og:url\" content=\"https:\/\/john.pentaidea.com\/?p=1619\" \/>\n<meta property=\"og:site_name\" content=\"John&#039;s Blog\" \/>\n<meta property=\"article:published_time\" content=\"2021-11-09T18:56:52+00:00\" \/>\n<meta name=\"author\" content=\"jj\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"jj\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/john.pentaidea.com\/?p=1619\",\"url\":\"https:\/\/john.pentaidea.com\/?p=1619\",\"name\":\"FluentBit output to Azure Event Hub using kafka - John&#039;s Blog\",\"isPartOf\":{\"@id\":\"https:\/\/john.pentaidea.com\/#website\"},\"datePublished\":\"2021-11-09T18:56:52+00:00\",\"dateModified\":\"2021-11-09T18:56:52+00:00\",\"author\":{\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f\"},\"breadcrumb\":{\"@id\":\"https:\/\/john.pentaidea.com\/?p=1619#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/john.pentaidea.com\/?p=1619\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/john.pentaidea.com\/?p=1619#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/john.pentaidea.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"FluentBit output to Azure Event Hub using kafka\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/john.pentaidea.com\/#website\",\"url\":\"https:\/\/john.pentaidea.com\/\",\"name\":\"John's Blog\",\"description\":\"\u4e16\u754c\u56e0\u601d\u7dd2\u800c\u8907\u96dc, \u4eba\u56e0\u5922\u60f3\u800c\u5049\u5927\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/john.pentaidea.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f\",\"name\":\"jj\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/john.pentaidea.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g\",\"caption\":\"jj\"},\"sameAs\":[\"https:\/\/john.pentaidea.com\"],\"url\":\"https:\/\/john.pentaidea.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"FluentBit output to Azure Event Hub using kafka - John&#039;s Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/john.pentaidea.com\/?p=1619","og_locale":"en_US","og_type":"article","og_title":"FluentBit output to Azure Event Hub using kafka - John&#039;s Blog","og_description":"Worked for me using fluentbit kafka output to Azure EventHub","og_url":"https:\/\/john.pentaidea.com\/?p=1619","og_site_name":"John&#039;s Blog","article_published_time":"2021-11-09T18:56:52+00:00","author":"jj","twitter_card":"summary_large_image","twitter_misc":{"Written by":"jj","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/john.pentaidea.com\/?p=1619","url":"https:\/\/john.pentaidea.com\/?p=1619","name":"FluentBit output to Azure Event Hub using kafka - John&#039;s Blog","isPartOf":{"@id":"https:\/\/john.pentaidea.com\/#website"},"datePublished":"2021-11-09T18:56:52+00:00","dateModified":"2021-11-09T18:56:52+00:00","author":{"@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f"},"breadcrumb":{"@id":"https:\/\/john.pentaidea.com\/?p=1619#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/john.pentaidea.com\/?p=1619"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/john.pentaidea.com\/?p=1619#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/john.pentaidea.com\/"},{"@type":"ListItem","position":2,"name":"FluentBit output to Azure Event Hub using kafka"}]},{"@type":"WebSite","@id":"https:\/\/john.pentaidea.com\/#website","url":"https:\/\/john.pentaidea.com\/","name":"John's Blog","description":"\u4e16\u754c\u56e0\u601d\u7dd2\u800c\u8907\u96dc, \u4eba\u56e0\u5922\u60f3\u800c\u5049\u5927","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/john.pentaidea.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/3d2e5980a7d3023e93b91d668d2a4a4f","name":"jj","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/john.pentaidea.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/a685414cd92056415bc823b5cfbbf95a44a98df88d8c040b18613724c1c0724b?s=96&d=mm&r=g","caption":"jj"},"sameAs":["https:\/\/john.pentaidea.com"],"url":"https:\/\/john.pentaidea.com\/?author=1"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts\/1619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1619"}],"version-history":[{"count":0,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=\/wp\/v2\/posts\/1619\/revisions"}],"wp:attachment":[{"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/john.pentaidea.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}