adding self balancing properties

Author: cnb0

WS/00.kfk.confluent.docker.setup/full-stack/zoo1/data/myid

@@ -0,0 +1 @@
+1

WS/00.kfk.confluent.docker.setup/zk-single-kafka-single.yml

@@ -1,7 +1,7 @@
 # Replace "kafka-topics" 
 # by "kafka-topics.sh" or "kafka-topics.bat" based on your system # (or bin/kafka-topics.sh or bin\windows\kafka-topics.bat if you didn't setup PATH / Environment variables)
 
-kafka-topics.sh
+/home/nobleprog/kafka_2.12-2.4.0/bin/kafka-topics.sh
 
 kafka-topics --zookeeper 127.0.0.1:2181 --list 
 

WS/00.kfk.gettingstarted/1-kafka-cli/0-kafka-topics.sh

@@ -0,0 +1,86 @@
+##
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+##
+
+# This file contains some of the configurations for the Kafka Connect distributed worker. This file is intended
+# to be used with the examples, and some settings may differ from those used in a production system, especially
+# the `bootstrap.servers` and those specifying replication factors.
+
+# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
+bootstrap.servers=localhost:9092
+
+# unique name for the cluster, used in forming the Connect cluster group. Note that this must not conflict with consumer group IDs
+group.id=connect-cluster
+
+# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
+# need to configure these based on the format they want their data in when loaded from or stored into Kafka
+key.converter=org.apache.kafka.connect.json.JsonConverter
+value.converter=org.apache.kafka.connect.json.JsonConverter
+# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
+# it to
+key.converter.schemas.enable=true
+value.converter.schemas.enable=true
+
+# Topic to use for storing offsets. This topic should have many partitions and be replicated and compacted.
+# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
+# the topic before starting Kafka Connect if a specific topic configuration is needed.
+# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
+# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
+# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
+offset.storage.topic=connect-offsets
+offset.storage.replication.factor=2
+#offset.storage.partitions=25
+
+# Topic to use for storing connector and task configurations; note that this should be a single partition, highly replicated,
+# and compacted topic. Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
+# the topic before starting Kafka Connect if a specific topic configuration is needed.
+# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
+# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
+# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
+config.storage.topic=connect-configs
+config.storage.replication.factor=2
+
+# Topic to use for storing statuses. This topic can have multiple partitions and should be replicated and compacted.
+# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
+# the topic before starting Kafka Connect if a specific topic configuration is needed.
+# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
+# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
+# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
+status.storage.topic=connect-status
+status.storage.replication.factor=2
+#status.storage.partitions=5
+
+# Flush much faster than normal, which is useful for testing/debugging
+offset.flush.interval.ms=10000
+
+# These are provided to inform the user about the presence of the REST host and port configs 
+# Hostname & Port for the REST API to listen on. If this is set, it will bind to the interface used to listen to requests.
+#rest.host.name=
+#rest.port=8083
+
+# The Hostname & Port that will be given out to other workers to connect to i.e. URLs that are routable from other servers.
+#rest.advertised.host.name=
+#rest.advertised.port=
+
+# Set to a list of filesystem paths separated by commas (,) to enable class loading isolation for plugins
+# (connectors, converters, transformations). The list should consist of top level directories that include 
+# any combination of: 
+# a) directories immediately containing jars with plugins and their dependencies
+# b) uber-jars with plugins and their dependencies
+# c) directories immediately containing the package directory structure of classes of plugins and their dependencies
+# Examples: 
+# plugin.path=/usr/local/share/java,/usr/local/share/kafka/plugins,/opt/connectors,
+plugin.path=/usr/share/java,/home/nobleprog/confluent-6.0.0/share/confluent-hub-components

kfk.confluent/01.confluent-control-center/selfbalancing-properties/connect-distributed.properties

@@ -0,0 +1,133 @@
+# (Copyright) Confluent, Inc.
+
+############################# Server Basics #############################
+
+# A comma separated list of Apache Kafka cluster host names (required)
+# NOTE: should not be localhost
+#bootstrap.servers=kafka1:9092
+
+# A comma separated list of ZooKeeper host names (for ACLs)
+#zookeeper.connect=zookeeper1:2181
+
+############################# Control Center Settings #############################
+
+# Unique identifier for the Control Center
+#confluent.controlcenter.id=1
+
+# Directory for Control Center to store data
+# NOTE: this should be changed to point to a reliable directory
+confluent.controlcenter.data.dir=/tmp/confluent/control-center
+
+# License string for the Control Center
+#confluent.license=XyZ
+
+# A comma separated list of Connect host names
+#confluent.controlcenter.connect.cluster=http://localhost:8083
+
+# KSQL cluster URL
+#confluent.controlcenter.ksql.ksqlDB.url=http://localhost:8088
+
+# Schema Registry cluster URL
+#confluent.controlcenter.schema.registry.url=http://localhost:8081
+
+# Kafka REST endpoint URL
+#confluent.controlcenter.streams.cprest.url=http://localhost:8090
+confluent.controlcenter.streams.cprest.url=http://localhost:8090,http://localhost:8091,http://localhost:8092,http://localhost:8093,http://localhost:8094
+
+# Settings to enable email alerts
+#confluent.controlcenter.mail.enabled=true
+#confluent.controlcenter.mail.host.name=smtp1
+#confluent.controlcenter.mail.port=587
+#[email protected]
+
+# Replication for internal Control Center topics.
+# Only lower them for testing.
+# WARNING: replication factor of 1 risks data loss.
+#confluent.controlcenter.internal.topics.replication=3
+
+# Number of partitions for Control Center internal topics
+# Increase for better throughput on monitored data (CPU bound)
+# NOTE: changing requires running `bin/control-center-reset` prior to restart
+#confluent.controlcenter.internal.topics.partitions=4
+
+# Topic used to store Control Center configuration
+# WARNING: replication factor of 1 risks data loss.
+#confluent.controlcenter.command.topic.replication=3
+
+# Enable automatic UI updates
+confluent.controlcenter.ui.autoupdate.enable=true
+
+# Enable usage data collection
+confluent.controlcenter.usage.data.collection.enable=true
+
+# Enable Controller Chart in Broker page
+#confluent.controlcenter.ui.controller.chart.enable=true
+
+############################# Control Center RBAC Settings #############################
+
+# Enable RBAC authorization in Control Center by providing a comma-separated list of Metadata Service (MDS) URLs
+#confluent.metadata.bootstrap.server.urls=http://localhost:8090
+
+# MDS credentials of an RBAC user for Control Center to act on behalf of
+# NOTE: This user must be a SystemAdmin on each Apache Kafka cluster
+#confluent.metadata.basic.auth.user.info=username:password
+
+# Enable SASL-based authentication for each Apache Kafka cluster (SASL_PLAINTEXT or SASL_SSL required)
+#confluent.controlcenter.streams.security.protocol=SASL_PLAINTEXT
+#confluent.controlcenter.kafka.<name>.security.protocol=SASL_PLAINTEXT
+
+# Enable authentication using a bearer token for Control Center's REST endpoints
+#confluent.controlcenter.rest.authentication.method=BEARER
+
+# Public key used to verify bearer tokens
+# NOTE: Must match the MDS public key
+#public.key.path=/path/to/publickey.pem
+
+############################# Broker (Metrics reporter) Monitoring #############################
+
+# Set how far back in time metrics reporter data should be processed
+#confluent.metrics.topic.skip.backlog.minutes=15
+
+############################# Stream (Interceptor) Monitoring #############################
+
+# Keep these settings default unless using non-Confluent interceptors
+
+# Override topic name for intercepted (should mach custom interceptor settings)
+#confluent.monitoring.interceptor.topic=_confluent-monitoring
+
+# Number of partitions for the intercepted topic
+#confluent.monitoring.interceptor.topic.partitions=12
+
+# Amount of replication for intercepted topics
+# WARNING: replication factor of 1 risks data loss.
+#confluent.monitoring.interceptor.topic.replication=3
+
+# Set how far back in time interceptor data should be processed
+#confluent.monitoring.interceptor.topic.skip.backlog.minutes=15
+
+############################# System Health (Broker) Monitoring #############################
+
+# Number of partitions for the metrics topic
+#confluent.metrics.topic.partitions=12
+
+# Replication factor for broker monitoring data
+# WARNING: replication factor of 1 risks data loss.
+#confluent.metrics.topic.replication=3
+
+############################# Streams (state store) settings #############################
+
+# Increase for better throughput on data processing (CPU bound)
+#confluent.controlcenter.streams.num.stream.threads=8
+
+################################## Confluent Telemetry Settings  ##################################
+
+# To start using Telemetry, first generate a Confluent Cloud API key/secret. This can be done with
+# instructions at https://docs.confluent.io/current/cloud/using/api-keys.html. Note that you should
+# be using the '--resource cloud' flag.
+#
+# After generating an API key/secret, to enable Telemetry uncomment the lines below and paste
+# in your API key/secret.
+#
+#confluent.telemetry.enabled=true
+#confluent.telemetry.api.key=<CLOUD_API_KEY>
+#confluent.telemetry.api.secret=<CCLOUD_API_SECRET>