Notes Kafka

Voir aussi :

• Notes - queue - messages - middleware - file d'attente de messages
• Redpanda (Implémentation en C++)
• NATS, RabbitMQ, NSQ

Source :

Voir :

• https://kafka.apache.org/quickstart
• https://www.tutorialkart.com/apache-kafka/install-apache-kafka-on-ubuntu/
• https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm
• https://www.educba.com/install-kafka/

Install Java

apt-get install default-jdk-headless

Télécharger Kafka https://kafka.apache.org/downloads

Apache Kafka dépend de Zookeeper pour la gestion des clusters. Par conséquent, avant de lancer Kafka, Zookeeper doit être démarré. Il n'est pas nécessaire d'installer explicitement Zookeeper, car il est fourni avec Apache Kafka.

Lancer Zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Lancer Kafka Server

bin/kafka-server-start.sh config/server.properties

Kafka/config/server.properties

delete.topic.enable = true

acks=all
min.insync.replica=2

Note en vrac

EnableSslCertificateVerification = false
security.protocol=SASL_SSL

kafkacat -b localhost:9092 -L
kafkacat -b localhost:9092 -L -d broker
 
kafka-topics --list --zookeeper zkinfo
kafka-topics --zookeeper zk01.example.com:2181 --list
 
 
# kafka-consumer-groups --new-consumer --bootstrap-server broker01.example.com:9092 --describe --group flume
export KAFKA_HEAP_OPTS=-Xmx1024M
kafka-consumer-groups --bootstrap-server host:9092 --list
kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo
 
kafka-consumer-groups --bootstrap-server host.server.com:9093 --describe --command-config client.properties

Dockerfile

FROM frolvlad/alpine-python2
COPY root/requirements.txt /
COPY root/opt /opt
 
#RUN apk add librdkafka
#RUN apk add gcc python2-dev libc-dev librdkafka-dev
 
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r /requirements.txt
WORKDIR /opt/app/
#CMD ["gunicorn", "-w 4", "main:app"]
CMD ["python", "/opt/app/run.py"]

requirements.txt

#librdkafka==0.11.6
simplejson
requests
six
#avro_json_serializer==0.5
avro_json_serializer
#avro==1.8.2
confluent_kafka

# Pour librdkafka ?
yum install gcc python-devel
 
# Pour client Oracle
yum install libaio
 
# Kafka-cli
yum install java-latest-openjdk-headless
curl -O http://miroir.univ-lorraine.fr/apache/kafka/2.4.0/kafka_2.13-2.4.0.tgz

Docker CentOS

Dockerfile

#FROM centos:centos6.9
FROM centos:centos7
 
COPY root/requirements.txt /
COPY root/opt/app /opt/app
 
RUN yum install -y epel-release.noarch
RUN yum install -y python-pip
 
# Pour CentOS6
#RUN curl https://bootstrap.pypa.io/2.6/get-pip.py -o get-pip.py
#RUN python get-pip.py
#RUN pip install --upgrade --ignore-installed pip setuptools
 
#RUN apk add librdkafka
#RUN apk add gcc python2-dev libc-dev librdkafka-dev
 
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r /requirements.txt
WORKDIR /opt/app
#CMD ["gunicorn", "-w 4", "main:app"]
CMD ["python", "/opt/app/run.py"]