Différences

Ci-dessous, les différences entre deux révisions de la page.

--- tech:notes_kafka [2025/03/24 15:06] – créée - modification externe 127.0.0.1
+++ tech:notes_kafka [2026/06/07 19:12] (Version actuelle) – modification externe 127.0.0.1
@@ Ligne 1: / Ligne 1: @@
+<!DOCTYPE markdown>
+{{tag>Brouillon}}
+# Notes Kafka
+Voir aussi :
+* [[Notes queue messages middleware file d attente de message|Notes - queue - messages - middleware - file d'attente de messages]]
+* Redpanda (Implémentation en C++)
+* NATS, RabbitMQ, NSQ
+## Serveur
+Source :
+Voir :
+* https://kafka.apache.org/quickstart
+* https://www.tutorialkart.com/apache-kafka/install-apache-kafka-on-ubuntu/
+* https://www.tutorialspoint.com/apache_kafka/apache_kafka_installation_steps.htm
+* https://www.educba.com/install-kafka/
+Install Java
+~~~bash
+apt-get install default-jdk-headless
+~~~
+Télécharger Kafka https://kafka.apache.org/downloads
+Apache Kafka dépend de Zookeeper pour la gestion des clusters. Par conséquent, avant de lancer Kafka, Zookeeper doit être démarré. Il n'est pas nécessaire d'installer explicitement Zookeeper, car il est fourni avec Apache Kafka.
+Lancer Zookeeper
+~~~bash
+bin/zookeeper-server-start.sh config/zookeeper.properties
+~~~
+Lancer Kafka Server
+~~~bash
+bin/kafka-server-start.sh config/server.properties
+~~~
+### Autres
+`Kafka/config/server.properties`
+~~~ini
+delete.topic.enable = true
+~~~
+~~~bash
+acks=all
+min.insync.replica=2
+~~~
+## Client
+Note en vrac
+~~~ini
+EnableSslCertificateVerification = false
+security.protocol=SASL_SSL
+~~~
+~~~bash
+kafkacat -b localhost:9092 -L
+kafkacat -b localhost:9092 -L -d broker
+kafka-topics --list --zookeeper zkinfo
+kafka-topics --zookeeper zk01.example.com:2181 --list
+# kafka-consumer-groups --new-consumer --bootstrap-server broker01.example.com:9092 --describe --group flume
+export KAFKA_HEAP_OPTS=-Xmx1024M
+kafka-consumer-groups --bootstrap-server host:9092 --list
+kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo
+kafka-consumer-groups --bootstrap-server host.server.com:9093 --describe --command-config client.properties
+~~~
+`Dockerfile`
+~~~bash
+FROM frolvlad/alpine-python2
+COPY root/requirements.txt /
+COPY root/opt /opt
+#RUN apk add librdkafka
+#RUN apk add gcc python2-dev libc-dev librdkafka-dev
+RUN pip install --upgrade pip
+RUN pip install --no-cache-dir -r /requirements.txt
+WORKDIR /opt/app/
+#CMD ["gunicorn", "-w 4", "main:app"]
+CMD ["python", "/opt/app/run.py"]
+~~~
+`requirements.txt`
+~~~python
+#librdkafka==0.11.6
+simplejson
+requests
+six
+#avro_json_serializer==0.5
+avro_json_serializer
+#avro==1.8.2
+confluent_kafka
+~~~
+### RedHat CentOS
+~~~bash
+# Pour librdkafka ?
+yum install gcc python-devel
+# Pour client Oracle
+yum install libaio
+# Kafka-cli
+yum install java-latest-openjdk-headless
+curl -O http://miroir.univ-lorraine.fr/apache/kafka/2.4.0/kafka_2.13-2.4.0.tgz
+~~~
+#### Docker
+Docker CentOS
+`Dockerfile`
+~~~bash
+#FROM centos:centos6.9
+FROM centos:centos7
+COPY root/requirements.txt /
+COPY root/opt/app /opt/app
+RUN yum install -y epel-release.noarch
+RUN yum install -y python-pip
+# Pour CentOS6
+#RUN curl https://bootstrap.pypa.io/2.6/get-pip.py -o get-pip.py
+#RUN python get-pip.py
+#RUN pip install --upgrade --ignore-installed pip setuptools
+#RUN apk add librdkafka
+#RUN apk add gcc python2-dev libc-dev librdkafka-dev
+RUN pip install --upgrade pip
+RUN pip install --no-cache-dir -r /requirements.txt
+WORKDIR /opt/app
+#CMD ["gunicorn", "-w 4", "main:app"]
+CMD ["python", "/opt/app/run.py"]
+~~~
+FIXME