kafka is an open-source information system, which is able to provide uniform, high volumn and low latency data. kafka classify messages according to topic and store them afterwards.
Features:
- Sequential I/O
- Memory Mapped Files
- Zero Copy based on sendfile
- Batch compression user cases:
- IOT
- microservice logs
components
- producer
producer message and send it to topic - consumer
consume message from certain topic - topic cluster of messages with same type
- broker
kafka instance (kafka server node), every broker has multi-topic - zookeeper zookeeper store meta data for kafka
process
producer produce messages and send them to broker, leader broker receive messages and write them to corresponding topic, after receiving the leader broker send follow broker a copy, consumer consume messages in the broker
comparasion with rabbmit
- IOT:
- rabbmit: data loss is unlikely to happen than kafka
- kafka: data loss could happen with high workload
- distributed or microservice logs
- kafka is fit for async collection of full link log
