What is Message Broker? Message broker is application with the goal to distribute a data to other services. When we use Message Broker we need to change the paradigm of developing the application from synchronous to asyncronous, it mean we send the data to some service and we forget about it. One of the use case of for this is, when you need to sending email and notification.
The reason to use kafka?
- You need scalability.
- Kafka can handle 3 Million writes per second in 3 machine. By default kafka need to be deployed at minimum 3 machines.
- Kafka can be easily to scaled horizontally, we can just join the new node kafka to the cluster we already have.
- Kafka maximum data is 1MB. Don't send too big message to kafka.
- Add new partition when you want to scale the consumer, but when you need to carefull adding new partition is not easy and it take times. The partion in kafka only can be consumed at maximum the max partition. So if the partition is 3 it can not consumed by 4 consumer.
- Squential guarated, but when it stuck at some point of the data the other data can not be consumed until the current issue is resolved. It also just work for on the same partition.
- Persistent, Kafka store all the data in a log file. So we don't have a lose data when it already consumed. So we can re read the log at the custom offset.
- Retention, expiration of the data can be configured by 2 ways, first by the time and the second by file size.
- Rebalance, when we scale the consumer, kafka can rebalance the traffic from each partition to the new consumer.
- Offset, offest will be attached to the consumer, so how we can know the which offest that have been consumed? When consumer already consumed the message they will send commit message event to the partition. There are several way to commit the message, we can commit by range time for example after 10 second, or by each message consumed we send the commit for it.
- Consumer group, we can consume the message by group of consumer and then the commit message will be set by the consumer group.
Kafka and RabbitMQ is the same platform, it was for doing message broker. Most of the time Kafka is the most prefered platform for doing Message broker but you need to consider.
- When you install kafka you need at minimum 3 server to deploy one cluster of Kafka.
- Kafka can handle high throughput, 2 million write in a second.
- Available client library for Kafka
- Can be deployed just with one server, but better at minimum 2 server.
- The throughput is a half of the Kafka(605MB/s) and RabbitMQ(38MB/s).
- Available RabbitMQ client library.
What is service discovery?
Service discovery is a process that enables dynamic configuration of applications and services. It allows applications and services to be automatically discovered and configured without manual intervention. Service discovery enables services to be located, identified, and authenticated in distributed systems, allowing for automated configuration of applications and services. Service discovery also simplifies the process of managing the availability and scalability of services in a distributed environment.
What is service registry?
Service registry is a type of registry that stores information about services running on a network. It is used to discover and connect to these services, as well as provide information about the services such as their availability and capabilities. Service registries are typically used in distributed systems and microservices architectures.