Up next in 10
Enhancing Microservices with Kafka: A Distributed Messaging Solution
0 views
Aug 6, 2025
An in-depth session on designing and implementing Kafka for an enterprise microservice-based application that spans multiple pods and containers with over 100 JVMs. In this session, we will explore how to effectively bridge various pods and JVMs while considering key factors such as scalability, resilience, high availability, and performance for intensive applications. 🔗 Conference Website: https://softwarearchitecture.live 📺 CSharp TV - Dev Streaming Destination http://csharp.tv 🌎 C# Corner - Community of Software and Data Developers https://www.c-sharpcorner.com #CSharpTV #CSharpCorner #CSharp #SoftwareArchitectureConf
View Video Transcript
0:03
so hi everyone good morning good
0:06
so hi everyone good morning good
0:06
so hi everyone good morning good afternoon good evening um just a bit
0:09
afternoon good evening um just a bit
0:09
afternoon good evening um just a bit about me Simon already spoke about this
0:11
about me Simon already spoke about this
0:11
about me Simon already spoke about this so without wasting time let me quickly
0:14
so without wasting time let me quickly
0:14
so without wasting time let me quickly jump start um I'm going to cover the
0:16
jump start um I'm going to cover the
0:16
jump start um I'm going to cover the basics first uh considering the audience
0:18
basics first uh considering the audience
0:18
basics first uh considering the audience from the various background before we
0:20
from the various background before we
0:20
from the various background before we tape di into the session um so what is
0:24
tape di into the session um so what is
0:24
tape di into the session um so what is microservices before we talk about
0:26
microservices before we talk about
0:26
microservices before we talk about microservices we have to understand what
0:28
microservices we have to understand what
0:28
microservices we have to understand what is the current Challenge and what are
0:30
is the current Challenge and what are
0:30
is the current Challenge and what are the monolithic
0:31
the monolithic
0:31
the monolithic applications so for for more than couple
0:34
applications so for for more than couple
0:34
applications so for for more than couple of tickets the standard way of
0:35
of tickets the standard way of
0:35
of tickets the standard way of developing application is using
0:37
developing application is using
0:37
developing application is using monolithic architecture that is all the
0:40
monolithic architecture that is all the
0:40
monolithic architecture that is all the components of application are in a
0:41
components of application are in a
0:41
components of application are in a single code base and the scaling and
0:44
single code base and the scaling and
0:44
single code base and the scaling and deployment of the application done as
0:46
deployment of the application done as
0:46
deployment of the application done as one
0:47
one
0:47
one unit this means the application must be
0:49
unit this means the application must be
0:49
unit this means the application must be written in a single language with
0:51
written in a single language with
0:51
written in a single language with completely one text stack and entire
0:54
completely one text stack and entire
0:54
completely one text stack and entire Global team working on different parts
0:56
Global team working on different parts
0:56
Global team working on different parts of the application still then have to
0:58
of the application still then have to
0:58
of the application still then have to coordinate better okay this way of
1:01
coordinate better okay this way of
1:01
coordinate better okay this way of standard development of applications and
1:05
standard development of applications and
1:05
standard development of applications and is complex and especially it's even
1:07
is complex and especially it's even
1:07
is complex and especially it's even becomes massive when the application has
1:09
becomes massive when the application has
1:09
becomes massive when the application has to be you know handled across Global uh
1:13
to be you know handled across Global uh
1:13
to be you know handled across Global uh teams the first of all this the first of
1:16
teams the first of all this the first of
1:16
teams the first of all this the first of all the main challenge is the
1:17
all the main challenge is the
1:17
all the main challenge is the coordination among team members is
1:19
coordination among team members is
1:19
coordination among team members is difficult as the code base are more
1:21
difficult as the code base are more
1:21
difficult as the code base are more Tangled okay and if we need to scale one
1:24
Tangled okay and if we need to scale one
1:24
Tangled okay and if we need to scale one particular functionality of the
1:25
particular functionality of the
1:25
particular functionality of the application we cannot do because we need
1:26
application we cannot do because we need
1:26
application we cannot do because we need to develop the entire stack of the
1:28
to develop the entire stack of the
1:28
to develop the entire stack of the application to get together and we have
1:31
application to get together and we have
1:31
application to get together and we have to scale it up all together so this
1:34
to scale it up all together so this
1:34
to scale it up all together so this means we will have higher infrastructure
1:36
means we will have higher infrastructure
1:36
means we will have higher infrastructure cost more challenges and the entire unit
1:39
cost more challenges and the entire unit
1:39
cost more challenges and the entire unit has to be compatible
1:42
has to be compatible
1:42
has to be compatible and everyone have to be on their do to
1:45
and everyone have to be on their do to
1:45
and everyone have to be on their do to support the application and the other
1:47
support the application and the other
1:47
support the application and the other challenges if we are using a third party
1:50
challenges if we are using a third party
1:50
challenges if we are using a third party CH and say if one functionality needs
1:53
CH and say if one functionality needs
1:54
CH and say if one functionality needs let's say version 3.6 and another
1:56
let's say version 3.6 and another
1:56
let's say version 3.6 and another functionality says Needs version 1.8
1:59
functionality says Needs version 1.8
1:59
functionality says Needs version 1.8 then we cannot have two versions We need
2:01
then we cannot have two versions We need
2:01
then we cannot have two versions We need to either use one way or scale up our
2:04
to either use one way or scale up our
2:04
to either use one way or scale up our application and fix the book or make it
2:07
application and fix the book or make it
2:07
application and fix the book or make it scalable to support other version couple
2:10
scalable to support other version couple
2:10
scalable to support other version couple of years back we had a log 4G
2:12
of years back we had a log 4G
2:12
of years back we had a log 4G vulnerability and if you remember there
2:15
vulnerability and if you remember there
2:15
vulnerability and if you remember there was version 2.14 um which was the
2:19
was version 2.14 um which was the
2:19
was version 2.14 um which was the version which found vulnerable and they
2:21
version which found vulnerable and they
2:21
version which found vulnerable and they were supposed to upgrade to 2.17 if I
2:23
were supposed to upgrade to 2.17 if I
2:23
were supposed to upgrade to 2.17 if I remember it
2:25
remember it
2:25
remember it right so the entire application has to
2:27
right so the entire application has to
2:27
right so the entire application has to be scaled up if we are using a lock for
2:30
be scaled up if we are using a lock for
2:30
be scaled up if we are using a lock for and if there are any dependent Jaws
2:32
and if there are any dependent Jaws
2:32
and if there are any dependent Jaws which is making use of this particular
2:34
which is making use of this particular
2:34
which is making use of this particular version then we need to work around it
2:38
version then we need to work around it
2:38
version then we need to work around it either to handle those jav files
2:40
either to handle those jav files
2:40
either to handle those jav files separately or make it as an exclusion or
2:44
separately or make it as an exclusion or
2:44
separately or make it as an exclusion or come up with Innovative way and do a
2:46
come up with Innovative way and do a
2:46
come up with Innovative way and do a patch on the hack just to keep those
2:49
patch on the hack just to keep those
2:49
patch on the hack just to keep those component aside and these all increase
2:52
component aside and these all increase
2:53
component aside and these all increase our turnaround time and the complexity
2:55
our turnaround time and the complexity
2:55
our turnaround time and the complexity of the application
2:56
of the application
2:56
of the application stack and this is where all this leads
2:59
stack and this is where all this leads
2:59
stack and this is where all this leads to the the microservices architecture so
3:02
to the the microservices architecture so
3:02
to the the microservices architecture so here in microservices architecture so we
3:05
here in microservices architecture so we
3:05
here in microservices architecture so we are going to break down the application
3:07
are going to break down the application
3:07
are going to break down the application into multiple smaller application so
3:10
into multiple smaller application so
3:10
into multiple smaller application so this is like several micro applications
3:13
this is like several micro applications
3:13
this is like several micro applications making up the big one
3:14
making up the big one
3:14
making up the big one application the breakdown of the
3:16
application the breakdown of the
3:16
application the breakdown of the application into microservices is purely
3:18
application into microservices is purely
3:18
application into microservices is purely dependent upon the business
3:19
dependent upon the business
3:19
dependent upon the business functionality and not by technical
3:22
functionality and not by technical
3:22
functionality and not by technical component so micros service of an
3:25
component so micros service of an
3:25
component so micros service of an e-commerce or a shopping website just
3:27
e-commerce or a shopping website just
3:27
e-commerce or a shopping website just for an example let's say a product
3:30
for an example let's say a product
3:30
for an example let's say a product shopping cart or a user information
3:32
shopping cart or a user information
3:32
shopping cart or a user information check out and payment will all be a
3:34
check out and payment will all be a
3:34
check out and payment will all be a separate microservice components so in
3:37
separate microservice components so in
3:37
separate microservice components so in terms of size each microservices can
3:39
terms of size each microservices can
3:39
terms of size each microservices can have an isolated functionality so one
3:42
have an isolated functionality so one
3:42
have an isolated functionality so one service will do one specific job and
3:46
service will do one specific job and
3:46
service will do one specific job and another important criteria while you
3:49
another important criteria while you
3:49
another important criteria while you know we are working on microservices is
3:51
know we are working on microservices is
3:51
know we are working on microservices is that it's Loosely coupled and
3:54
that it's Loosely coupled and
3:54
that it's Loosely coupled and independent so we can independently
3:56
independent so we can independently
3:57
independent so we can independently deploy and scale each functionality or
3:59
deploy and scale each functionality or
3:59
deploy and scale each functionality or the services and it gives a greater
4:02
the services and it gives a greater
4:02
the services and it gives a greater flexibility and they can have their own
4:04
flexibility and they can have their own
4:04
flexibility and they can have their own versions without depending upon the
4:05
versions without depending upon the
4:05
versions without depending upon the other
4:06
other
4:06
other component this makes it opening up a
4:10
component this makes it opening up a
4:10
component this makes it opening up a wide platform to support different uh
4:13
wide platform to support different uh
4:13
wide platform to support different uh programming languages so if I going to
4:16
programming languages so if I going to
4:16
programming languages so if I going to integrate in a machine learning
4:18
integrate in a machine learning
4:18
integrate in a machine learning component into my functional uh
4:20
component into my functional uh
4:20
component into my functional uh application I can have it in a python
4:23
application I can have it in a python
4:23
application I can have it in a python where rest of my application can still
4:24
where rest of my application can still
4:25
where rest of my application can still run in Java so this makes it completely
4:29
run in Java so this makes it completely
4:29
run in Java so this makes it completely agnostic to the underlying programming
4:31
agnostic to the underlying programming
4:31
agnostic to the underlying programming languages if one microservices goes down
4:34
languages if one microservices goes down
4:34
languages if one microservices goes down for whatever reason the application is
4:36
for whatever reason the application is
4:36
for whatever reason the application is still functionable and the rest of the
4:38
still functionable and the rest of the
4:38
still functionable and the rest of the components can support it say um if the
4:41
components can support it say um if the
4:41
components can support it say um if the payment services have some glitch still
4:44
payment services have some glitch still
4:44
payment services have some glitch still people can view their products act to
4:45
people can view their products act to
4:45
people can view their products act to the shopping cart and they can check out
4:47
the shopping cart and they can check out
4:47
the shopping cart and they can check out at a later point of
4:50
at a later point of
4:50
at a later point of time in addition when we design micro
4:54
time in addition when we design micro
4:54
time in addition when we design micro Services these are some of the basic
4:56
Services these are some of the basic
4:56
Services these are some of the basic characteristics and the designs of
4:58
characteristics and the designs of
4:58
characteristics and the designs of microservices which you know we have to
5:00
microservices which you know we have to
5:00
microservices which you know we have to consider it like the Isolation part and
5:03
consider it like the Isolation part and
5:04
consider it like the Isolation part and statelessness so to talk about
5:06
statelessness so to talk about
5:06
statelessness so to talk about statelessness the microservices it's
5:09
statelessness the microservices it's
5:09
statelessness the microservices it's good to be have it as an item put that
5:11
good to be have it as an item put that
5:11
good to be have it as an item put that is if a messages comes multiple times
5:14
is if a messages comes multiple times
5:14
is if a messages comes multiple times the functional and behavior of the
5:16
the functional and behavior of the
5:16
the functional and behavior of the microservices will be good to have mimic
5:19
microservices will be good to have mimic
5:19
microservices will be good to have mimic the same output so every time a same
5:22
the same output so every time a same
5:22
the same output so every time a same message replayed into a service it will
5:25
message replayed into a service it will
5:25
message replayed into a service it will yeld the same output and even driven
5:28
yeld the same output and even driven
5:28
yeld the same output and even driven makes the microservices more robust and
5:31
makes the microservices more robust and
5:31
makes the microservices more robust and functionable and lot of organizations
5:34
functionable and lot of organizations
5:34
functionable and lot of organizations including um you know from my past
5:36
including um you know from my past
5:36
including um you know from my past experience we are heavily on even
5:40
experience we are heavily on even
5:40
experience we are heavily on even driven and performance- wise This is
5:43
driven and performance- wise This is
5:43
driven and performance- wise This is highly scalable because we can um scale
5:47
highly scalable because we can um scale
5:47
highly scalable because we can um scale up the ports or in respect of whether
5:50
up the ports or in respect of whether
5:50
up the ports or in respect of whether you running on a quetes or any other
5:52
you running on a quetes or any other
5:52
you running on a quetes or any other platforms we can independently scale up
5:54
platforms we can independently scale up
5:54
platforms we can independently scale up them and you can deploy and decentralize
5:58
them and you can deploy and decentralize
5:58
them and you can deploy and decentralize the components so it giv gives a greater
6:04
flexibility so how do microservices
6:09
communicates that normal traditional way
6:12
communicates that normal traditional way
6:12
communicates that normal traditional way where lot of applications goes through
6:14
where lot of applications goes through
6:14
where lot of applications goes through the API calls that is if one they expose
6:18
the API calls that is if one they expose
6:18
the API calls that is if one they expose a htps port open so one Services can
6:22
a htps port open so one Services can
6:22
a htps port open so one Services can send a request and probably will wait
6:24
send a request and probably will wait
6:24
send a request and probably will wait forment from other service through an
6:26
forment from other service through an
6:26
forment from other service through an endpoint this is very common and this is
6:28
endpoint this is very common and this is
6:28
endpoint this is very common and this is an way that is each component has to
6:31
an way that is each component has to
6:31
an way that is each component has to wait for a request and a response and it
6:34
wait for a request and a response and it
6:34
wait for a request and a response and it needs to be
6:35
needs to be
6:35
needs to be processed next is the messaging broker
6:38
processed next is the messaging broker
6:38
processed next is the messaging broker which we will be going to Deep dive um
6:41
which we will be going to Deep dive um
6:41
which we will be going to Deep dive um after this one but just to give an high
6:43
after this one but just to give an high
6:43
after this one but just to give an high level servers one let's say a uh
6:47
level servers one let's say a uh
6:47
level servers one let's say a uh shopping cart publishes the list of
6:50
shopping cart publishes the list of
6:50
shopping cart publishes the list of uh products in the cart to a messaging
6:53
uh products in the cart to a messaging
6:53
uh products in the cart to a messaging broker and the consumers like like a
6:55
broker and the consumers like like a
6:55
broker and the consumers like like a dispatcher or something will consume
6:57
dispatcher or something will consume
6:57
dispatcher or something will consume them in the sequence or or in whatever
7:01
them in the sequence or or in whatever
7:01
them in the sequence or or in whatever order how we are defining it we'll
7:03
order how we are defining it we'll
7:03
order how we are defining it we'll process that order and we'll dispatch it
7:06
process that order and we'll dispatch it
7:06
process that order and we'll dispatch it so here the service one is not dependent
7:09
so here the service one is not dependent
7:09
so here the service one is not dependent on service 2 how it is being processing
7:12
on service 2 how it is being processing
7:12
on service 2 how it is being processing the uh messages so the role of the
7:14
the uh messages so the role of the
7:15
the uh messages so the role of the service one is just to publish the
7:16
service one is just to publish the
7:16
service one is just to publish the messages it's done and it will move on
7:19
messages it's done and it will move on
7:19
messages it's done and it will move on to its next functionality and service 2
7:23
to its next functionality and service 2
7:23
to its next functionality and service 2 if it is just about consuming it it
7:25
if it is just about consuming it it
7:25
if it is just about consuming it it keeps on consuming and processing
7:26
keeps on consuming and processing
7:26
keeps on consuming and processing whatever it's intend to do this way it
7:28
whatever it's intend to do this way it
7:29
whatever it's intend to do this way it is Asen synchronous and event driven and
7:31
is Asen synchronous and event driven and
7:31
is Asen synchronous and event driven and the third one is the service mesh where
7:35
the third one is the service mesh where
7:35
the third one is the service mesh where there will be a site card container
7:36
there will be a site card container
7:36
there will be a site card container acting like a proess service within a
7:38
acting like a proess service within a
7:38
acting like a proess service within a container along with a uh application
7:41
container along with a uh application
7:41
container along with a uh application servers and it will be connecting to a
7:44
servers and it will be connecting to a
7:44
servers and it will be connecting to a control panel which is Network managed
7:47
control panel which is Network managed
7:47
control panel which is Network managed and they will be handling this
7:49
and they will be handling this
7:49
and they will be handling this communication part and this is quite
7:51
communication part and this is quite
7:52
communication part and this is quite widely popular and is also used but
7:55
widely popular and is also used but
7:55
widely popular and is also used but which one to make use of is also
7:57
which one to make use of is also
7:57
which one to make use of is also dependent upon the application
7:58
dependent upon the application
7:58
dependent upon the application architecture and the
8:03
functionality
8:05
functionality
8:05
functionality so what is kfka how it is connected to
8:08
so what is kfka how it is connected to
8:08
so what is kfka how it is connected to the micros Services kfco is a messaging
8:11
the micros Services kfco is a messaging
8:11
the micros Services kfco is a messaging broker eventually this it was originated
8:13
broker eventually this it was originated
8:13
broker eventually this it was originated in a Linkin platform and handed over to
8:15
in a Linkin platform and handed over to
8:15
in a Linkin platform and handed over to Apache software Foundation as an open
8:17
Apache software Foundation as an open
8:17
Apache software Foundation as an open source it is a distributed message
8:20
source it is a distributed message
8:20
source it is a distributed message streaming platform and it can handle
8:23
streaming platform and it can handle
8:23
streaming platform and it can handle real-time data fits with high tro and
8:26
real-time data fits with high tro and
8:27
real-time data fits with high tro and low latencies and it is more preferred
8:30
low latencies and it is more preferred
8:31
low latencies and it is more preferred message broker in microservices
8:32
message broker in microservices
8:32
message broker in microservices architecture due to his robot features
8:35
architecture due to his robot features
8:35
architecture due to his robot features and ability to manage large volumes of
8:37
and ability to manage large volumes of
8:37
and ability to manage large volumes of data
8:39
data
8:39
data efficiently and um based on my
8:41
efficiently and um based on my
8:41
efficiently and um based on my experience personally we have done
8:43
experience personally we have done
8:43
experience personally we have done handling um millions of transaction
8:46
handling um millions of transaction
8:46
handling um millions of transaction every single day and we haven't seen any
8:48
every single day and we haven't seen any
8:48
every single day and we haven't seen any single uh latency issues or throughputs
8:51
single uh latency issues or throughputs
8:51
single uh latency issues or throughputs purely from a Kafka broker
8:53
purely from a Kafka broker
8:53
purely from a Kafka broker perspectives Kafka supports both publish
8:56
perspectives Kafka supports both publish
8:56
perspectives Kafka supports both publish subscriber and Qing model which is makes
8:59
subscriber and Qing model which is makes
8:59
subscriber and Qing model which is makes it more versatile for various use case
9:01
it more versatile for various use case
9:01
it more versatile for various use case scenarios uh let's take an example right
9:04
scenarios uh let's take an example right
9:04
scenarios uh let's take an example right again I'm sticking to the same um
9:06
again I'm sticking to the same um
9:06
again I'm sticking to the same um e-commerce or shopping website so um I'm
9:09
e-commerce or shopping website so um I'm
9:09
e-commerce or shopping website so um I'm picking up two users uh names in random
9:12
picking up two users uh names in random
9:12
picking up two users uh names in random let's say Adam is a confused consumer
9:16
let's say Adam is a confused consumer
9:16
let's say Adam is a confused consumer who likes to add uh many items to a
9:18
who likes to add uh many items to a
9:18
who likes to add uh many items to a shopping cart and then removes it and
9:21
shopping cart and then removes it and
9:21
shopping cart and then removes it and adds back a different product so there
9:24
adds back a different product so there
9:24
adds back a different product so there is a sequence associated with the items
9:26
is a sequence associated with the items
9:26
is a sequence associated with the items he is ordering to it in parallel let's
9:29
he is ordering to it in parallel let's
9:29
he is ordering to it in parallel let's say John is very um aware of what item
9:33
say John is very um aware of what item
9:33
say John is very um aware of what item he needs he just goes and adds the item
9:36
he needs he just goes and adds the item
9:36
he needs he just goes and adds the item whatever he wants to his cut and checks
9:39
whatever he wants to his cut and checks
9:39
whatever he wants to his cut and checks out so we have two users placing their
9:41
out so we have two users placing their
9:41
out so we have two users placing their order and they can be processed in
9:45
order and they can be processed in
9:45
order and they can be processed in parallel however the ERS items placed by
9:48
parallel however the ERS items placed by
9:48
parallel however the ERS items placed by Adam has to be processed in sequence
9:49
Adam has to be processed in sequence
9:49
Adam has to be processed in sequence because he added an item removed it and
9:52
because he added an item removed it and
9:52
because he added an item removed it and then added something else back if the
9:54
then added something else back if the
9:54
then added something else back if the sequence is missed up this product
9:57
sequence is missed up this product
9:57
sequence is missed up this product whatever is going to get delivered will
9:58
whatever is going to get delivered will
9:58
whatever is going to get delivered will be impacted
10:00
be impacted
10:00
be impacted so in case of Adam's order the sequence
10:02
so in case of Adam's order the sequence
10:02
so in case of Adam's order the sequence has to be maintained and however Adam
10:05
has to be maintained and however Adam
10:06
has to be maintained and however Adam and John orders can be processed in
10:08
and John orders can be processed in
10:08
and John orders can be processed in parallel so this is where Kafka is you
10:11
parallel so this is where Kafka is you
10:12
parallel so this is where Kafka is you know even more versatile and supports
10:14
know even more versatile and supports
10:14
know even more versatile and supports this scenario in Kafka we have something
10:17
this scenario in Kafka we have something
10:17
this scenario in Kafka we have something like partition key so this is what we
10:19
like partition key so this is what we
10:19
like partition key so this is what we are going to go a little bit deeper here
10:22
are going to go a little bit deeper here
10:22
are going to go a little bit deeper here so partition key is the one which is
10:25
so partition key is the one which is
10:25
so partition key is the one which is associated with every single message and
10:27
associated with every single message and
10:27
associated with every single message and defines in which uh partition they will
10:30
defines in which uh partition they will
10:30
defines in which uh partition they will land up with so Kafka have uh two major
10:34
land up with so Kafka have uh two major
10:34
land up with so Kafka have uh two major things one is the topic and second is a
10:37
things one is the topic and second is a
10:37
things one is the topic and second is a partition topic is the one it's more
10:39
partition topic is the one it's more
10:39
partition topic is the one it's more like a channel and within a channel we
10:41
like a channel and within a channel we
10:41
like a channel and within a channel we can have multiple
10:43
can have multiple
10:43
can have multiple partitions if we say uh check out to
10:48
partitions if we say uh check out to
10:48
partitions if we say uh check out to um delivery then that can be a topic and
10:52
um delivery then that can be a topic and
10:52
um delivery then that can be a topic and within those topics we can have multiple
10:54
within those topics we can have multiple
10:54
within those topics we can have multiple partitions in this case it can be S one
10:57
partitions in this case it can be S one
10:57
partitions in this case it can be S one topic for check out to delivery
10:59
topic for check out to delivery
10:59
topic for check out to delivery and Adam's order and John's order can go
11:02
and Adam's order and John's order can go
11:02
and Adam's order and John's order can go into multiple partitions and within the
11:05
into multiple partitions and within the
11:05
into multiple partitions and within the partition we can Define the partition
11:07
partition we can Define the partition
11:07
partition we can Define the partition key this is what will guarantee the
11:09
key this is what will guarantee the
11:09
key this is what will guarantee the sequence so Adam's order will go in the
11:13
sequence so Adam's order will go in the
11:13
sequence so Adam's order will go in the right sequence and will get fulfilled on
11:16
right sequence and will get fulfilled on
11:16
right sequence and will get fulfilled on the consumer side Define as long as the
11:19
the consumer side Define as long as the
11:19
the consumer side Define as long as the partion key is properly defined so this
11:21
partion key is properly defined so this
11:21
partion key is properly defined so this way the sequence is guaranteed um just
11:25
way the sequence is guaranteed um just
11:25
way the sequence is guaranteed um just to talk about this architecture flow
11:27
to talk about this architecture flow
11:27
to talk about this architecture flow right so here
11:29
right so here
11:29
right so here uh on the left hand side you will find
11:31
uh on the left hand side you will find
11:31
uh on the left hand side you will find the producer where I listed few services
11:34
the producer where I listed few services
11:34
the producer where I listed few services and on the right hand side there are
11:37
and on the right hand side there are
11:37
and on the right hand side there are consumers they can see a list of
11:39
consumers they can see a list of
11:39
consumers they can see a list of consumers taking the messages here the
11:42
consumers taking the messages here the
11:42
consumers taking the messages here the producer is this is an asynchronous
11:45
producer is this is an asynchronous
11:45
producer is this is an asynchronous architecture so the role of the producer
11:46
architecture so the role of the producer
11:46
architecture so the role of the producer is just to publish the message and
11:50
is just to publish the message and
11:50
is just to publish the message and consumers will consume and will process
11:52
consumers will consume and will process
11:52
consumers will consume and will process the message as per the functionality
11:55
the message as per the functionality
11:55
the message as per the functionality here the producer is not waiting for an
11:56
here the producer is not waiting for an
11:56
here the producer is not waiting for an acknowledgement or anything from a
11:58
acknowledgement or anything from a
11:58
acknowledgement or anything from a consumer the one
12:00
consumer the one
12:00
consumer the one latency is producer will get an ement
12:03
latency is producer will get an ement
12:03
latency is producer will get an ement from the KF Brokers that the message has
12:06
from the KF Brokers that the message has
12:06
from the KF Brokers that the message has been published and consumer will consume
12:09
been published and consumer will consume
12:09
been published and consumer will consume in whatever capacity they can and let's
12:12
in whatever capacity they can and let's
12:12
in whatever capacity they can and let's say the functionality of the consumer is
12:16
say the functionality of the consumer is
12:16
say the functionality of the consumer is very massive and then we need can scale
12:18
very massive and then we need can scale
12:18
very massive and then we need can scale up the more consumers without impacting
12:20
up the more consumers without impacting
12:20
up the more consumers without impacting the producer so here I just listed three
12:24
the producer so here I just listed three
12:24
the producer so here I just listed three broker Kafka Brokers uh normally the
12:27
broker Kafka Brokers uh normally the
12:27
broker Kafka Brokers uh normally the Kafka Brokers are selected in odd
12:29
Kafka Brokers are selected in odd
12:29
Kafka Brokers are selected in odd numbers and the number of replications
12:33
numbers and the number of replications
12:33
numbers and the number of replications or eventually can have all replications
12:35
or eventually can have all replications
12:36
or eventually can have all replications or at least like uh n+ 1 by two so that
12:40
or at least like uh n+ 1 by two so that
12:40
or at least like uh n+ 1 by two so that you will have uh better replications in
12:42
you will have uh better replications in
12:42
you will have uh better replications in terms of one broker goes down so I'm
12:45
terms of one broker goes down so I'm
12:45
terms of one broker goes down so I'm going to cover about the replications in
12:47
going to cover about the replications in
12:47
going to cover about the replications in a bit later um but um let's stick to
12:51
a bit later um but um let's stick to
12:51
a bit later um but um let's stick to this one so here each broker will have
12:56
this one so here each broker will have
12:56
this one so here each broker will have all the topics so if we have let's say
12:59
all the topics so if we have let's say
12:59
all the topics so if we have let's say five topics all the Brokers will have
13:01
five topics all the Brokers will have
13:01
five topics all the Brokers will have all the the five topics will be created
13:02
all the the five topics will be created
13:02
all the the five topics will be created on all the Brokers and each uh topics
13:06
on all the Brokers and each uh topics
13:06
on all the Brokers and each uh topics let's say we have three partitions
13:10
let's say we have three partitions
13:10
let's say we have three partitions then the leader for a partition will be
13:14
then the leader for a partition will be
13:14
then the leader for a partition will be you know won't be across one particular
13:16
you know won't be across one particular
13:16
you know won't be across one particular broker it will be divided across the
13:18
broker it will be divided across the
13:18
broker it will be divided across the Brokers this way there is more
13:21
Brokers this way there is more
13:21
Brokers this way there is more fragmentation and it is eventually
13:23
fragmentation and it is eventually
13:23
fragmentation and it is eventually distributed so in this example if you
13:26
distributed so in this example if you
13:26
distributed so in this example if you can see the partition zero broker one is
13:29
can see the partition zero broker one is
13:29
can see the partition zero broker one is is the leader and in broker 2 it is for
13:32
is the leader and in broker 2 it is for
13:32
is the leader and in broker 2 it is for the partition two and broker 3 is for
13:33
the partition two and broker 3 is for
13:33
the partition two and broker 3 is for partition one whereas the other Brokers
13:35
partition one whereas the other Brokers
13:35
partition one whereas the other Brokers are replicated this way one broker is
13:37
are replicated this way one broker is
13:37
are replicated this way one broker is not overloaded the functionality of the
13:40
not overloaded the functionality of the
13:40
not overloaded the functionality of the broker and things are equally shared
13:42
broker and things are equally shared
13:42
broker and things are equally shared load balanced pretty much and they can
13:45
load balanced pretty much and they can
13:45
load balanced pretty much and they can replicate and how many replications we
13:47
replicate and how many replications we
13:47
replicate and how many replications we need for each messages this is again to
13:50
need for each messages this is again to
13:50
need for each messages this is again to ensure fall tolerance we can Define it
13:53
ensure fall tolerance we can Define it
13:53
ensure fall tolerance we can Define it and if say one broker goes down then the
13:56
and if say one broker goes down then the
13:56
and if say one broker goes down then the rest of the two Brokers as long as they
13:58
rest of the two Brokers as long as they
13:58
rest of the two Brokers as long as they are replicated will elect a primary
14:00
are replicated will elect a primary
14:00
are replicated will elect a primary leader and will start supporting the
14:03
leader and will start supporting the
14:03
leader and will start supporting the functionality and zookeeper as it says
14:05
functionality and zookeeper as it says
14:05
functionality and zookeeper as it says it just to ensure the uh Kafka Brokers
14:08
it just to ensure the uh Kafka Brokers
14:08
it just to ensure the uh Kafka Brokers uh are connected and scaled
14:11
uh are connected and scaled
14:11
uh are connected and scaled up
14:14
up
14:14
up now one producer can uh publish to one
14:17
now one producer can uh publish to one
14:17
now one producer can uh publish to one message or there can be multiple
14:19
message or there can be multiple
14:19
message or there can be multiple producers publishing to one topic so
14:22
producers publishing to one topic so
14:22
producers publishing to one topic so that's where I just captured two a few
14:24
that's where I just captured two a few
14:24
that's where I just captured two a few Services here so the service uh two
14:27
Services here so the service uh two
14:27
Services here so the service uh two which is like considered as like two
14:29
which is like considered as like two
14:30
which is like considered as like two instances of a microservices which can
14:32
instances of a microservices which can
14:32
instances of a microservices which can produce a message and can publish it so
14:35
produce a message and can publish it so
14:35
produce a message and can publish it so this is where the scalability comes into
14:36
this is where the scalability comes into
14:36
this is where the scalability comes into the part if you're going to increase the
14:38
the part if you're going to increase the
14:38
the part if you're going to increase the number of BS to scale up then this will
14:41
number of BS to scale up then this will
14:41
number of BS to scale up then this will support that functionality and same way
14:44
support that functionality and same way
14:44
support that functionality and same way on the consumer side I can have uh one
14:46
on the consumer side I can have uh one
14:46
on the consumer side I can have uh one consumer and if I need to scale up uh to
14:50
consumer and if I need to scale up uh to
14:50
consumer and if I need to scale up uh to increase the throughput I can scale up
14:52
increase the throughput I can scale up
14:52
increase the throughput I can scale up the consumers and when the consumer
14:54
the consumers and when the consumer
14:54
the consumers and when the consumer group is unique then the micros Serv and
14:58
group is unique then the micros Serv and
14:58
group is unique then the micros Serv and kka uh eventually goes hand in hand here
15:01
kka uh eventually goes hand in hand here
15:01
kka uh eventually goes hand in hand here so we can divide the partitions to the
15:04
so we can divide the partitions to the
15:04
so we can divide the partitions to the different consumer threats so the
15:06
different consumer threats so the
15:06
different consumer threats so the service 2A let's say it's like the order
15:09
service 2A let's say it's like the order
15:09
service 2A let's say it's like the order of fulfillment can uh consume from
15:12
of fulfillment can uh consume from
15:12
of fulfillment can uh consume from partition zero and one and second one
15:14
partition zero and one and second one
15:14
partition zero and one and second one can consume uh you know the second
15:16
can consume uh you know the second
15:16
can consume uh you know the second instance can consume from partition
15:24
two so Kafka resilience and high ility
15:29
two so Kafka resilience and high ility
15:29
two so Kafka resilience and high ility is this what we talk U I want to Deep
15:32
is this what we talk U I want to Deep
15:32
is this what we talk U I want to Deep dive so it is highly um you know fall
15:35
dive so it is highly um you know fall
15:35
dive so it is highly um you know fall tolerant and there are a lot of Disaster
15:38
tolerant and there are a lot of Disaster
15:38
tolerant and there are a lot of Disaster Recovery strategies which we can make
15:40
Recovery strategies which we can make
15:40
Recovery strategies which we can make use of so this is where the partition
15:42
use of so this is where the partition
15:42
use of so this is where the partition things will come into the picture in
15:44
things will come into the picture in
15:44
things will come into the picture in addition we are replicating the Brokers
15:47
addition we are replicating the Brokers
15:47
addition we are replicating the Brokers right in also the Clusters can also be
15:50
right in also the Clusters can also be
15:50
right in also the Clusters can also be replicated across different data centers
15:54
replicated across different data centers
15:54
replicated across different data centers so how it is um before we talk about the
15:57
so how it is um before we talk about the
15:57
so how it is um before we talk about the disaster how it is fter in general C
15:59
disaster how it is fter in general C
16:00
disaster how it is fter in general C pures the message to the disk which
16:02
pures the message to the disk which
16:02
pures the message to the disk which ensures the data durability and also a
16:05
ensures the data durability and also a
16:05
ensures the data durability and also a ability to reprocess them internally
16:07
ability to reprocess them internally
16:08
ability to reprocess them internally within the cluster we can choose the
16:09
within the cluster we can choose the
16:09
within the cluster we can choose the number of
16:10
number of
16:10
number of replications and to make it even more
16:13
replications and to make it even more
16:13
replications and to make it even more replicated we can have mirror maker or
16:16
replicated we can have mirror maker or
16:16
replicated we can have mirror maker or Kafka replicator so if we have two
16:19
Kafka replicator so if we have two
16:19
Kafka replicator so if we have two clusters say one sits in uh Nam Data
16:22
clusters say one sits in uh Nam Data
16:22
clusters say one sits in uh Nam Data Center and another cluster sits in uh
16:25
Center and another cluster sits in uh
16:25
Center and another cluster sits in uh emia or ape data center now one cluster
16:30
emia or ape data center now one cluster
16:30
emia or ape data center now one cluster generates a message which acts like a
16:31
generates a message which acts like a
16:31
generates a message which acts like a primary and your disaster recovery
16:33
primary and your disaster recovery
16:33
primary and your disaster recovery system is somewhere sits in a different
16:35
system is somewhere sits in a different
16:35
system is somewhere sits in a different uh data center then the mirror maker can
16:40
uh data center then the mirror maker can
16:40
uh data center then the mirror maker can connect these two clusters they will be
16:41
connect these two clusters they will be
16:41
connect these two clusters they will be replicating the messages from one
16:43
replicating the messages from one
16:43
replicating the messages from one cluster to the other and if we have some
16:46
cluster to the other and if we have some
16:46
cluster to the other and if we have some Network alas wi IPS or something then it
16:49
Network alas wi IPS or something then it
16:49
Network alas wi IPS or something then it makes it easier so if this cluster goes
16:52
makes it easier so if this cluster goes
16:52
makes it easier so if this cluster goes down seamlessly it will connect to the
16:55
down seamlessly it will connect to the
16:55
down seamlessly it will connect to the data center cluster this way the micros
16:57
data center cluster this way the micros
16:57
data center cluster this way the micros servic is completely Loosely coupled
16:59
servic is completely Loosely coupled
16:59
servic is completely Loosely coupled from the under underlying infrastructure
17:01
from the under underlying infrastructure
17:01
from the under underlying infrastructure components
17:03
components
17:03
components and scalability wise so there are uh two
17:08
and scalability wise so there are uh two
17:08
and scalability wise so there are uh two major things we can handle with so first
17:12
major things we can handle with so first
17:12
major things we can handle with so first to talk about is the vertical scaling so
17:15
to talk about is the vertical scaling so
17:15
to talk about is the vertical scaling so vertical scaling is like adding more CPU
17:19
vertical scaling is like adding more CPU
17:19
vertical scaling is like adding more CPU Ram uh that is the memory uh and the dis
17:22
Ram uh that is the memory uh and the dis
17:22
Ram uh that is the memory uh and the dis space so that if the cluster runs slow
17:25
space so that if the cluster runs slow
17:25
space so that if the cluster runs slow on the underlying U infrastructure uh
17:28
on the underlying U infrastructure uh
17:28
on the underlying U infrastructure uh component
17:29
component
17:29
component then we can scale up them whereas in the
17:31
then we can scale up them whereas in the
17:31
then we can scale up them whereas in the horizontal scaling we are actually
17:33
horizontal scaling we are actually
17:33
horizontal scaling we are actually adding more clusters sorry more Brokers
17:36
adding more clusters sorry more Brokers
17:36
adding more clusters sorry more Brokers to the cluster so um earlier we saw like
17:39
to the cluster so um earlier we saw like
17:39
to the cluster so um earlier we saw like a three broker cluster we can make it
17:41
a three broker cluster we can make it
17:41
a three broker cluster we can make it like a five broker cluster or something
17:43
like a five broker cluster or something
17:43
like a five broker cluster or something but the downside of this is either of
17:45
but the downside of this is either of
17:45
but the downside of this is either of this approach will um you know need some
17:49
this approach will um you know need some
17:49
this approach will um you know need some downtime which will impact the
17:51
downtime which will impact the
17:51
downtime which will impact the functional components and especially it
17:53
functional components and especially it
17:53
functional components and especially it depends on whether we are running on a
17:54
depends on whether we are running on a
17:54
depends on whether we are running on a bare metal uh for
17:56
bare metal uh for
17:56
bare metal uh for a vertical scaling then it needs a
17:59
a vertical scaling then it needs a
17:59
a vertical scaling then it needs a reboot or it depends on the virtual
18:01
reboot or it depends on the virtual
18:01
reboot or it depends on the virtual machine and something that's becoming
18:04
machine and something that's becoming
18:04
machine and something that's becoming popular in the recent time is elastic
18:06
popular in the recent time is elastic
18:06
popular in the recent time is elastic scaling so just like how the
18:08
scaling so just like how the
18:08
scaling so just like how the microservices application architecture
18:10
microservices application architecture
18:10
microservices application architecture runs in the microservice environment uh
18:14
runs in the microservice environment uh
18:14
runs in the microservice environment uh in a cloud the infrastructure components
18:16
in a cloud the infrastructure components
18:16
in a cloud the infrastructure components can also run in Cloud which will enable
18:19
can also run in Cloud which will enable
18:19
can also run in Cloud which will enable it to scale seamlessly adding more
18:22
it to scale seamlessly adding more
18:22
it to scale seamlessly adding more Brokers to the cluster or having
18:24
Brokers to the cluster or having
18:24
Brokers to the cluster or having multiple clusters you know
18:25
multiple clusters you know
18:25
multiple clusters you know interconnected this way it uses the
18:29
interconnected this way it uses the
18:29
interconnected this way it uses the downtime or literally to say
18:30
downtime or literally to say
18:31
downtime or literally to say theoretically uh 0 percentage uh so the
18:34
theoretically uh 0 percentage uh so the
18:34
theoretically uh 0 percentage uh so the applications will always be connected to
18:36
applications will always be connected to
18:36
applications will always be connected to Kafka Brokers and they are all load
18:38
Kafka Brokers and they are all load
18:38
Kafka Brokers and they are all load balanced so either of the cluster or
18:41
balanced so either of the cluster or
18:41
balanced so either of the cluster or Brokers will be available to serve the
18:49
components
18:50
components
18:50
components and to talk about the utilities
18:54
and to talk about the utilities
18:54
and to talk about the utilities um I'll try to keep this short um
18:57
um I'll try to keep this short um
18:57
um I'll try to keep this short um because um we'll have some time for the
18:59
because um we'll have some time for the
18:59
because um we'll have some time for the questions so Kafka in offers case SQL
19:04
questions so Kafka in offers case SQL
19:04
questions so Kafka in offers case SQL that is we can go and query the messages
19:07
that is we can go and query the messages
19:07
that is we can go and query the messages using a k SQL DP through which we can uh
19:11
using a k SQL DP through which we can uh
19:11
using a k SQL DP through which we can uh extract some valuable insight and
19:13
extract some valuable insight and
19:13
extract some valuable insight and normally this is not widely used unless
19:15
normally this is not widely used unless
19:15
normally this is not widely used unless there is a specific use cases and um lot
19:18
there is a specific use cases and um lot
19:18
there is a specific use cases and um lot of Enterprise architectures normally
19:20
of Enterprise architectures normally
19:20
of Enterprise architectures normally serialize the messages over Kafka like
19:23
serialize the messages over Kafka like
19:23
serialize the messages over Kafka like uh Google using Google Proto or
19:25
uh Google using Google Proto or
19:25
uh Google using Google Proto or something in that case this one come
19:27
something in that case this one come
19:27
something in that case this one come handy as long as if you using a string
19:29
handy as long as if you using a string
19:29
handy as long as if you using a string or Json serializer this will work where
19:31
or Json serializer this will work where
19:31
or Json serializer this will work where we'll be able to query using the um raw
19:35
we'll be able to query using the um raw
19:35
we'll be able to query using the um raw language but if it is serialized then
19:38
language but if it is serialized then
19:38
language but if it is serialized then this won't be very efficient and it will
19:40
this won't be very efficient and it will
19:40
this won't be very efficient and it will uh increase the load on the
19:42
uh increase the load on the
19:42
uh increase the load on the Brokers Kafka connect is like a utility
19:45
Brokers Kafka connect is like a utility
19:45
Brokers Kafka connect is like a utility tools where we can Source from a
19:47
tools where we can Source from a
19:47
tools where we can Source from a different
19:49
different
19:49
different uh components like a database or elastic
19:52
uh components like a database or elastic
19:52
uh components like a database or elastic search and publish the messages to K
19:54
search and publish the messages to K
19:54
search and publish the messages to K more like an utility out of the box
19:57
more like an utility out of the box
19:57
more like an utility out of the box without developing a component together
19:59
without developing a component together
19:59
without developing a component together um control center is something very uh
20:02
um control center is something very uh
20:02
um control center is something very uh critical because this is where we can
20:05
critical because this is where we can
20:05
critical because this is where we can monitor the health of the cluster we
20:07
monitor the health of the cluster we
20:07
monitor the health of the cluster we will know what how many consumers or
20:08
will know what how many consumers or
20:08
will know what how many consumers or producers how they um throughput what is
20:12
producers how they um throughput what is
20:12
producers how they um throughput what is that uh range and if there are any
20:16
that uh range and if there are any
20:16
that uh range and if there are any lacks we can dynamically scale up the uh
20:19
lacks we can dynamically scale up the uh
20:19
lacks we can dynamically scale up the uh consumer ports so this gives and we can
20:22
consumer ports so this gives and we can
20:22
consumer ports so this gives and we can even generate alerts and if we
20:24
even generate alerts and if we
20:24
even generate alerts and if we integrated this with the microservice
20:26
integrated this with the microservice
20:26
integrated this with the microservice component we can sense okay the
20:28
component we can sense okay the
20:28
component we can sense okay the throughputs are increasing uh or it's
20:31
throughputs are increasing uh or it's
20:31
throughputs are increasing uh or it's low and that means we need to scale up
20:34
low and that means we need to scale up
20:34
low and that means we need to scale up more consumers to increase the
20:36
more consumers to increase the
20:36
more consumers to increase the throughput so The Lax can be drained
20:38
throughput so The Lax can be drained
20:38
throughput so The Lax can be drained faster replicate meraker is the one
20:41
faster replicate meraker is the one
20:41
faster replicate meraker is the one which is covered bit so this is mainly
20:43
which is covered bit so this is mainly
20:43
which is covered bit so this is mainly for the uh replication across different
20:46
for the uh replication across different
20:46
for the uh replication across different uh clusters or data which might be
20:48
uh clusters or data which might be
20:48
uh clusters or data which might be residing in different data
20:56
centers so benefits of the using Kafka
20:59
centers so benefits of the using Kafka
20:59
centers so benefits of the using Kafka in the micros Services part this is the
21:01
in the micros Services part this is the
21:01
in the micros Services part this is the highr put which we were talking about it
21:04
highr put which we were talking about it
21:04
highr put which we were talking about it can handle millions of
21:06
can handle millions of
21:06
can handle millions of transactions seamlessly uh we are
21:09
transactions seamlessly uh we are
21:09
transactions seamlessly uh we are talking about in uh seconds and in
21:12
talking about in uh seconds and in
21:12
talking about in uh seconds and in general Kafka can have one broker in a
21:16
general Kafka can have one broker in a
21:16
general Kafka can have one broker in a cluster can handle 4,000 partitions and
21:19
cluster can handle 4,000 partitions and
21:19
cluster can handle 4,000 partitions and if we talking about a three note cluster
21:21
if we talking about a three note cluster
21:21
if we talking about a three note cluster we are um talking about like a 12,000
21:24
we are um talking about like a 12,000
21:24
we are um talking about like a 12,000 partitions that 12,000 partitions can be
21:26
partitions that 12,000 partitions can be
21:26
partitions that 12,000 partitions can be across multiple topics which makes it
21:29
across multiple topics which makes it
21:29
across multiple topics which makes it more robust to Scale based on the
21:31
more robust to Scale based on the
21:31
more robust to Scale based on the application needs and their FA tolerance
21:35
application needs and their FA tolerance
21:35
application needs and their FA tolerance and the message and durable device they
21:36
and the message and durable device they
21:36
and the message and durable device they are persisted in the hard drive so it's
21:40
are persisted in the hard drive so it's
21:40
are persisted in the hard drive so it's easily recordable and it decoupled uh
21:43
easily recordable and it decoupled uh
21:43
easily recordable and it decoupled uh the micros service components and mainly
21:45
the micros service components and mainly
21:45
the micros service components and mainly for the realtime processing so using the
21:48
for the realtime processing so using the
21:48
for the realtime processing so using the partition key we are ordering the
21:49
partition key we are ordering the
21:49
partition key we are ordering the messaging orders and they are all replay
21:52
messaging orders and they are all replay
21:52
messaging orders and they are all replay capabilities um which we where we can
21:55
capabilities um which we where we can
21:55
capabilities um which we where we can republish the messages
21:57
republish the messages
21:57
republish the messages um on the consumer side uh there is
22:00
um on the consumer side uh there is
22:00
um on the consumer side uh there is something called like a consumer offset
22:02
something called like a consumer offset
22:02
something called like a consumer offset number so when a consumer consumes a
22:04
number so when a consumer consumes a
22:04
number so when a consumer consumes a message it up uh updates offset this is
22:08
message it up uh updates offset this is
22:08
message it up uh updates offset this is where the broken knows um okay this
22:10
where the broken knows um okay this
22:11
where the broken knows um okay this message has been consumed and if we need
22:13
message has been consumed and if we need
22:13
message has been consumed and if we need to replay messages obviously we can go
22:15
to replay messages obviously we can go
22:15
to replay messages obviously we can go and resar the offset um to a number or
22:18
and resar the offset um to a number or
22:18
and resar the offset um to a number or to the beginning and we can replay the
22:20
to the beginning and we can replay the
22:20
to the beginning and we can replay the entire messages and in case if a
22:23
entire messages and in case if a
22:23
entire messages and in case if a consumer Co
22:25
consumer Co
22:25
consumer Co down the broker will hold the messages
22:28
down the broker will hold the messages
22:28
down the broker will hold the messages as long as the consumer group name is
22:29
as long as the consumer group name is
22:29
as long as the consumer group name is same and when the cons uh consumer comes
22:31
same and when the cons uh consumer comes
22:31
same and when the cons uh consumer comes up it can start consuming the messages
22:33
up it can start consuming the messages
22:33
up it can start consuming the messages so the messages are not lost and the
22:37
so the messages are not lost and the
22:37
so the messages are not lost and the offsets are maintained properly and with
22:40
offsets are maintained properly and with
22:40
offsets are maintained properly and with Kafka we can also have the streaming
22:41
Kafka we can also have the streaming
22:41
Kafka we can also have the streaming processing which is the Kafka streams
22:44
processing which is the Kafka streams
22:44
processing which is the Kafka streams and uh where we can go by uh window
22:47
and uh where we can go by uh window
22:47
and uh where we can go by uh window paste to aggregate the messages or based
22:50
paste to aggregate the messages or based
22:50
paste to aggregate the messages or based on the rule where we need to drop
22:52
on the rule where we need to drop
22:52
on the rule where we need to drop certain events uh this comes in handy
22:55
certain events uh this comes in handy
22:55
certain events uh this comes in handy when we want to ignore certain messages
22:57
when we want to ignore certain messages
22:57
when we want to ignore certain messages or aggregate certain events by time
22:59
or aggregate certain events by time
23:00
or aggregate certain events by time window but one thing with an
23:02
window but one thing with an
23:02
window but one thing with an window-based approach is like the
23:04
window-based approach is like the
23:05
window-based approach is like the windows should be Rel smaller because
23:07
windows should be Rel smaller because
23:07
windows should be Rel smaller because that's going to put a lot of uh stress
23:09
that's going to put a lot of uh stress
23:09
that's going to put a lot of uh stress on the broker because it's constantly uh
23:12
on the broker because it's constantly uh
23:12
on the broker because it's constantly uh holding the window for the messages to
23:14
holding the window for the messages to
23:14
holding the window for the messages to handle and during the if there is a high
23:16
handle and during the if there is a high
23:16
handle and during the if there is a high throughput messages high volume
23:18
throughput messages high volume
23:18
throughput messages high volume applications then we are increasing the
23:21
applications then we are increasing the
23:21
applications then we are increasing the complexity now that being said uh if
23:26
complexity now that being said uh if
23:26
complexity now that being said uh if there is a necessity to do that uh there
23:28
there is a necessity to do that uh there
23:28
there is a necessity to do that uh there is a other component that's where it
23:30
is a other component that's where it
23:30
is a other component that's where it comes uh Flink so Apache Flink is
23:35
comes uh Flink so Apache Flink is
23:35
comes uh Flink so Apache Flink is something where we it offers a big data
23:38
something where we it offers a big data
23:38
something where we it offers a big data kind of streaming so if you are heavily
23:41
kind of streaming so if you are heavily
23:41
kind of streaming so if you are heavily dependent on streaming for a batch
23:42
dependent on streaming for a batch
23:42
dependent on streaming for a batch processing then this is not the
23:45
processing then this is not the
23:45
processing then this is not the component where we need to build it we
23:46
component where we need to build it we
23:46
component where we need to build it we can switch out to the
23:50
Flink so some uh tips to handle
23:56
Flink so some uh tips to handle
23:56
Flink so some uh tips to handle um avoid delaying messages like I said
23:59
um avoid delaying messages like I said
23:59
um avoid delaying messages like I said do not build the messages or have a
24:02
do not build the messages or have a
24:02
do not build the messages or have a large window uh for the messages to hold
24:05
large window uh for the messages to hold
24:05
large window uh for the messages to hold it across so this will increase the
24:09
it across so this will increase the
24:09
it across so this will increase the stress on the broker and Kafka is mainly
24:11
stress on the broker and Kafka is mainly
24:11
stress on the broker and Kafka is mainly for the real-time even driven
24:12
for the real-time even driven
24:13
for the real-time even driven application
24:14
application
24:14
application so stick to that part um so it was good
24:18
so stick to that part um so it was good
24:18
so stick to that part um so it was good in that case it's not a caching solution
24:21
in that case it's not a caching solution
24:21
in that case it's not a caching solution so there have been scenarios I have seen
24:23
so there have been scenarios I have seen
24:23
so there have been scenarios I have seen people uh using it for caching using a
24:25
people uh using it for caching using a
24:25
people uh using it for caching using a key value kind of pair no go for a
24:27
key value kind of pair no go for a
24:27
key value kind of pair no go for a distributed caching solutions that could
24:29
distributed caching solutions that could
24:29
distributed caching solutions that could be rers or mongod TPS or a bunch of
24:31
be rers or mongod TPS or a bunch of
24:31
be rers or mongod TPS or a bunch of things available um based on your
24:34
things available um based on your
24:34
things available um based on your application needs but it's not for Kafka
24:37
application needs but it's not for Kafka
24:37
application needs but it's not for Kafka uh and when we talk about microservices
24:39
uh and when we talk about microservices
24:39
uh and when we talk about microservices right A lot of people
24:42
right A lot of people
24:42
right A lot of people um have a dependency on database for
24:44
um have a dependency on database for
24:44
um have a dependency on database for static data or to Pur the messages it
24:47
static data or to Pur the messages it
24:47
static data or to Pur the messages it will be good to decouple them have its
24:49
will be good to decouple them have its
24:49
will be good to decouple them have its own separate micros service layer and
24:52
own separate micros service layer and
24:52
own separate micros service layer and use the Kafka to publish the messages
24:55
use the Kafka to publish the messages
24:55
use the Kafka to publish the messages this way if there are any underlying
24:57
this way if there are any underlying
24:57
this way if there are any underlying bottlenecks uh with your database layer
25:00
bottlenecks uh with your database layer
25:00
bottlenecks uh with your database layer where it can process one certain number
25:01
where it can process one certain number
25:01
where it can process one certain number of messages for insert or a trigger
25:04
of messages for insert or a trigger
25:04
of messages for insert or a trigger based on the complexities it is
25:06
based on the complexities it is
25:06
based on the complexities it is decoupled and the application can still
25:08
decoupled and the application can still
25:08
decoupled and the application can still run on the caching
25:09
run on the caching
25:09
run on the caching solution so K SQL Kafka streams and
25:12
solution so K SQL Kafka streams and
25:13
solution so K SQL Kafka streams and Flink so this is the part I was covering
25:15
Flink so this is the part I was covering
25:15
Flink so this is the part I was covering uh mentioning a little bit earlier Flink
25:17
uh mentioning a little bit earlier Flink
25:17
uh mentioning a little bit earlier Flink is from Apache kfka which is for Big
25:21
is from Apache kfka which is for Big
25:21
is from Apache kfka which is for Big Data streaming and
25:22
Data streaming and
25:23
Data streaming and Analysis which is efficient if you want
25:25
Analysis which is efficient if you want
25:25
Analysis which is efficient if you want to have a batch or realtime streaming
25:28
to have a batch or realtime streaming
25:28
to have a batch or realtime streaming and
25:29
and
25:29
and it has its own pros and cons so but
25:32
it has its own pros and cons so but
25:32
it has its own pros and cons so but definitely Kafka streams and Flink have
25:35
definitely Kafka streams and Flink have
25:35
definitely Kafka streams and Flink have their own use cases do not to use Kafka
25:37
their own use cases do not to use Kafka
25:37
their own use cases do not to use Kafka streams for analytics or holding the
25:40
streams for analytics or holding the
25:40
streams for analytics or holding the messages for batch processing instead go
25:42
messages for batch processing instead go
25:42
messages for batch processing instead go for Flink and K
25:44
for Flink and K
25:44
for Flink and K SQL is for quering the messages unless
25:47
SQL is for quering the messages unless
25:47
SQL is for quering the messages unless there isn't any dependency uh it's not
25:51
there isn't any dependency uh it's not
25:51
there isn't any dependency uh it's not widely used then be mindful while
25:55
widely used then be mindful while
25:55
widely used then be mindful while defining the number of partitions and
25:56
defining the number of partitions and
25:56
defining the number of partitions and topics so
25:59
topics so
25:59
topics so what does it mean as I said each broker
26:00
what does it mean as I said each broker
26:00
what does it mean as I said each broker have a limit of 4,000 partitions and if
26:03
have a limit of 4,000 partitions and if
26:03
have a limit of 4,000 partitions and if we are creating one topic with 100
26:05
we are creating one topic with 100
26:05
we are creating one topic with 100 partitions then eventually by the simple
26:08
partitions then eventually by the simple
26:08
partitions then eventually by the simple math then we are limiting the number of
26:10
math then we are limiting the number of
26:10
math then we are limiting the number of topics we could create so create a
26:13
topics we could create so create a
26:13
topics we could create so create a partitions and it can increase the
26:15
partitions and it can increase the
26:15
partitions and it can increase the number of partitions as there is a need
26:17
number of partitions as there is a need
26:17
number of partitions as there is a need for the application to scale up
26:22
[Music]