The Internet of Things(IoT) has gained heaps of recognition in recent years, because of its rise in use cases. What was ab initio thought of as a shopper discretionary technical school nology is currently thought of to be the long run of tech. IoT devices run as separate processes and also the biggest challenge is to merge all the assorted knowledge processes to make one meaning knowledge stream.

For example, as trivial as it sounds, mind you it isn’t. Suppose you’re asking your connected automotive to administer you directions for the closest dish joint. What happens within the side is, the car-connected pc passes this question to the cloud, the cloud then processes multiple knowledge streams to administer you the relevant info. 

Firstly it shows you the map knowledge and recommends the quickest route to the dish joint. the pc also will take under consideration the stockpile in your automotive and counsel supply if the placement may be a bit additional. Then it shows you the traffic info and calculable wait time, that is gathered from another knowledge stream. Then it shows you varied different close businesses which might additionally serve you smart pizzas. Then it additionally shows you the weather info and it additionally shows you if there’s any massive event happening within the neighborhood of the food place. This can be what proportion of knowledge is processed for an easy seek for smart dishes during a connected automotive. 

Kafka Connect may be a framework already enclosed within the package. This helps integrate Kafka into different systems. This may assist you to add a replacement system to an ascendable and secure event streaming network.

Here square measure some use cases of IoT and Datastream platforms :

In-Store Shopping: period info sharing between mobile apps, weather, geo-location, CRM, and loyalty programs to make a customized recommendation for looking.

Connected Car: period knowledge sharing between the cloud and also the device victimisation of a web network to indicate relevant info to the client.

Industrial Machinery: Industrial kinds of machinery typically have heaps of moving components, thus it helps to understand early if a {particular|an exact|a precise|a definite|an explicit} part is nearing its lifecycle or not operating optimally in order that it is replaced or scrapped while not important loss of machine-hours.

Kafka in python?

Although there square measure varied libraries accessible within the python artificial language to use Kafka, given below square measure a number of the popular ones:

Kafka-Python: this can be Associate in Nursing ASCII text file library designed by the Python community.

PyKafka: This library is maintained by Parsley and it’s claimed to be a Pythonic API. However, we tend to cannot produce dynamic topics during this library like Kafka-Python

Confluent Python Kafka: This library is provided by convergent as a skinny wrapper around librdkafka. Thus, it performs higher than the higher than 2.

Below may be a orient a way to set-up “Kafka-Python” :

I am reaching to install Kafka-Python employing a pip installer.

Code Syntax: $ pip install kafka-python  

Project Code:

Lets produce a knowledge producer that generates numbers from one to five hundred and send them finally to the Kafka broker. Later a shopper can scan that knowledge from the broker and keep them during a MongoDB assortment.

One of the necessary edges of victimisation Kafka is that just in case one amongst the patron brokers stops operating, there’s continuously a backup system which is able to persevere feeding knowledge from the last recorded purpose. This can be a helpful technique that helps heaps within the times of would like. The info integrity is maintained throughout the method. Here may be Kafka stream info.

Let’s currently produce a python program file known as and import some libraries and modules. 

File: turn

# commercialism the specified libraries  

from time import sleep  

from json import dumps  

from Kafka import KafkaProducer

# initializing the Kafka producer  

my_producer = KafkaProducer(  

    bootstrap_servers = [‘localhost:9092’],  

    value_serializer = lambda x:dumps(x).encode(‘utf-8’)  


Tip: If you see whether or not it’s operating or not, notice the serializer. it’ll mechanically rework and write the info

Next Step:

File: turn 

# generating the numbers starting from one to five hundred  

for n in range(500):  

    my_data =   

    my_producer.send(‘testnum’, price = my_data)  



Want to envision the code?

it is counseled to make a replacement topic and send the info to it freshly generated topic. This methodology can avoid any case of duplicate values and doable confusion within the testnum topic once we are going to be testing the producer and shopper along.

Now however will we use this knowledge for consumption:

Before we tend to start with the secret writing part of the patron, allow us to produce a replacement Python program file and name it We’ll import a number of the modules like json.loads, MongoClient and KafkaConsumer. # commercialism the specified modules  

from json import masses  

from Kafka import KafkaConsumer  

from pymongo import MongoClient  

Use this operate to get a Kafka consumer:

# generating the Kafka shopper  

my_consumer = KafkaConsumer(  


     bootstrap_servers = [‘localhost : 9092’],  

     auto_offset_reset = ‘earliest’,  

     enable_auto_commit = True,  

     group_id = ‘my-group’,  

     value_deserializer = lambda x : loads(x.decode(‘utf-8’)) 

my_client = MongoClient(‘localhost : 27017’)  

my_collection = my_client.testnum.testnum  

for message in my_consumer:  

    message = message.value  


    print(message + ” additional to ” + my_collection) 

We have used the for-loop to repeat through the patron so as to extract the info. currently so as to check the code, one will execute the turn file 1st so

Read also: 7 Things Every Computer User Should Know

By Anil kondla

Anil is an enthusiastic, self-motivated, reliable person who is a Technology evangelist. He's always been fascinated at work from 7 years especially at innovation that causes benefit to the students, working professionals or the companies. Being unique and thinking Innovative is what he loves the most, supporting his thoughts he will be ahead for any change valuing social responsibility with a reprising innovation. His interest in various fields like Tech, entertainment, gadgets, travel and lifestyle that urge to explore, led him to find places to put himself to work and design things than just learning. Follow him on LinkedIn

Leave a comment

Your email address will not be published. Required fields are marked *