Get message from first offset with kafka consumer in python

background

I put real-time data in kafka, but I wanted to get the data from the beginning of offset. The method is not written in the document, so write it as a memorandum

Library

There seem to be some kafka libraries kafka-I decided to use python 1.3.2

code
from kafka import KafkaConsumer, TopicPartition
consumer = KafkaConsumer(bootstrap_servers='hoge')

tp = TopicPartition('topic_name', 0)
consumer.assign([tp])
consumer.seek(tp, 0)

for msg in consumer:
    // do something
Description

Specify the topic name using TopicPartition without specifying the topic in the constructor of KafkaConsumer. After that, you can specify Topic Partition and offset in seek.

Recommended Posts

Get message from first offset with kafka consumer in python
Get data from Quandl in Python
Get started with Python in Blender
Get additional data in LDAP with python
Get html from element with Python selenium
Get exchange rates from open exchange rates in Python
[Note] Get data from PostgreSQL with Python
Get battery level from SwitchBot in Python
Get metric history from MLflow in Python
Get time series data from k-db.com in Python
[Python] Get the files in a folder with Python
[Python] Get one year's message history from Slack
Get only articles from web pages in Python
[Cloud102] # 1 Get Started with Python (Part 1 Python First Steps)
Exception message in Python
Get date in Python
Get date with python
[First API] Try to get Qiita articles with Python
[Python] Get the numbers in the graph image with OCR
Get schedule from Garoon SOAP API with Python + Zeep
Get data from GPS module at 10Hz in Python
Get the result in dict format with Python psycopg2
Python beginners get stuck with their first web scraping
Get mail from Gmail and label it with Python3
Get data from database via ODBC with Python (Access)
Get the value of a specific key in a list from the dictionary type in the list with Python
Get country code with python
Scraping with selenium in Python
Get last month in python
Working with LibreOffice in Python
Get your heart rate from the fitbit API in Python!
Get Twitter timeline with python
Scraping with chromedriver in python
Debugging with pdb in Python
OCR from PDF in Python
Working with sounds in Python
Scraping with Selenium in Python
First neuron simulation with NEURON + Python
Select file in dialog with python → Display file name in message box
Tweet with image in Python
Get Terminal size in Python
Explicitly get EOF in python
Combined with permutations in Python
Hit REST in Python to get data from New Relic
Get macro constants from C (++) header file (.h) in Python
Get data from analytics API with Google API Client for python
Get started with Python! ~ ② Grammar ~
Get stock price with Python
Get Evernote notes in Python
Get home directory with python
Get keyboard events with python
With skype, notify with skype from python!
Get Alembic information with Python
Get Japanese synonyms in Python
How is the progress? Let's get on with the boom ?? in Python
Get PowerShell commands from malware dynamic analysis site with BeautifulSoup + Python
Post a message from IBM Cloud Functions to Slack in Python
How to get a string from a command line argument in python
Get data from MySQL on a VPS with Python 3 and SQLAlchemy
Get US stock price from Python with Web API with Raspberry Pi
Get a list of files in a folder with python without a path