Hello everyone,
i am trying to implement a Data streaming application and since i am familiar with Django i want to use Django channels that work good with Redis. However my starting plan was to take data from a Kafka cluster. Is there a way to connect Redis to kafka and use it as a cashe for my application? Is Redis capable enough to replace Kafka and use it as my main message bus and send data to another database and other consumers? Any advice will be apreciated.
Thanks a lot!!
Hello, there is both a way to connect Redis with Kafka and a way to use Redis instead of Kafka to meet your goals.
To seamlessly connect Redis with Kafka you can leverage our Kafka Connect sink connector which can be downloaded from Redis Connector (Source and Sink) by Redis | Confluent Hub.
To replace Kafka as a publish-subscribe distributed message broker you can leverage the Redis Streams data structure which is a time-ordered in-memory log. You can think of each Redis Stream key as the equivalent to a Kafka Topic Partition including the ability to have multiple Consumer Groups listening onto it. There is a good tutorial about it here - Redis Streams tutorial | Redis
Let us know what you decide and if you have any further questions.
Best of luck, Allen
1 Like
First of all thank you a lot for this reply and the links for the tutorials, i haven’t found much on the interaction between these two technologies.
Since my data are not that many, or too complex the second case of using redis as an alternative to Kafka will work for me. However it is a project for my thesis and as a first idea i wanted to use Kafka cause of its scalability and its wider use in industrial environments.
My question is; since Django is a Python framework i must use a python version of kafka, and the confluent connectors are not supported in these versions. Am i right or am i missing something?
For the time being i will try to implement the second option and i will decide later on any possible change of path, however any advice on both matters would be appreciated. 
If you are using a python framework then you likely only require a python client in order to interact with Kafka and/or Redis. In most client-server architectures, as long as the protocol and data serialization is supported on both ends then the programming languages do not need to match.
If you are looking to see how Redis and Kafka complement each other check this out -
1 Like