i am trying to implement a Data streaming application and since i am familiar with Django i want to use Django channels that work good with Redis. However my starting plan was to take data from a Kafka cluster. Is there a way to connect Redis to kafka and use it as a cashe for my application? Is Redis capable enough to replace Kafka and use it as my main message bus and send data to another database and other consumers? Any advice will be apreciated.
Thanks a lot!!
Hello, there is both a way to connect Redis with Kafka and a way to use Redis instead of Kafka to meet your goals.
To seamlessly connect Redis with Kafka you can leverage our Kafka Connect sink connector which can be downloaded from Redis Connector (Source and Sink) by Redis | Confluent Hub.
To replace Kafka as a publish-subscribe distributed message broker you can leverage the Redis Streams data structure which is a time-ordered in-memory log. You can think of each Redis Stream key as the equivalent to a Kafka Topic Partition including the ability to have multiple Consumer Groups listening onto it. There is a good tutorial about it here - Redis Streams tutorial | Redis
Let us know what you decide and if you have any further questions.
Best of luck, Allen