Documenting and testing Redis stream data

Redis client libraries (Java, Python, JS, etc.)
1

We are making heavy use of Redis Streams for interservice communication and background processing (task queue) in our microservice(s).
The messages that are passed through Redis streams are expected to be of a certain format or ‘shape’, which is poorly or not documented. Services that produce messages are often maintained by different teams/developers to services that consume messages.
Other actions may also be expected when producing/consuming messages (e.g. setting some additional hashes and so on).
There is a clear need to formalize and document these interfaces and also to provide tools to teams to ensure that their implementation of a producer or consumer is correct.

How do you go about documenting interfaces based on Redis?
How do you test them?

2

Thanks for the question, jramm. This is a great point. You’re right there’s no built-in mechanism for managing message schemas.

One recommendation would be to limit the stream to a few fields:

  1. For a schema version / type
  2. For the serialized payload.

This way, you could use a serialization framework like Protocol Buffers to manage this across application stacks.

3

Thanks for the info. Protocol Buffers is one way to go that we are thinking of. I guess by adopting this though, we basically limit the stream to one field, that being the protocol buffer document (e.g. called ‘entity’). Currently we use multiple fields for each part of the message. Could there be any disadvantages to storing structured message data in a single field, when redis streams can provide such structuring for you otb?