I have a stream created by execute('XADD', 'tokeniser_to_matcher{%s}' % hashtag(), '*', 'sentence_key', f"{key}") out of sharded stream reader function.
The function above is 4th step in NLP pipeline, now I am writing 5th, which can potentially be StreamReader function as well. My understanding I will have to run collect() before I can process those streams on one shard using gb.run.
What is the best way to write and test StreamReader based function? Is any best practice or guideline?
@AlexMikhalev not sure I follow the question, you wrote a function that is part of a gears pipe that start with a stream reader? and you want to debug it?
Yes, I have a set of gear functions to process data, and it’s function number 5th in pipeline. Previous function uses and generates sharded streams. Re-processing data until that point quite straightforward, but time consuming, so I thought may be a better way: some form of test pattern/best practice - isolate Stream Reader as single component, collect all streams to single threat, or mock sharded streams somehow.
Sorry @AlexMikhalev but I still not follow the question. Can you give a more detailed explanation?
In general to debug a function in terms of how long each step takes you can use the profile execution option, but not sure this is what you meant (https://oss.redislabs.com/redisgears/configuration.html#profileexecutions)