I have to make a query where both dataset and result sets are in the range of few hundreds. Dataset consists JSON documents which contain some user defined fields and we need to filter documents based upon these fields. Clearly, we cannot index our documents on these fields.
We are currently using filter in FT.AGGREGATE query, loading these fields using LOAD and after that we are using FILTER to filter those results.
I read that LOAD hurts performance of the system, so can we use filtering in FT.SEARCH for non-indexed fields. Or is it better to process these results in Lua itself because anyway filtering happens once we fetch the results using query in FT.AGGREGATE? Or is there any better way to filter the results.
PS:- result set is a part of transaction and will be modified.