Time series rules dont update value

diob1 · March 17, 2023, 4:10pm

Hi there,

I have created a TS in redis with some aggregation rules. However the rules dont get updated, they always come as empty list.

TS.MGET “LATEST” “FILTER” “symbol=btcusdt”

1. “btcusdt”
2. (empty list or set)
3. 1. “1679061819032”
  2. “26555.09”
1. “btcusdt:1800000”
2. (empty list or set)
3. (empty list or set)
1. “btcusdt:300000”
2. (empty list or set)
3. (empty list or set)
1. “btcusdt:3600000”
2. (empty list or set)
3. (empty list or set)

–
TS.INFO btcusdt

“totalSamples”
“279”
“memoryUsage”
“8664”
“firstTimestamp”
“1679061816361”
“lastTimestamp”
“1679061820124”
“retentionTime”
“604800000”
“chunkCount”
“2”
“chunkSize”
“4096”
“chunkType”
“uncompressed”
“duplicatePolicy”
“null”
“labels”
1. 1. “exchange”
2. “binance”
1. “symbol”
2. “btcusdt”
“sourceKey”
“null”
“rules”
1. 1. “btcusdt:60000”
2. “60000”
3. “AVG”
1. “btcusdt:300000”
2. “300000”
3. “AVG”
1. “btcusdt:900000”
2. “900000”
3. “AVG”
1. “btcusdt:1800000”
2. “1800000”
3. “AVG”
1. “btcusdt:3600000”
2. “3600000”
3. “AVG”
1. “btcusdt:86400000”
2. “86400000”
3. “AVG”

LiorKogan · March 19, 2023, 8:20am

Did you create the rules before inserting the samples? Note that only new samples that are added into the source series after the creation of the rule will be aggregated.

Can you please provide a minimal set of commands that reproduce this problem, starting with an empty database, on v1.8.9?

diob1 · March 27, 2023, 4:01pm

I tried to create rules before but no success. Using v 1.6.17 from redislabs.

LiorKogan · March 30, 2023, 7:16am

Please provide a minimal set of commands that demonstrates the problem you are facing (starting with an empty database).

diob1 · March 30, 2023, 11:52am

First I create the TS for sensor1 and sensor2

08:43:29.363 [0 172.17.0.1:42662] info
08:43:30.920 [0 172.17.0.1:34770] TS.CREATE sensor1 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.921 [0 172.17.0.1:34770] TS.CREATE sensor1:60000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.923 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:60000 AGGREGATION AVG 60000
08:43:30.924 [0 172.17.0.1:34770] TS.CREATE sensor1:300000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.924 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:300000 AGGREGATION AVG 300000
08:43:30.925 [0 172.17.0.1:34770] TS.CREATE sensor1:900000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.926 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:900000 AGGREGATION AVG 900000
08:43:30.927 [0 172.17.0.1:34770] TS.CREATE sensor1:1800000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.928 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:1800000 AGGREGATION AVG 1800000
08:43:30.929 [0 172.17.0.1:34770] TS.CREATE sensor1:3600000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.930 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:3600000 AGGREGATION AVG 3600000
08:43:30.931 [0 172.17.0.1:34770] TS.CREATE sensor1:86400000 DUPLICATE_POLICY block RETENTION 604800000 LABELS symbol sensor1
08:43:30.932 [0 172.17.0.1:34770] TS.CREATERULE sensor1 sensor1:86400000 AGGREGATION AVG 86400000

After that, data was added to two time-series “sensor1” and “sensor2” and their latest values were queried using TS.MGET LATEST and filtered by symbol:

08:47:06.804 [0 172.17.0.1:53102] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.804 [0 172.17.0.1:53098] TS.ADD sensor2 1680176826 1769.41 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor2 timestamp 1680044959
08:47:06.806 [0 172.17.0.1:53120] TS.ADD sensor1 1680176826 27193.91 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor1 timestamp 1680044960
08:47:06.806 [0 172.17.0.1:53106] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.808 [0 172.17.0.1:53132] TS.ADD sensor2 1680176826 1774.58 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS timestamp 1680044962 device nice3 symbol sensor2
08:47:06.808 [0 172.17.0.1:53124] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.810 [0 172.17.0.1:53162] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.810 [0 172.17.0.1:53146] TS.ADD sensor2 1680176826 1769.41 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor2 timestamp 1680044965
08:47:06.811 [0 172.17.0.1:53184] TS.ADD sensor1 1680176826 27251.9 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice2 symbol sensor1 timestamp 1680044966
08:47:06.811 [0 172.17.0.1:53176] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.813 [0 172.17.0.1:53200] TS.ADD sensor1 1680176826 27193.9 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor1 timestamp 1680044967
08:47:06.813 [0 172.17.0.1:53192] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.814 [0 172.17.0.1:53220] TS.ADD sensor2 1680176826 1774.2 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS timestamp 1680044968 device nice2 symbol sensor2
08:47:06.814 [0 172.17.0.1:53214] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.817 [0 172.17.0.1:53242] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.822 [0 172.17.0.1:53226] TS.ADD sensor1 1680176826 27251.89 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice2 symbol sensor1 timestamp 1680044970
08:47:06.842 [0 172.17.0.1:53244] TS.ADD sensor2 1680176826 1773.88 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice2 symbol sensor2 timestamp 1680044971
08:47:06.842 [0 172.17.0.1:53252] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.845 [0 172.17.0.1:53268] TS.ADD sensor1 1680176826 27193.05 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor1 timestamp 1680044971
08:47:06.845 [0 172.17.0.1:53266] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.847 [0 172.17.0.1:53280] TS.ADD sensor2 1680176826 1774.58 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS symbol sensor2 timestamp 1680044974 device nice3
08:47:06.847 [0 172.17.0.1:53272] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.848 [0 172.17.0.1:53310] TS.ADD sensor1 1680176826 27190.77 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS symbol sensor1 timestamp 1680044975 device nice1
08:47:06.849 [0 172.17.0.1:53296] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.850 [0 172.17.0.1:53330] TS.ADD sensor2 1680176826 1769.24 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor2 timestamp 1680044979
08:47:06.850 [0 172.17.0.1:53322] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.852 [0 172.17.0.1:53344] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.852 [0 172.17.0.1:53332] TS.ADD sensor1 1680176826 27250.01 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice2 symbol sensor1 timestamp 1680044980
08:47:06.853 [0 172.17.0.1:53362] TS.ADD sensor1 1680176826 27247.8 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice3 symbol sensor1 timestamp 1680044981
08:47:06.853 [0 172.17.0.1:53358] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.855 [0 172.17.0.1:53370] TS.ADD sensor2 1680176826 1774.01 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice2 symbol sensor2 timestamp 1680044983
08:47:06.855 [0 172.17.0.1:53366] TS.MGET LATEST FILTER symbol=sensor1
08:47:06.857 [0 172.17.0.1:53398] TS.ADD sensor2 1680176826 1769.25 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor2 timestamp 1680044984
08:47:06.857 [0 172.17.0.1:53382] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.859 [0 172.17.0.1:53426] TS.MGET LATEST FILTER symbol=sensor2
08:47:06.859 [0 172.17.0.1:53414] TS.ADD sensor1 1680176826 27190.77 ON_DUPLICATE block UNCOMPRESSED RETENTION 604800000 LABELS device nice1 symbol sensor1 timestamp 1680044985

However the TS created for the rules dont populate its data.

sensor1:60000 total samples is always 0
sensor1:* total samples is always

LiorKogan · March 30, 2023, 3:55pm

I can see that you are adding samples with identical timestamp: 1680176826
This means that the compacted series has a single bucket which is still open.
The latest, possible open bucket of a compacted series is not reported.

Let me explain this more carefully:
Till version 1.8, when a time series is a compaction, TS.GET, TS.MGET, TS.RANGE, TS.REVRANGE, TS.MRANGE, and TS.MREVRANGE did not report the compacted value of the latest bucket. The reason is that the data in the latest bucket of a compaction is still partial. A bucket is ‘closed’ and compacted only upon arrival of data that ‘opens’ a ‘new latest’ bucket.

There are use cases, however, where the compaction of the latest bucket should be retrieved as well. For example, a user may want to receive the count of events since the start of the decade, and the retention period for raw data is only one month. Till version 1.8, the user would have to run two queries - one on a compaction and one on the latest raw data, and then sum the results. Since version 1.8, by specifying LATEST, it is possible to retrieve the latest (possibly partial) bucket as well.

To report aggregations for the latest bucket, use the new optional LATEST flag to TS.GET, TS.MGET, TS.RANGE, TS.REVRANGE, TS.MRANGE, and TS.MREVRANGE.

diob1 · March 30, 2023, 4:07pm

Not sure what I have to do

LiorKogan · March 30, 2023, 4:19pm

With the RedisTimeSeries version installed on your cluster (1.6), you can verify that compactions works as expected by creating a sample that closes the previous bucket.
For example, with your data, try adding another sample with a timestamp >7 days than the previous one. This will close the current bucket and will open a new bucket, and you will be able to see the compacted value of the previous bucket.

Once your cluster is upgraded to 1.8 you’ll be able to simply add the LATEST flag to existing commands and retrieve the compacted value for the latest (still open) bucket as well.

diob1 · March 30, 2023, 4:35pm

Thank you @LiorKogan