Can we set expiry for each document we add to redis index?

Hi,
Is there any function to set expiry for each document we add to redis index.

require_once ‘vendor/autoload.php’;

use Ehann\RediSearch\Index;

$bookIndex = new Index();

$bookIndex->addTextField(‘title’)

->addTextField(‘author’)

->addNumericField(‘price’)

->addNumericField(‘stock’)

->create();

$bookIndex->add([

new TextField(‘title’, ‘Tale of Two Cities’),

new TextField(‘author’, ‘Charles Dickens’),

new NumericField(‘price’, 9.99),

new NumericField(‘stock’, 231),

])->expiry(20); //Eg 20 seconds

Regards

namjith

At the moment - no.
RediSearch is unaware of expirations, and will not delete the documents from the index when they expire.

In the next version of redis support for expiration notifications to modules will be added, and then we can add support for it. But currently it’s a limitation of the module engine itself that prevents us from doing that.

Hi, when can we expect the expiration feature in RediSearch?

This wouldn’t work out the way you’d expect it to, because if a document is expired from Redis (e.g. in the form of a Redis hash) it won’t get deleted from RediSearch. What is the use case?
Mark Nunberg | Senior Software Engineer
Redis Labs - home of Redis

Email: mark@redislabs.com

Well, I expect the document in the RediSearch index to expire whenever the TTL is reached and it expires in the Redis db.

The use case is that we have auctions in our Redis database. Every auction has an end date and thus we want to have it expired on that end date. We are now building an auction search on top of our Redis database with RediSeach. This search should also take the auction end dates into consideration.

That’s an interesting use case, thanks.

So, IIUC, you’re using FT.ADDHASH? If so, would you consider using FT.ADD instead?

As for setting an expiry on a document, would you accept using something like FT.EXPIRE instead of Redis’?

We are currently still exploring the possibilities of RediSearch and in these tests we were using FT.ADD already, so no problem to do that.
FT.EXPIRE seems the right command to do this, so it would be nice if something like this could be added.

Gotcha - care to submit an https://github.com/RedisLabsModules/RediSearch/issues ?

There are three dimensions to deletion:

  1. Removing the actual contents of the document

  2. Removing the index entries related to the document (which is separate from the document)

  3. Omitting deleted documents from search results.

Implementing such a feature, (1) and (3) are fairly simple to do. (1) can be done because documents are just Redis hashes (at the moment). So expiring/deleting them is built into Redis. For (3), we can add an optional ‘expires’ timestamp into the document, and then compare it with the current time when processing results. Documents which are shown to be expired are then simply marked as deleted and then ignored (perhaps collected by gc).

Removing the index entries is done via garbage collection, which will not at the moment work the same way that Redis’ expiration does (but rather, how FT.DEL does). Since your use case is an Auction platform, lazy garbage collection should not be a problem.

  1. Removing them from storage/disk/ram/index
    Mark Nunberg | Senior Software Engineer
    Redis Labs - home of Redis

Email: mark@redislabs.com

Sure, no problem.

Its created: https://github.com/RedisLabsModules/RediSearch/issues/473

I don’t agree that lazy garbage collection should not be a problem.

Yes, if we only do a simple search we can probably work with the 1) and 3) solution. But if we start working with aggregations (for faceted search functionarlity), which depends on the index, the index should really be up to date at every moment. We don’t want to include expired documents in the aggregation count.

Garbage collection only operates on the physical removal of the index data from the given index. When an index is scanned and an entry is returned, that entry is checked to see if the document it refers to still exists, or hasn’t been marked as deleted. The marking of a document as deleted is done immediately via FT.DEL (and can be done implicitly when checking timestamps as well). The issue of lazy GC in this case is more one of resource consumption.
Mark Nunberg | Senior Software Engineer
Redis Labs - home of Redis

Email: mark@redislabs.com