Upsert memory usage redux.

[dsnow]

I have a use case where I batch upsert many documents once per day, and then additionally upsert throughout the day.

I’m experiencing unexpected memory usage after upserting, both after the batch (using pipelining) as well as incrementally through the day. Image is redislabs/redisearch:1.4.10, clients are redisearch-py 0.7.1 and jredisearch 0.22.0. Any help would be appreciated. Here is a little script to demonstrate:

import redis

import redisearch

if name == ‘main’:

r = redis.Redis()

rs = redisearch.Client(‘index’, conn=r)

try:

rs.create_index([redisearch.TextField(‘field’)])

except Exception:

pass

while True:

rs.add_document(‘doc’, replace=True, partial=True, field=‘1 2’)

print(r.info(section=‘memory’)[‘used_memory’], end=’\r’)

``

In the time that it took to write this up, the used_memory is up to 13829320 and stays after stopping the script and debug reloading.

The DBSIZE consistently shows 4 keys as expected. Output of KEYS:

1. “ft:index/2”

2. “ft:index/1”

3. “doc”

4. “idx:index”

``

INFO MEMORY after script has been stopped:

Memory

used_memory:13808448

used_memory_human:13.17M

used_memory_rss:120729600

used_memory_rss_human:115.14M

used_memory_peak:68664976

used_memory_peak_human:65.48M

used_memory_peak_perc:20.11%

used_memory_overhead:841246

used_memory_startup:791360

used_memory_dataset:12967202

used_memory_dataset_perc:99.62%

allocator_allocated:15296824

allocator_active:19308544

allocator_resident:84508672

total_system_memory:2095869952

total_system_memory_human:1.95G

used_memory_lua:37888

used_memory_lua_human:37.00K

used_memory_scripts:0

used_memory_scripts_human:0B

number_of_cached_scripts:0

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:1.26

allocator_frag_bytes:4011720

allocator_rss_ratio:4.38

allocator_rss_bytes:65200128

rss_overhead_ratio:1.43

rss_overhead_bytes:36220928

mem_fragmentation_ratio:8.77

mem_fragmentation_bytes:106962176

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:49694

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0

INFO MEMORY after DEBUG RELOAD shows the same:

Memory

used_memory:13808496

used_memory_human:13.17M

used_memory_rss:131489792

used_memory_rss_human:125.40M

used_memory_peak:68664976

used_memory_peak_human:65.48M

used_memory_peak_perc:20.11%

used_memory_overhead:841278

used_memory_startup:791360

used_memory_dataset:12967218

used_memory_dataset_perc:99.62%

allocator_allocated:15586288

allocator_active:19595264

allocator_resident:97402880

total_system_memory:2095869952

total_system_memory_human:1.95G

used_memory_lua:37888

used_memory_lua_human:37.00K

used_memory_scripts:0

used_memory_scripts_human:0B

number_of_cached_scripts:0

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:1.26

allocator_frag_bytes:4008976

allocator_rss_ratio:4.97

allocator_rss_bytes:77807616

rss_overhead_ratio:1.35

rss_overhead_bytes:34086912

mem_fragmentation_ratio:9.55

mem_fragmentation_bytes:117722312

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:49694

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0

INFO MEMORY after FLUSHDB shows most but not quite all memory released:

Memory

used_memory:1220400

used_memory_human:1.16M

used_memory_rss:131559424

used_memory_rss_human:125.46M

used_memory_peak:68664976

used_memory_peak_human:65.48M

used_memory_peak_perc:1.78%

used_memory_overhead:841054

used_memory_startup:791360

used_memory_dataset:379346

used_memory_dataset_perc:88.42%

allocator_allocated:2672768

allocator_active:6660096

allocator_resident:97402880

total_system_memory:2095869952

total_system_memory_human:1.95G

used_memory_lua:37888

used_memory_lua_human:37.00K

used_memory_scripts:0

used_memory_scripts_human:0B

number_of_cached_scripts:0

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

allocator_frag_ratio:2.49

allocator_frag_bytes:3987328

allocator_rss_ratio:14.62

allocator_rss_bytes:90742784

rss_overhead_ratio:1.35

rss_overhead_bytes:34156544

mem_fragmentation_ratio:111.55

mem_fragmentation_bytes:130380048

mem_not_counted_for_evict:0

mem_replication_backlog:0

mem_clients_slaves:0

mem_clients_normal:49694

mem_aof_buffer:0

mem_allocator:jemalloc-5.1.0

active_defrag_running:0

lazyfree_pending_objects:0

[meirsh]

RediSearch is using a garbage collector to clean the deleted document from the inverted index. If there are not to many delete/update operations then the default GC is probably good enough but when the delete/update is intensive then the GC can not clean the garbage fast enough and you will see the memory keeps growing. For such use-cases we have a much faster garbage collector called FORK_GC. The FORK_GC is able to collect the garbage much faster then the default GC.

For further reading about FORK GC and how to enable it:
https://medium.com/@meir.spielrein/how-we-increased-garbage-collection-performance-with-redisearch-1-4-1-73cdb8c579f4
https://oss.redislabs.com/redisearch/Configuring.html

[dsnow]

Ok, I’m familiar with the FORK_GC, and it doesn’t seem to help. I have a couple questions/points:

First: I would expect if the GC couldn’t keep up, after stopping the upserting process it would catch up at that time, but it doesn’t. Whatever memory is used after the process stops is kept by Redis - and indeed still stored in the indices because that memory usage persists after a DEBUG RELOAD. Would that be expected behavior?

Second: The memory usage doesn’t go steadily upward - it seems to plateau, with both default and fork GC’s keeping up, then after some time (maybe 10-15 minutes) jumps by a couple megabytes, and the process repeats. It seems to me that GC speed inadequacy would manifest as a linear progression, but maybe not.

Lastly: Running the script above, slightly amended so as to write to two containers simultaneously, one running GC_POLICY FORK and the other default GC shows that there is no improvement in memory usage. Arguably the volume of upserts could be too high for the fork to keep up as well, but I would expect that it would fare better. Thoughts?

Thanks again for your help.

[meirsh]

First: You are checking on a very small indexes, I guess that most of the memory is taken for the doctable and not for the inverted indexes them-self. You can confirm this assumption by setting the MAXDOCTABLESIZE parameter to something very low (like the number of document for example) and then see the memory usage.

Second: Again I think its the doctable which does not decrease but only increases until reaching MAXDOCTABLESIZE

Lastly: Still think its the doctable but notice the bellow comment

I would like to know what version are you running with, we recently added to FORK_GC the capability to not just cleaning the inverted indexes but also squash them (the default GC does not do it). When inverted index get very large its not enough to just clean the garbage but it also important to squash the inverted index to make it optimize regarding the memory usage.

So to sum up, I would advise to increase the size of the index (a few Gb and not couple of Mb), set MAXDOCTABLESIZE to the number of documents, enable FORK_GC and use the latest version. If you still see memory usage increasing then its sounds like a leak and we need to fix it.

[dsnow]

Dropping the MAXDOCTABLESIZE to 1 resolved the problem on RS 1.4.10 - even using default GC and the small index, so that’s great. Thank you! The script that I supplied was just an effort to demonstrate what we’re doing on a much larger scale.

So in the real environment, I have a database that is currently ~10GB. We do batch upserting of about 2.8M docs daily and well as streaming upserts. Of interest is that after the batch job, the DB size climbs by ~.3%, basically every day, but only at most a few thousand of those docs are new, the rest are just updates. The new docs do not remotely account for the additional memory consumption. I believe that it’s the same problem with MAXDOCTABLESIZE, which for that DB is currently set at 5M. FWIW the max_doc_id for that DB is well past 5M but the growth continues.

Given that I expect the number of docs to continuously grow (albeit slowly), is there a way to allow for this growth but not allow the doc table to grow out of control? Does the doc table need one bucket per document, or can I have multiple docs per bucket?

[dsnow]

FYI I was able to replicate the issue with MAXDOCTABLESIZE set to the number of docs and GC FORK. As before, memory usage does not recover after stopping the upserting. I have attached a small script which repeatedly upserts the same set of 100k docs and replicates the issue. Do you have any other suggestions?

docker run -d -p 6379:6379 redislabs/redisearch:1.4.10 sh -c “redis-server --loadmodule /usr/lib/redis/modules/redisearch.so GC_POLICY FORK MAXDOCTABLESIZE 100000”

import random

import string

import redis

import redisearch

def get_str(num, space=False):

ltrs = string.ascii_letters

if space:

    ltrs += ' '

return ''.join([random.choice(ltrs) for _ in range(num)])

def gen_d(size):

print(f'generating dict with {size} keys...', end='', flush=True)

d = {

    get_str(8): {

        f'field_{x}': get_str(16, space=True) for x in range(15)

    }

    for i in range(size)

}

print('done.')

return d

if name == ‘main’:

r = redis.Redis()

rs = redisearch.Client('index', conn=r)

try:

    rs.create_index([redisearch.TextField(f'field_{x}') for x in range(15)]) # noqa

except Exception:

    pass

D = gen_d(100000)

while True:

    print('inserting into redis...', end='\r', flush=True)

    for doc, d in D.items():

        rs.add_document(doc, replace=True, partial=True, **d)

    print('used memory: {}'.format(str(r.info(section='memory')['used_memory']))) # noqa

``

Output:

generating dict with 100000 keys…done.

used memory: 403349608.

used memory: 419102464.

used memory: 449529024.

inserting into redis…

[meirsh]

Thanks for providing the script, I will take a look at let you know ASAP.

[meirsh]

To understand why memory increase you need to understand how the inverted index looks like. Basically its an array of blocks such that each block contains 100 docids. The GC scanning all the inverted indexes and if it see a deleted document it fix the relevant block. One constrains about FORK_GC is that it does not scan the last block (this is for implementation simplicity and we think its not that significant not to clean at most 100 deleted documents per inverted index).

In your test you create many inverted index (cause the terms in each document is random) each inverted index is very small (contains one document for each test iteration). So we ends up with many inverted index with one block each and as I said before the last block (in the case the first block is also the last block cause we have only one block) are not get cleaned by the FORK_GC. So in each update iteration you see the memory keeps growing. After 100 update iterations, a new block will be created for each inverted index and the FORKGC will clean the first block (you will see the memory usage decrease dramatically).

I tested it with a much small index (100 docs) and the memory is increasing for 100 update iterations and then decrease.

To sum up, wait until you finish 100 update iterations and you will see the memory usage decrease dramatically.

[dsnow]

This makes sense now, thank you for the explanation! Is there a way to configure the block size to force a clean-up after each update iteration? It looks like from your documentation that GC_SCANSIZE (btw it looks like this should be updated to GCSCANSIZE) may be related, but the doc is not very specific and lowering that value appears to have no effect. If not, is this a reasonable configuration option to implement, and shall I put in a feature request? I have many docs that have (partly) unique field values, and the only other thing I could think of to stop the large memory variance would be doing a diff on each doc before each update and then updating only the changed fields, which would be slow.

Additionally, I would like to be able to use geofields to do geo queries on moving objects - it seems like this is not possible unless the last block of an index is also cleaned up - provided that the geo indexes work in a similiar fashion - because the index key would be a likely unique value that would not have additional docids added unless that exact location was visited again. Would this be an accurate understanding?

[dsnow]

I answered my own question, I see that this is hardcoded: https://github.com/RediSearch/RediSearch/blob/b4116f43705c50cc362085017e53315c56c8082c/src/inverted_index.c#L17

I would like to know though if its worthwhile to make this configurable for cases like mine or if there is too large a performance penalty? Or alternatively clean up the last block, but after cursory examination it looks like the geo indexes do not work the same way, so I’m guessing that wouldn’t help with the moving object geo queries and so maybe it would just be better to provide option #1.

[meirsh]

David I just want to clarify that the fact that we do not delete the last block does not mean that we will return deleted documents. We do mark the documents as deleted and ignore them on queries. Regarding the GC_SCANSIZE it has not effect of forkgc, I will document it (thanks for pointing this out).

Whether or not it is reasonable request to adjust the block size? I am not sure, your use-case is very rare, in most of the use-cases that requires Free Text Search the inverted indexes are very large and not cleaning the last block really makes no difference. It might be useful for you to use the UPDATE IF, this way you can specify a condition and only if this condition is true then the document will be updated (see documentation for more details https://oss.redislabs.com/redisearch/Commands.html#ftadd).

Regarding the geo field, it works differently and cleaned immediately when you delete the document (no GC on geo fields).