Why is my Redis server loading data so slowly?

Redis administration
,
1

I’ve created a Redis Stack server in local via below command:

docker run -d --name redis-stack-rdb-aof -e REDIS_ARGS=“–requirepass redis --save 300 100 --stop-writes-on-bgsave-error no --appendonly yes --appendfsync always --no-appendfsync-on-rewrite no --auto-aof-rewrite-percentage 100 --auto-aof-rewrite-min-size 64mb --aof-use-rdb-preamble yes” -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

After server startup, I created a vector index and inserted 2M vectors in it.

But after restarting the server, I find the warmup time of the server is dramatically long. Time passed 1 hour and it is sill printing " LOADING Redis is loading the dataset in memory" when executing “dbsize”.
Below is the log after restarting:

2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 * Reading RDB preamble from AOF file…
2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 * Loading RDB produced by version 6.2.11
2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 * RDB age 52456 seconds
2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 * RDB memory usage when created 5487.78 Mb
2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 * RDB has an AOF tail
2023-04-20 13:20:26 9:M 20 Apr 2023 05:20:26.891 # creating vector index. Server memory limit: 32088838144B, required memory: 98300528B, available memory: 32087785128B

Is it normal in Redis or any mistakes I made?

2

Here comes the log for the warmup. It cost over 2 hours to complete the process. Does Redis need to re-build the ANN index from scratch each time it restarts?

2023-04-20 15_26_52-Window
2023-04-20 15_26_52-Window1232×311 30.7 KB

3

We’re working on enhancing the load time, it should be much faster on the up coming version.

5

Hi Korland,
From the logs above, my understanding is that existing Redis server needs to re-build the ANN index from scratch every time it restarts. Using RDB or AOF mode won’t have impact on this fact. Is my understanding correct? THX

6

You’re correct RDB only stores the data and the index definition, but the upcoming version will be able to utilize parallel indexing, so rebuilding the index will be able to be much faster.

1 Like
7

Hi Gkorland, may I know if the new version has been released to fix the slow index rebuilding issue? Thx!