Max capacity of RedisBloom in a single key

Redis modules RedisBloom
1

Hi all,

I am trying to find out the max capacity of a RedisBloom stored in a single key.

Given that max value size is 512 MB in Redis, I was under the impression that byte count reported by BF.DEBUG cannot exceed 512 MB.

However, in my BF.RESERVE command trials (ee below) I am able to create bloom filter where BF.DEBUG shows 8GB or even 16GB.

  1. Is 512MB limit now changed to 16GB? I get an “ERR bad capacity” if my request exceeds 16 GB limit.

  2. Also, something is wrong with the capacity reported with 16 GB bloom. It reports capacity as only ~485 million where I requested 4 billion and got OK. Any idea why?

The capacity is supposed to be 4.78 billion per bloom filter mathematical calculation here (https://hur.st/bloomfilter/?n=&p=0.000001&m=16GiB&k=).

Thanks,

-Serkan

2

Hello Serkan,

Thank you for your excellent feedback. This is very interesting.

Indeed the limit for key size in Redis is 512MB. This limit is not on the internal storage but on the RESP protocol. To overcome this, RedisBloom chunks up large keys. So that should answer your first question.

About the size confusion, RedisBloom rounds the capacity to the next pow(2,x).

2B * 20 hashes / ln(2) = 2^31 bytes * 28 bits ~ 2^33 = 8589934592.

5 Billion fails as we check if it is larger than UINT32_MAX.

4 Billion probably fails b/c we round up and then keep some slack for the scalable filters. It is a bug indeed.

May I ask what do you plan on using the filter for?

Regards,

Ariel