Redis Bulk Fetch of 5-10 MB From HMSET

Redis commands & data structures
1

0

Use Case: our data structure is like below:

tp1 “i1” : {object hash}, “i2” : {object hash}

tp2 “i3” : {object hash}, “i4” : {object hash}

tp1 and tp2 are hmset keys. we are referring as tp keys.

Each tp key can have 100-200 records in it. And each hash has a size of 1-1.5 KB.

Below is our implementation with spring data:

public Map<String, Map<String, T>> getAllMulti(List<String> keys) {
    long start = System.currentTimeMillis();
    log.info("Redis pipeline fetch started with keys size :{}", keys.size());
    Map<String, Map<String, T>> responseMap = new HashMap<>();
    if (CollectionUtils.isNotEmpty(keys)) {
        List<Object> resultSet = redisTemplate.executePipelined((RedisCallback<T>) connection -> {
            for (String key : keys) {
                connection.hGetAll(key.getBytes());
            }
            return null;
        });
        responseMap = IntStream.range(0, keys.size())
                .boxed()
                .collect(Collectors.toMap(keys::get, i -> (Map<String, T>) resultSet.get(i)));
    }

    long timeTaken = System.currentTimeMillis() - start;
    log.info("Time taken in redis pipeline fetch: {}", timeTaken);
    return responseMap;
}

Objective: Our objective is to load hashes of around 500-600 tp keys. We thought of using redis pipeline for this purpose. But as we are increasing the number of tp keys, the response time is increasing significantly. And it is not consistent also.

For response time improvement we have tried compression/messagePack, still no benefit.

One more solution we have tried, where we have partitioned our tpkeys into multiple partition and run the above implementation in parallel. Observation is if the number of tpkeys is small then the batch takes less time. if tpkeys size is increasing,time taken for the batch with same number of keys is increasing.

Any help/lead will be appreciated. Thanks

1 Like