ftSearch on vector embeddings returns vector_score of -nan

Hello everyone, I am using Redis to store vector embeddings, and would like to do a vector similarity search using the JedisClient. I have 1536 dimension embeddings , and I am storing the same embeddings in a HNSW field, as well as in a FLAT vector field. When I attempt to search for matches to this embedding using jedis.ftSearch(), I seem to get irrelevant documents with a vector_score of -nan Is there anything glaringly obvious that I am missing here?

I am using 4.3.0 from GitHub - redis/jedis: Redis Java client designed for performance and ease of use.
Index creation:

          Schema schema = new Schema().addHNSWVectorField("hnsw_field", vectorHnswAttributes).addTagField("raw_text").addFlatVectorField("flat_field",vectorFlatAttributes);
          uJedis.ftCreate(INDEX_NAME, IndexOptions.defaultOptions(), schema);

Search:

         int K = 1; // Get K nearest neighbors
         Query q_hnsw = new Query("*=>[KNN $K @hnsw_field $BLOB AS my_vector_score]").addParam("K", K).addParam("BLOB",floatToByte(vec))
                .setSortBy("my_vector_score", false)
                .limit(0,K).returnFields("my_vector_score","raw_text").dialect(2);

        List<Document> docs_hnsw = uJedis.ftSearch(INDEX_NAME, q_hnsw).getDocuments();

Found this from a vector-similarity-search channel on discord. Sharing in case others run into this…

Just following up on my earlier post in case some other person runs into the same issue using the Java Redis client while trying to do vector similarity search. If you are indexing a redis vector field (i.e., FLAT or HNSW), and you need to pass your float32 array data as bytes using HSET , you need to use little endian order when writing the bytes from the float32 values. In Java system libraries, the default is big-endian, which is not compatible with RediSearch. Here is the updated code I’m using to convert float array data to byte array. Now, my vector searches are working properly.

public static byte[] floatArrayToBytesLittleEndianOrder(float[] vector) throws IOException {
        ByteArrayOutputStream bas = new ByteArrayOutputStream();
        DataOutputStream dos = new DataOutputStream(bas);
        for (float f : vector) {
            byte[] bytes = getBytesLittleEndianOrder(f);
            dos.write(bytes);
        }
        return bas.toByteArray();
    }

    private static byte[] getBytesLittleEndianOrder(float f) {
        int intBits =  Float.floatToIntBits(f);
        return new byte[]{(byte) (intBits), (byte) (intBits >> 8), (byte) (intBits >> 16), (byte) (intBits >> 24)};
    }
1 Like