bloommask

the question

Does datatrove's deduplication Bloom filter address the full m_bytes*8 bits it allocates, or does the hash reduction restrict it further?

what came out

datatrove reduces a hash to a bit index with AND against m_bytes (the byte count) instead of the bit count, so only 2^popcount(m_bytes) positions are reachable -- two for any power-of-two size. At m_bytes=2^20, k=7, 49 of 50 unique documents are dropped while the logged false-positive rate reads 2.8e-29.

→ read the code and re-run it