Extend hash length

[michaelmckinsey1]

Summary

Previous analysis from #53 (June '23) aimed to avoid hash collisions for up to O(1000) profiles, since the bottleneck was reader performance. With improvements to reader performance, such as #170 and #185 (June '24) we are able to read O(10000) profiles in reasonable time. Our hashes in develop are length 8-10 digits in length ( ceil(log_10(16^n - 1)) = ceil(9.6) = 10 ). Results in #53 suggest probable hash collisions around O(100000) at 10 digits.

Increasing truncated hex length to 11 results in hashes that are 11-14 digits in length. Only negative implication of extending hash length is longer hashes are more inconvenient for data analysis.

Analysis

KDE plots when truncating the MD5 hash before converting to integer. 1000 trials, sample size is the number of profiles, where each "profile" is a randomly generated unique string of length 32. Empty plot indicates no collisions (we can expect collision rate to be negligible).

hash length = 8 (develop)

hash length = 9

hash length = 10

hash length = 11

All of these collision rates are pretty low, but we can see that collisions are possible at O(10000) with a hash length of 8. A hash length of 10 makes collisions negligible up to ~50000 profiles and length 11 up to ~200,000 profiles. Although these are estimates bumping up the hash length makes sense now that we can read more files into Thicket.