-
Notifications
You must be signed in to change notification settings - Fork 23
Description
While optimizing GeoHash conversions in Garnet, I came across small error in the McLoughlin's quantization approach that this library is seemingly also based on.
The statement
Since y is in the range [1,2], the largest power of two less than or equal to y is 1.0 and the exponent field will always be 1023.
is wrong for the upper-bound of the range. Exponent bias is
1023for[1.0, 2.0)and exponent for2.0flips over to1024. This means that90.0and180.0that are clamped to the range maximum2, have their signicand (and subsequently, the geohash integer repesentation) zeroed and dequantization will give equivalent output as-90.0and-180.0respectively.
Most straight-forward fix would probably be just guarding this corner-case and returning max 32-bit value for these cases seperately. I believe dequantization would then return closest value to 2.0.
I'd like to ask whether this behavior was already known and whether this actually applies to the rust implementation..
TLDR: Quantization makes 90.0, 180.0 => -90.0, -180.0. Do we care?