Date: Thu, 13 Dec 2018 03:25:30 +0000 From: Taylor R Campbell campbell+tor-dev@mumble.net
Binary80 arithmetic tickled a problem in the lerp used for binning -- [...]
Correction: while making the lerp less naive addresses this problem, it also arises only when binary80 arithmetic and binary64 arithmetic are mixed, which you get on 32-bit x86 in C double with the x87 unit in binary80 mode so (a) double is binary64, but (b) intermediate expressions are evaluated in binary80.
The attached program shows this, by putting either one (bad) or two (good) intermediate expressions in double variables. On x86, if you compile it with -mfpmath=387 (default on 32-bit), you'll see the bad result, a negative answer; if you compile it with -mfpmath=sse (default on 64-bit), you'll see only good results, zero. Convert everything to use long double (and %Le) instead so that all the arithmetic and intermediate quantities are binary80, and it's fine.
% cc -o loser -mfpmath=387 loser.c && ./loser bad -2.45760000000000000e+04 good 0.00000000000000000e+00 % cc -o loser -mfpmath=sse loser.c && ./loser bad 0.00000000000000000e+00 good 0.00000000000000000e+00
(This is why I don't like x87 and the automagic evaluation of expressions in higher precision...)