Knowledge Vault

Floating-Point Arithmetic

3 min read

Floating-Point Arithmetic

IEEE 754 is the standard governing floating-point representation used by virtually every modern CPU. A 64-bit double-precision number encodes a sign bit, an 11-bit biased exponent, and a 52-bit mantissa (significand), representing values of the form (-1)^s × 1.mantissa × 2^(exponent-1023). This representation covers roughly ±1.8 × 10^308 with about 15–17 significant decimal digits of precision. Because the set of representable values is finite, most real numbers — including many simple decimals like 0.1 — cannot be represented exactly, and arithmetic operations introduce rounding error that accumulates in ways that violate familiar algebraic identities.

Key Concepts

Representation and Precision — The gap between adjacent representable values is not constant — it scales with magnitude. Near 1.0 the unit of least precision (ULP) is about 2.2 × 10^-16; near 10^15 the gap is ~1, meaning consecutive integers beyond 2^53 cannot all be represented as doubles. This is why 0.1 + 0.2 != 0.3 in most languages: both 0.1 and 0.2 are rounded to their nearest representable doubles, and the sum of these rounded values is not the rounded value of 0.3.

Special Values — IEEE 754 reserves bit patterns for three special values:

  • Infinity (+Inf, -Inf): produced by overflow or division by zero; propagates through arithmetic
  • NaN (Not a Number): produced by undefined operations like 0/0 or sqrt(-1); NaN is the only value not equal to itself
  • Subnormal numbers: small values near zero that trade precision for range, filling the gap between zero and the smallest normal number

Rounding Modes — IEEE 754 defines five rounding modes. The default is round-to-nearest-ties-to-even (banker’s rounding), which rounds ties to the nearest even mantissa bit, reducing statistical bias in long computations.

Catastrophic Cancellation — Subtracting two nearly equal numbers destroys significant digits. If a = 1.0000001 and b = 1.0000000, the difference has only one significant digit even if both operands had seven. Algorithms that repeatedly subtract nearly equal values (naïve summation of alternating series, certain matrix operations) can lose all precision.

Associativity Violations — Floating-point addition is not associative: (a + b) + c may not equal a + (b + c). Compilers with -ffast-math or equivalent flags may reorder operations for speed, silently changing numerical results. This matters for reproducibility in scientific and financial software.

Compensated Summation — The Kahan summation algorithm reduces accumulated rounding error in large sums from O(n·ε) to O(ε) (where ε is machine epsilon) by tracking a running correction term. It is significantly more accurate than naïve summation for large arrays of floating-point values.

Decimal Arithmetic — Binary floating-point is wrong for money. Financial software uses decimal floating-point (IEEE 754-2008 decimal formats, SQL DECIMAL, Python decimal.Decimal) or integer arithmetic in the smallest currency unit (cents, millicents) to avoid binary-to-decimal rounding surprises.

Common Pitfalls

  • Equality comparison of floating-point values with == — compare with an epsilon tolerance instead
  • Accumulating small increments (animation loops with t += 0.016) — use integer frame counts and derive floats from them
  • Mixing single and double precision in a pipeline and expecting no precision loss at conversion points
  • Assuming abs(a - b) < epsilon is a safe equality check for all magnitudes — use relative tolerance for large values

Best Practices

  • Use math.isclose() (Python) or equivalent for floating-point equality checks
  • Prefer integer arithmetic for counters, currency, and any quantity that must round-trip exactly
  • Sort values before summing to reduce cancellation (sum small values first)
  • Use long double or arbitrary-precision libraries when high accuracy is required in sensitive computations

See Also

Graph View

Backlinks