What a number actually is: two's complement, floating point, and the price of quantization

A number is an idea. A numeral is how you write that idea down. “5”, “V”, “101”, and “five” are four different numerals for the same number. Binary is just another numeral system, one that happens to use two symbols instead of ten, and it wins for a boring, physical reason: a computer is built from billions of switches, and a switch has exactly two reliable states, off and on. Ten voltage levels would mean ten ways for noise to flip a digit into the wrong value. Two symbols is what survives being built out of switches.

Reading a binary number is the same place-value idea as decimal, just with a different base:

Binary:    1  0  1  1
Position:  3  2  1  0   (counting from the right, starting at 0)
Value:    2³ 2² 2¹ 2⁰  =  8  4  2  1

Multiply each bit by its position’s weight and sum: $1{\times}8 + 0{\times}4 + 1{\times}2 + 1{\times}1 = 11$. Going the other way, decimal to binary, is repeated division by 2, reading the remainders bottom-to-top: $11 \div 2 = 5$ r$1$, $5 \div 2 = 2$ r$1$, $2 \div 2 = 1$ r$0$, $1 \div 2 = 0$ r$1$, giving 1011.

Two’s complement: making subtraction disappear#

Negative numbers are where a naive design goes wrong. Store a sign bit separately, the way “+11” and “-11” work in English, and you need different circuit logic for addition than for subtraction, plus you end up with two representations of zero, $+0$ and $-0$. Two’s complement avoids all of it by redefining negative numbers so that ordinary addition just works, no special-case circuitry required.

To negate a number: flip every bit, then add 1.

 5 in 4-bit binary:  0101
 flip all bits:      1010
 add 1:               + 1
 result (-5):        1011

Check it by adding: $0101 + 1011 = 1\,0000$. Drop the 5th bit, it overflowed a 4-bit register, and you’re left with $0000 = 0$. Five plus negative five is zero, using the exact same addition circuit that adds two positive numbers.

Formally, for an $n$-bit signed number, the top bit’s place value simply flips sign:

$V = -b_{n-1} \cdot 2^{n-1} + \sum_{i=0}^{n-2} b_i \cdot 2^i$

Nothing about this is a flag or a special case, the most significant position just counts backward. One consequence worth internalizing: the range is asymmetric. An 8-bit signed integer runs from $-128$ to $127$, not $-128$ to $128$, because $+128$ has no representation. This asymmetry is a real, recurring source of bugs: abs(-128) in an 8-bit signed system has nowhere to go.

Picture a sensor-fusion system that stores steering-angle deltas as 8-bit signed integers to save bandwidth. A late correction term pushes the running sum to exactly $+128$. There’s no such value, so it silently wraps to $-128$, and whatever consumes that number now reads a hard turn in the opposite direction from what was intended. Signed-integer wraparound is a well-documented category of embedded-systems bug precisely because it fails silently instead of crashing.

def twos_complement_negate(bits: str) -> str:
    flipped = ''.join('1' if b == '0' else '0' for b in bits)
    flipped_list = list(flipped)
    carry = 1
    for i in range(len(flipped_list) - 1, -1, -1):
        if carry == 0:
            break
        if flipped_list[i] == '1':
            flipped_list[i] = '0'    # 1 + 1 = 0, carry stays 1
        else:
            flipped_list[i] = '1'    # 0 + 1 = 1, carry becomes 0
            carry = 0
    return ''.join(flipped_list)

neg5 = twos_complement_negate('00000101')
total = (int(neg5, 2) + int('00000011', 2)) & 0xFF   # mask to 8 bits, simulate real hardware
print(format(total, '08b'))   # 11111110  ->  decodes to -2

The & 0xFF is doing the work a real 8-bit register does for free: Python integers are arbitrary-precision and never overflow on their own, so the mask is what forces the simulation to behave like actual hardware.

Floating point: scientific notation in binary#

Integers can’t hold $3.14$, or $0.001$, or the mass of the sun in kilograms, without wasting enormous space. Floating point solves this the same way scientific notation does: instead of storing a number directly, store a sign, an exponent (how big or small), and a mantissa (the precise digits). $6.02 \times 10^{23}$ instead of 24 digits written out.

An IEEE 754 float lays out as [sign][exponent][mantissa], and for a normal (non-zero) value:

$V = (-1)^S \times 1.M \times 2^{(E - \text{bias})}$

$S$ is the sign bit. $E$ is the stored exponent field, read as an unsigned integer, then shifted by a fixed bias so it can represent negative exponents without needing its own sign bit. $M$ is the stored mantissa, interpreted as a fraction, and the leading 1. in front of it is never stored: any nonzero binary number can be written as $1.xxxxx \times 2^e$, the leading digit is always 1, so it’s free precision that costs zero bits.

Worked example: encode $-6.5$. Sign is 1 (negative). $6.5 = 110.1_2$. Normalized: $1.101 \times 2^2$, so the mantissa bits are 101 and the actual exponent is 2. FP32’s bias is 127, so the stored exponent is $2 + 127 = 129 = \texttt{10000001}_2$. Full encoding: 1 10000001 10100000000000000000000. Decoding it back: mantissa 101 gives $1.101_2 = 1.625$, exponent gives $2^{129-127} = 4$, and $-1 \times 1.625 \times 4 = -6.5$. It round-trips exactly.

import struct

def float_to_ieee754_bits(f: float) -> str:
    packed = struct.pack('>f', f)
    bits = ''.join(f'{byte:08b}' for byte in packed)
    return f"{bits[0]} {bits[1:9]} {bits[9:]}"   # sign | exponent | mantissa

print(float_to_ieee754_bits(-6.5))
# 1 10000001 10100000000000000000000

print(0.1 + 0.2)                  # 0.30000000000000004

0.1 has no finite binary representation, the same way $1/3$ has no finite decimal one, so it’s stored as the nearest representable value, and that tiny rounding error survives the addition. This isn’t a bug, it’s the direct consequence of the encoding above.

The number of exponent bits versus mantissa bits is a deliberate trade-off, not an accident, and it’s worth naming precisely because it explains why the formats below exist at all: more exponent bits buys a wider range of magnitudes at the cost of coarser gaps between representable values; more mantissa bits buys finer precision at the cost of hitting overflow or underflow sooner.

Format	Bits (sign / exp / mantissa)	Range	Precision	Primary use
FP32	1 / 8 / 23	Very wide	High, roughly 7 decimal digits	Training math, the “master copy” of weights
FP16	1 / 5 / 10	Narrower, can overflow to infinity	Medium	Older mixed-precision training
BF16	1 / 8 / 7	Same as FP32	Lower than FP16	Default for training and full-quality inference on modern GPUs
FP8 (E4M3 / E5M2)	1 / 4 / 3 or 1 / 5 / 2	Narrow	Low	High-throughput inference on recent hardware

BF16 cuts bits from the mantissa, not the exponent, so it keeps FP32’s full range while giving up precision. FP16 instead shrinks the exponent to 5 bits to fit the same 16-bit budget while keeping more mantissa, and that’s exactly why FP16 training runs are more prone to gradients silently overflowing to infinity or underflowing to zero: range, not precision, turned out to be the thing large models actually need most, which is why BF16 displaced FP16 as the default despite having less mantissa precision.

Quantization: paying for less precision with more error#

Quantization takes a number stored with many bits and stores it with fewer, accepting error to save memory and bandwidth. It’s writing someone’s location as “Boston” instead of exact coordinates: cheaper to store, and usually good enough. It is also not optional at scale. Running a 176-billion-parameter model at full FP32 precision needs multiple top-tier GPUs just to hold the weights; quantization is the entire reason running large models on a single consumer GPU is possible at all, and at frontier-lab scale it’s the difference between serving a model to millions of people profitably and not being able to serve it.

The core operation, symmetric linear quantization to $b$ bits:

$s = \frac{\max(|x|)}{2^{b-1}-1}, \qquad q = \text{round}\left(\frac{x}{s}\right), \qquad \hat{x} = q \times s$

$s$ is the scale, chosen from the largest magnitude in the tensor. $q$ is what actually gets stored. $\hat{x}$ is the reconstructed approximation used at inference time, and $\hat{x} \ne x$ in general: that gap is the quantization error, bounded by half the scale, $|\epsilon| \le s/2$, the same way rounding to the nearest integer has a max error of $0.5$.

That $\max(|x|)$ term is where quantization quietly breaks. Picture a tensor where 99.9% of values are small and a handful are enormous, which is exactly the shape real transformer activations tend to take. One global scale has to accommodate the huge values, which forces $s$ up, which crushes the small majority into a handful of representable buckets.

Worked example. Quantize $x = [0.02, -0.01, 0.03, 4.8]$ to 8-bit signed. $\max(|x|) = 4.8$, so $s = 4.8 / 127 \approx 0.0378$.

q(0.02)  = round(0.02 / 0.0378)  = round(0.529)  = 1   ->  dequantized: 0.0378  (89% error)
q(-0.01) = round(-0.01 / 0.0378) = round(-0.265) = 0   ->  dequantized: 0       (100% error, vanished)
q(0.03)  = round(0.03 / 0.0378)  = round(0.794)  = 1   ->  dequantized: 0.0378  (26% error)
q(4.8)   = round(4.8 / 0.0378)   = 127                 ->  dequantized: 4.8     (~0% error)

The outlier reconstructs almost perfectly. It did so by destroying every small value’s precision. That’s the whole mechanism behind “naive INT8 quantization silently wrecks model quality” reduced to four numbers you can check by hand.

The fix is either a better format or a smarter scale. FP8 keeps a floating-point exponent, so unlike INT8’s fixed linear grid it packs more representable values near zero and fewer far from it, which happens to match how weight and activation distributions actually look: mostly small, occasionally huge. If you’re stuck with INT8 anyway, the practical fix is mixed-precision decomposition: detect the small fraction of outlier channels, route just those through FP16, and quantize everything else with a much smaller, more honest scale, instead of letting one outlier set the scale for an entire tensor.

import numpy as np

def quantize_int8_symmetric(x: np.ndarray):
    scale = np.max(np.abs(x)) / 127.0
    q = np.clip(np.round(x / scale), -127, 127).astype(np.int8)
    return q, scale, q.astype(np.float32) * scale

def quantize_with_outlier_protection(x: np.ndarray, threshold: float):
    is_outlier = np.abs(x) > threshold
    x_normal = np.where(is_outlier, 0.0, x)
    _, _, reconstructed = quantize_int8_symmetric(x_normal)
    return np.where(is_outlier, x, reconstructed)   # outliers pass through untouched

x = np.array([0.02, -0.01, 0.03, 4.8], dtype=np.float32)
print(quantize_int8_symmetric(x)[2])                       # small values crushed
print(quantize_with_outlier_protection(x, threshold=1.0))  # small values survive

How much quality that actually costs, measured rather than guessed, is the part worth remembering: across a recent batch of 70B-class models, FP8 lands within about 0.4 points of full FP16 on standard reasoning and coding benchmarks, INT8 within about 0.7 points, and 4-bit quantization within about 1.6 points.

Format	Memory (7B model)	Typical quality cost	Best for
FP16 / BF16	~14 GB	None (baseline)	Development, quality-critical serving
FP8	~7 GB	~0.4 points	Default first choice for most production deployments
INT8 (AWQ)	~7 GB	~0.7 points	Broad hardware support, including older GPUs
INT4 (AWQ)	~3.5 GB	~1.6 points	Memory-constrained or consumer GPUs, expect visible loss on hard reasoning

Halving memory for a fraction of a point on an eval is close to a free lunch. That’s exactly why it’s tempting to skip measuring it, and exactly why you shouldn’t.

Common mistakes#

Using an unsigned type for a quantity that can go below zero: the classic C loop for (unsigned i = length; i >= 0; i--) never terminates, because 0 - 1 wraps to the maximum unsigned value instead of going negative.

Picking a numeric format by “more bits is safer” instead of by what’s actually scarce, range or precision, for the values involved: FP16’s exponent, not its mantissa, is why it underflows and overflows more than BF16 during training.

Applying one global scale to an entire tensor instead of per-channel scales: this is the exact outlier failure mode worked by hand above, and it’s still the most common way a quantization pass silently tanks model quality.

Quantizing without an eval suite. Cutting memory in half tells you nothing about whether you also cut reasoning quality in half on your specific workload; the only way to know is to measure it, not assume it from a table like the one above.

Try it yourself#

Beginner. Convert 37 to binary by hand using repeated division, then convert 1101 back to decimal.

Intermediate. Represent $-37$ in 8-bit two’s complement and verify $-37 + 37 = 0$ using the flip-and-carry method by hand. Separately, encode $+2.0$ in FP32 by hand and check it against float_to_ieee754_bits.

Advanced. Extend quantize_with_outlier_protection so the outlier threshold is computed per-channel (per row of a 2D matrix) rather than as one global cutoff, and demonstrate on a synthetic matrix with a single outlier row that this fixes the crushed-precision problem the global-scale version has.

---

The one-sentence version: binary place value plus a sign-flipped top bit gives you integers that add and subtract with one circuit, floating point spends those same bits on a sign, an exponent, and a mantissa to trade range against precision on purpose, and quantization pushes that trade-off further still by choosing, deliberately and measurably, how much precision a model can afford to lose. Every one of those decisions about what a number is made of, and how much of it survives being written down, is exactly the layer that dot products, matrix multiplication, and GPU tensor cores build on top of next, and it’s also exactly what gets revisited, with sharper tools, when deploying an already-trained model rather than training one.