Performance

The same column in every format

Seven column shapes, 4 million elements each, sizes in MB. S2R is the smallest form the library can produce for that shape.

The two red bars crossed the input: on a high-cardinality column the dictionary stores a dictionary the size of the data, and hands back 41.01 MB for the 30.52 MB it was given. Smart2Raw has no such bar — the widest class it has is the input. Look at rows B, C and D too, where the classical format wins: they are in the chart for the same reason as the rest. And note what this chart does not measure: it counts bytes, and bytes are only one axis — the other one is just below.

cc -O2 -std=c11 -I include benchmarks/format_matrix.c -o format_matrix
./format_matrix 4000000

The dictionary column is the theoretical floor of that format — codes of ceil(log2(k)) bits plus a dictionary of k values at 8 bytes, with no implementation overhead added. It is the most generous number the format can possibly produce.

Read row E and row F. On a high-cardinality column the dictionary is the size of the data, and the total lands at 41.01 MB against a 30.52 MB baseline — the format made the column larger than it was. RLE does the same on anything unordered, one run per value. A bitmap only exists at all when there are exactly two distinct values.

Smart2Raw has no such row, and not by tuning: it classifies by range, and its widest class is the int64 input. Row F is the proof — maximum entropy, and the result ties the baseline exactly. assert(s <= raw) runs inside the loop before each line is printed.

Bytes are only one axis

The table above answers "how much space". It does not answer the question that comes next, and that one decides what it costs to run: what you can ask of the bytes without first turning them into something else.

This chart isolates exactly that. The column is the same one, and both formats take practically the same space — 11.44 MB against 11.45 MB. With the bytes tied, what is left in the drawing is processing alone.

* Reaching a non-SQL kernel: a quantized dot product, a convolution, an int8 matmul, a DSP filter — each needs a contiguous native-width buffer. The warehouse format has to produce that buffer before it can start. Our pool already is that buffer.

The peer here is not a straw man. The dictionary is implemented at its best, over sorted distinct values — which makes the predicate a comparison on the codes themselves, with nothing to decode. That is why COUNT comes out level, and saying so is what earns the other two rows their credit.

• COUNT is parity. 1.05x, with the range over five runs straddling 1.00 in both directions. A sorted dictionary costs the peer nothing here — and buys it nothing either.
• SUM has no such shortcut. A code is not an addable operand: the peer must histogram over codes and fold the dictionary in afterwards. That scatter is structural, and it measures 17x.
• The 7.9 ms in the third row do not move. It is not a slow implementation — it is the format's definition, and no better unpacker removes it, because the output buffer has to exist somewhere. It is the one measurement here that a better-implemented peer cannot change.

And where we lose is measured alongside. On a column of 12 distinct values spread across a wide range, the peer with 4-bit codes wins: 11.44 MB against 5.72 MB and 0.468 ms against 0.326 ms — 2x larger and 1.44x slower on our side. The floor is arithmetic: 12 distinct values need log₂(12) = 3.58 bits, the peer uses 4, and the smallest native class here is 8. That absence is the decision that buys the 7.9 ms in the row above, not an oversight.

The full report — including what was retracted from an earlier version of it — is in benchmarks/warehouse/WAREHOUSE_FORMAT_BENCH.md. It measures 3.4.0, when the same loss was 4x and 3.2x; the affine factoring in 3.5.0 is what reduced it.

Query timings

The cumulative index pays for itself in 11 queries and then costs nothing, because 2 KB does not grow with the column. It also refuses to answer when it is stale: the pool carries an epoch, every write bumps it, and the index records the epoch it was built at.

Against a real database: SQLite

The tables above compare against formats. A format is an abstraction, and almost nobody runs one in production — almost everybody runs a SQLite. benchmarks/maestro/ compares against it: the same input CSV, a real .db and a set of real .s2r files written to disk, both openable afterwards.

python benchmarks/maestro/smart2raw_bench.py your_data.csv

Python here is only the conductor: it uses the standard library for SQLite and calls the real C kernels through ctypes — the native engine is compiled from s2r_shim.c on first run. With no compiler it still prints memory, disk and data moved, which are exact counts, and skips the two timed sections. Columns that genuinely need 64 bits, or that are floating point, show 0% on purpose.

What that 161× is not. SQLite gives you SQL, indexes, transactions and durability; here it is doing one job — scanning and aggregating integer columns, with no index — against a column built for exactly that job. The number measures the distance between a general engine interpreting bytecode row by row and a SIMD loop over contiguous bytes. It is not a flaw in SQLite; it is the price of generality, and it is large precisely because of that.

And the data is small on purpose — the demo CSV is about 670 KB, so everything fits in cache and what dominates is the interpreter's per-row cost. On much larger data the timed ratio does not hold; what holds are the three rows marked exact, because those are byte counts rather than a clock: 1.49× on disk, 2.73× in memory, 3.15× less data moved per scan.

Look at the 1.18×. The range filter barely improves, and the row is here for that reason. SUM on u8 processes eight values per lane where int64 processes one, and jumps to 3.29×; the filter was already cheap per element and its bottleneck is elsewhere. A space win does not become a time win by decree — it becomes one when the operation was memory-bound, and this row is the counterexample inside our own table.

Measure it in your own browser

The demonstration on the home page times the same queries on your own data, in your own browser, with warm-up and a varying argument so nothing can be cached — and it prints the result of every path so you can see them agree.

It will also show you where the win in space does not become a win in time: on a column small enough to sit in cache, reading a quarter of the bytes does not take a quarter of the time. Raise the element count and the difference appears.

How the whole thing is checked

• 31 test suites, 0 failures, including a differential fuzz suite of 100,950 checks against a naive reference, with fixed seeds.
• ASan and UBSan clean.
• The same code run on x86-64 with SSE2 and AVX2, ARM with NEON and SVE2, RISC-V with RVV, big-endian, and in lean mode with no stdio, no mmap and no SIMD. Only x86-64 is real hardware; ARM64 and big-endian (s390x) are repeated by CI on every commit on the real ISA, on a machine emulated by QEMU — and for big-endian the job prints the byte order from inside the binary before testing anything, with 250,212 checks, 0 failures on top of it. The RVV and SVE2 kernels run against a scalar reference with the vector length swept from 128 to 1024 bits, which is more than a single board would give.
• File compatibility measured in both directions between versions.

bash scripts/build_and_test.sh

That fuzz suite exists because of a real defect it found: an unsigned column crossing 2^63 used to return truncated values in the block-wise layer, with no error, no warning and a valid CRC. Twenty-five suites of chosen cases missed it. It is fixed, and the minimal case {1, UINT64_MAX} is now a fixed test.