The server you don't have to buy.

Where it fits: databases, telemetry, IoT, AI, embedded

The question is always the same one — where is the integer column? Once you start looking, it is almost everywhere, and almost always eight bytes per element, because the width was chosen by the type declaration and not by the data.

Each one in detail, with where to start →

Bytes are only one axis. The other one decides what it costs to run.

Storing fewer bytes is half the story, and it is the half everyone tells. The other half is what you can ask of the bytes without first turning them into something else — and it only becomes visible when you take bytes out of the equation.

This chart does exactly that. The column is the same one, and both formats take practically the same space: 11.44 MB against 11.45 MB. With the bytes tied, what is left in the drawing is processing alone.

The peer is implemented at its best, with the dictionary over sorted distinct values — which is why COUNT comes out level. Saying so is what earns the other two rows their credit. SUM has no such shortcut: a code is not an addable operand. And the 7.9 ms in the last row do not move with a better implementation — that is the format's definition.

A dictionary code is meaningful only to the engine that owns the dictionary. A native-width integer is meaningful to every instruction on the machine — including the ones a warehouse format cannot reach without materialising a buffer first.

Every number, with the command that reproduces it →

How it works, in three steps

int8_t cls = s2r_classify_array(v, n);

s2r_pool_init(&p, cls, n);

s2r_count_gt_fast(&p, 100);

There are three shapes, and the library chooses between them by measuring rather than guessing: the flat pool, the affine form (v = base + stride·i, with the stride found by gcd) and the block-wise form, each block relative to its own minimum. s2r_recommend() prices all three — because the obvious entry point is usually the worst one.

The whole mechanism, and why it cannot expand your data →

Against what you already have

Comparing against "a dictionary" is comparing against an abstraction. Almost nobody runs one in production; almost everybody runs a SQLite. The repository ships a comparator that runs on your CSV and prints the whole table:

python benchmarks/maestro/smart2raw_bench.py your_data.csv

Python here is only the conductor: it uses the standard library for SQLite and calls the real C kernels through ctypes. Columns that genuinely need 64 bits, or that are floating point, show 0% on purpose — because this is choosing the right native type, not compressing.

The full table, what the 161× is not, and the 1.18× row where we barely win →

Why this matters now

Open under AGPL-3.0. And there is a version that is not published.

The edition on this site is the complete, auditable one: the whole core, the .s2r format, the SIMD predicates, the analytics layer, the ports and the 31 test suites. It is free to use, study, modify and redistribute — under AGPL-3.0-or-later, which means everything you build on top, including software offered over a network, is released under the same licence.

That clause is not a trap: it is what makes it possible to publish a complete, auditable, citable edition without giving up the commercial right. If what you build is not published under AGPL — software sold to customers, SaaS, firmware, a device — then it is a commercial licence, and it is a short conversation.

31 test suites, 0 failures · 100,950 differential fuzz checks with fixed seeds · 6 versions deposited with their own DOI · 2 defects found and published with the minimal case that reproduces each

Why everything on this page can be checked

The demonstration above is not an imitation written in JavaScript. It is include/smart2raw.h itself, compiled to WebAssembly and running in your browser. When it reports a size, the answer came from s2r_pool_bytes(); when it offers a .s2r to download, those bytes were written by s2r_blocked_save().

And every number that can be checked against a naive loop is checked, before it is printed. A disagreement lights a red badge instead of printing a pretty number.