How to benchmark honestly

I have published a lot of benchmarks. Most of my readmes have a section for them. Some of those numbers were good science. Some were marketing that I believed at the time. This post is about the difference, and the habits I use now to stay on the right side of it.

It is a companion to JavaScript performance tips I actually use. That post is about making code fast. This one is about knowing whether you did.

#A benchmark is a claim

When you put 12,924,662 ops/sec in a readme, you are not reporting a number. You are making a claim: this code is faster than that code, for the work you care about. Every part of that sentence can be false.

• Faster can be noise. Two runs on the same machine can differ by more than the gap you measured.
• This code can be doing less work than that code, so you are comparing two different jobs.
• The work you care about is almost never the input in the benchmark.

So before I trust a number, I try to answer three questions. Is it the same work? Is it the real work? Is the difference bigger than the noise?

#Is it the same work

The first thing my benchmark files do is not benchmark. They check that every candidate gives the right answer.

The dequal bench loads every contender, runs each one against a list of assertions, and prints a pass or a fail before any timing starts:

assert(func(1, 1), true, "equal numbers");
assert(func(1, 2), false, "not equal numbers");
assert(func(1, [1]), false, "number vs array");
assert(func(0, null), false, "number vs null");
assert(func(0, undefined), false, "number vs undefined");
assert(func(foo, bar), true, "kitchen sink");

The complex fixture then only races the libraries that passed. A deep-equal function that returns true too early is very fast and completely useless. If you skip this step, the fastest library in your chart can be the most broken one.

The clsx bench takes this a step further. Some candidates have a different API, so their output is not comparable even when it looks similar. Those get a ≠ next to their name. They still appear in the table, but the reader knows they are not doing the same job.

Same work also means same setup. If one candidate gets a pre-parsed input and another parses inside the loop, you are timing the parser.

#Is it the real work

The Polka readme has a note under its benchmark table. I wrote it in 2018, and I think it is the most important line in the file:

Time is mostly spent in your application code rather than Express or Polka code!

The benchmark measured an app with two middleware, a route, and a one-line response. That is a fair test of framework overhead. It is not a fair test of your app, because your app talks to a database. Once a request waits 20ms on Postgres, a framework that saves 0.1ms does not matter much.

I knew this when I wrote the table. I still put the table at the top, and the note below it. I would reverse that order today. See Polka, eight years later for more about what those numbers did and did not prove.

The general rule: benchmark the operation at the size and shape your users have. If your users call a function once per page load, its cold time matters and its hot loop does not. The dequal bench prints load times for each library before it runs anything, because for a lot of apps require() is the most expensive call they will ever make to it.

#Is the difference bigger than the noise

A single run is an anecdote. Here is what I do to get a real number.

Warm up. JavaScript engines compile hot code in tiers. The first thousand calls run in a slower tier than the next million. For HTTP benchmarks, I do one full warm-up run with wrk and throw it away. For function benchmarks, the harness handles this, but only if you let it run long enough.

Sample, then report the spread. benchmark.js prints a margin of error with every result:

clsx         x 12,924,662 ops/sec ±0.16% (102 runs sampled)

That ±0.16% is the part I read first. If two candidates are 2% apart and each has a 3% margin, there is no winner. Say so.

Control the machine. Close the browser. Plug in the laptop. Do not run the suite while your build runs in another tab. If the result changes when you run it again tomorrow, it was not a result.

Run it more than once, and in a different order. Order effects are real. The first candidate can pay for garbage the others leave behind, or the last one can run on a hotter CPU.

#Lies the engine tells you

Microbenchmarks have their own special failure: the compiler is smarter than your test.

let input = "hello world";

bench.add("toUpperCase", () => {
  input.toUpperCase();
});

The result is never used. An optimizing compiler is allowed to see that and remove the work entirely. You then measure an empty function and publish a miracle. The fix is to consume the result in a way the engine cannot prove is pointless:

let input = "hello world";
let sink = 0;

bench.add("toUpperCase", () => {
  sink += input.toUpperCase().length;
});

Constant inputs cause a quieter version of the same problem. If every call gets the same string, the engine can specialize on it in ways your real code never sees. Use a set of inputs and rotate through them.

The opposite also happens. A benchmark can feed one candidate many different object shapes and the other only one, and then the gap is about inline caches, not about the code you wrote. Keep the inputs the same for every candidate. The dequal harness clones the fixture fresh for each contender with klona so that no candidate sees state another one left behind.

#One engine is one opinion

V8 is not JavaScript. A trick that wins in Node can lose in Safari, and a pattern that JavaScriptCore loves can be slow in SpiderMonkey.

For anything that ships to browsers, I run the suite in Chrome, Firefox, and Safari as well as Node. The clsx bench folder has a screenshot from each. The rankings are not always the same. When they disagree, I either find a version that is good everywhere, or I pick the one that is good where the users are and I write that down.

Engine versions matter too. Every number should say what it ran on. The uvu table says Node 10. The Polka table says Node 10.4.0. Those numbers are history now, and the label is what makes them honest history instead of a current claim.

#Tools

benchmark.js was the standard for years, and most of my bench folders still use it. It has not had a release since 2017. mitata and tinybench are the active options now, and both report percentiles and not only a mean. For HTTP, wrk and autocannon are still what I reach for. For startup cost, nothing beats timing the whole process from the outside, which is how the uvu benchmark caught what the self-reported times missed.

#Benchmark wars

If you publish a table with your name at the top, someone will publish a table with their name at the top. This is fine. It is also where most of my bad habits came from.

In a benchmark war, the goal stops being "know the truth" and becomes "win the chart." You tune the fixture. You pick the Node version where you look best. You stop mentioning the case where you lose. Each step is small, and each one is defensible on its own.

What helped me was a simple test. Would I be comfortable if the other author ran my benchmark on their machine, read every line, and wrote a reply? If not, the benchmark is not done.

#What I report now

When I publish a number, I try to include:

1. The machine, the runtime, and its version.
2. The exact command or the harness file, linked.
3. A correctness check that runs before the timing.
4. The margin of error, or percentiles.
5. Every candidate I tried, including the ones where I lose.
6. One sentence about what the number does not mean.

That last item is the hardest to write and the most useful to read.

#The point of it all

A benchmark is a tool for finding out. When it becomes a tool for persuading, it stops working, slowly and without telling you.

I still like seeing my library at the top of a chart. I just want to know that it earned the spot, and I want the reader to be able to check.