lukeed

uvu, and why test runners are slow

Most of a test run is spent before your first assertion, and uvu was an experiment in skipping that part.

1,293 wordstesting · javascript · performance · uvu

Here is a test file:

import { test } from "uvu";
import * as assert from "uvu/assert";

test("Math.sqrt()", () => {
  assert.is(Math.sqrt(4), 2);
  assert.is(Math.sqrt(144), 12);
});

test.run();

Two assertions. On any modern machine, they take less than a millisecond. Now run the same two assertions through a large test runner and watch the clock. The command takes most of a second, sometimes more. The assertions did not get slower. Everything around them did.

I wrote uvu in May 2020 to find out how much of that time was necessary. The answer was: not much.

#Where the time goes

When I benchmarked the popular runners against the same suite, I timed two things. The first was the whole process, from node starting to node exiting. The second was the time each runner reported for itself, if it reported one.

These are the numbers from the uvu readme, on Node 10:

~> "ava"   took   594ms  (  ???  )
~> "jest"  took   962ms  (356  ms)
~> "mocha" took   209ms  (  4  ms)
~> "tape"  took   122ms  (  ???  )
~> "uvu"   took    72ms  (  1.3ms)

Look at the gap between the two columns. Mocha says it ran the tests in 4ms, but the process lived for 209ms. Jest says 356ms, and the process lived for almost a second. The tests are the small part. The rest is the runner getting ready to run them.

That setup work falls into three buckets:

  1. Startup. The runner loads itself, its config loader, its reporters, its plugins, and their dependencies. A large runner is hundreds of modules before your code is even read.
  2. Transforms. Many runners pass every test file, and every file it imports, through Babel or another compiler. Caches help on the second run. The first run in CI pays in full.
  3. Isolation. Each test file gets a fresh module registry, a fresh global scope, or a whole worker process. This is a real feature. It also costs a lot, and you pay it once per file.

None of this shows up in the self-reported time, because the runner starts its own timer after it is ready. The number on the screen is honest. It is just answering a different question than the one you asked.

#What uvu skipped on purpose

uvu is small because it decided not to do most of the above. Each omission was a trade, and I want to be clear about what was traded.

No globals. test and assert are imports. That means a test file is a normal module, and a normal module can run with plain node:

$ node tests/math.js

No CLI required. The CLI exists to find files and queue them, but a single file never needs it. This is the feature I use most, and it fell out of a decision to not inject anything.

No workers. The CLI loads every test file into one process, one after the other. There is no pool to spin up and no messages to pass. On a small suite, this is the fastest possible option. On a large suite with slow, I/O-heavy tests, parallel workers would win. I chose the small suite.

No transforms. uvu runs whatever Node can run. If you want TypeScript or JSX, you bring a loader with -r, and it is the same loader you would give node:

$ uvu -r tsm tests
$ node -r tsm tests/math.ts

The runner does not know or care what language you wrote. That kept compiler choice in your hands, and kept compiler cost out of mine.

No globs. The CLI takes a directory and an optional pattern, not a glob. The smallest glob parsers are about as large as all of uvu, and that cost is paid at every startup, for a feature most suites use in the simplest possible way. I wrote a short rant about this in the CLI docs. I still agree with it.

No magic. No auto-mocking, no snapshot files on disk, no watch mode with a dependency graph. assert.snapshot compares strings inline. If you want module mocks, you structure the code so it can take the dependency, or you reach for a library.

What is left is a queue of functions, some hooks, a diff printer, and a reporter. The runtime source is three files.

#The cost of being fast

Every item in that list has a matching issue in the tracker.

The biggest one is isolation. Because every file shares one process, state can leak. One file patches a global, and the next file sees it. One file leaves a timer running, and the process does not exit. In a runner with per-file isolation, these bugs are hidden. In uvu, they surface. I think surfacing them is good. Not everybody wants to find them on a Friday afternoon.

The second is features people expect because other runners have them: parallel files, watch mode, coverage built in, mocking, describe blocks that nest. uvu has suite(), which covers most of what people use describe for, but it does not nest. Coverage works through c8 or nyc, which are better tools for the job than anything I would have built, but they are one more thing to install.

import { suite } from "uvu";
import * as assert from "uvu/assert";

const Sqrt = suite("Math.sqrt", { input: 0 });

Sqrt.before.each((ctx) => {
  ctx.input = 144;
});

Sqrt("perfect square", (ctx) => {
  assert.is(Math.sqrt(ctx.input), 12);
});

Sqrt.run();

The third cost is mine. A small runner with a strong position gets a lot of issues that ask it to change its position. Each one is reasonable on its own. Together, they describe Jest. Saying no to each of them was the job, and I was not always quick or kind about it.

#Then the ground moved

uvu came out in 2020. Since then, two things happened that changed the picture.

Vitest arrived and made the transform problem mostly go away. It reuses the Vite pipeline, so your tests compile the same way your app does, and it caches aggressively. It also kept a Jest-compatible API, which meant teams could switch without rewriting tests. Startup is still heavier than uvu, but in watch mode that cost is paid once, and the reruns are fast.

node:test shipped in Node itself, first as experimental in Node 18, then as stable in Node 20. It has no install, imports instead of globals, and a CLI that runs files directly. In a lot of ways, it is the shape I was arguing for: tests as plain modules, run by the runtime you already have.

Bun and Deno ship test runners too. The idea that a test runner is something you install and configure is fading. That is a good thing.

#Does uvu still have a place

The last release is 0.5.6, from mid-2022. It still gets over 20 million downloads a month, and most of those are not new projects. They are existing suites that work and have no reason to move.

That is fine. A test runner that does not change is a test runner that does not break your CI.

For new work, my honest advice is this. If your project already uses Vite, use Vitest. If you want zero dependencies and you are on Node 20 or later, use node:test. If you want the smallest thing that runs a test file with node file.js and prints a readable diff, uvu still does that, and it will keep doing it.

#What I learned

The thing I took away was not "small runners are fast." It was that the number on the screen is rarely the number you care about. The reported test time was never the problem. The problem was everything the runner did before it started its stopwatch.

That lesson keeps coming back. Most slow tools are slow at the edges: startup, loading, compiling, isolating. The core loop is usually fine. If you want to know where the time goes, time the whole process from the outside, and then ask what each part is for. I wrote more about measuring things this way in How to benchmark honestly.

uvu was my answer to one question: how little does a test runner need? It turned out to be a queue, a diff, and the courage to say no. The ecosystem has mostly caught up, and I like it better this way.