Flaky tests are the quiet tax on every CI pipeline. A test fails, someone reruns the job, it passes, and everyone learns to ignore red. jvq started as a gate that fails pull requests when benchmarks blow their budget. Since 0.7 it has a second job: spotting flaky tests and quarantining them without hiding real failures.
Two releases on November 4 made that second gate much easier to live with.

0.8, three lines of YAML
The flaky gate moved into jvq’s GitHub Action. You name your test result files with a flaky: entry and the action does the rest. It keeps the history in the Actions cache, adds to it on pushes to the default branch, and judges pull requests against it without writing anything back. So a bad branch can’t poison the record.
It also fixed a nasty gap from 0.7. When a quarantined Go test crashed, the rest of its package silently never ran. Now jvq lists those tests as not run and puts a warning on the job.
0.9, a history that fits
The history file used to repeat every test name on every run. Now each name is stored once. For 2,000 tests over 200 runs that takes it from 26.8 MB to 319 KB. Small enough that nobody thinks about the cache anymore.
Pinning is the other addition. In jvq.toml a test can be set to always, which quarantines it from the first run, or never, which keeps it out of quarantine whatever its record says. A pinned test that actually breaks still blocks the build. There’s also submodules = true for repos whose benchmarks live in Git submodules.
| Release | What it added |
|---|---|
| 0.4 | GitLab merge request support |
| 0.5 | History of runs on the default branch, to catch slow drift |
| 0.6 | Instruction counting under cachegrind, for small slowdowns |
| 0.7 | The flaky test gate, reading JUnit XML and go test JSON |
| 0.8 | Flaky gate inside the GitHub Action; crash-skipped tests reported |
| 0.9 | History about 80 times smaller; pinned tests |
Where it fits
Flaky test handling is usually sold as part of a bigger CI analytics product, and teams either pay for it or live with reruns. jvq takes the opposite route: one small open source binary, Apache 2.0, with the history kept in your own CI cache. Both releases went through two independent reviews before shipping. Those found eighteen problems in total, among them a pin that could hide a compile error and a crafted history file that could eat 2 GB of memory. All were fixed with tests.
More on the studio’s tools on our Software Projects Desk.