Stroppy 6.2 is out. This release is mostly about the work around a benchmark: keeping its results, understanding what happened, comparing it with another run, and writing a workload when the built-ins do not quite fit.

The Go engine arrived in 6.0. Since then we have been using it, finding rough edges, and filling in the parts that make an experiment repeatable. There are also new database presets, a banking workload, and a few ways to control TPC-C traffic more carefully. None of that makes a benchmark fair on its own, but it makes fewer details depend on what we remember after the terminal closes.

We released Stroppy v6.0.0. This is the largest change we have made to the project so far: Stroppy is now a single Go binary with its own workload engine, typed parameters, native metrics, and Go implementations of every built-in workload.

Stroppy v5 was a k6 extension. Stroppy v6 is not.

We did not make that change because k6 is a poor load-testing tool. k6 gave us virtual users, scenarios, metrics, reporting, and a JavaScript runtime when Stroppy was much smaller. It helped us reach a working multi-database benchmark faster than building every subsystem ourselves.

The problem was fit. As Stroppy became a database benchmark rather than a thin extension, we used less of what k6 was built to provide and added more adapters around the parts we did use. Eventually the adapter layer became the product.

Point two TPC-C load generators at the same database, same data, same concurrency, same duration — and they report noticeably different throughput. Stroppy (ours) said ~3.6k transactions/sec; go-tpc said ~6.1k. We wanted to know where that gap comes from before trusting either number.

This is the trail of guesses and tests we went through. We reached an answer we believe, and found two real bugs in Stroppy along the way.

The Problem​

In performance-critical systems, ideally we want to validate every source code change and make sure no performance regressions are introduced. Functional tests handle correctness, but performance is trickier — how do you know a change won't slow things down?

We worked with the OrioleDB maintainers to build a CI pipeline that runs TPC-C benchmarks on every PR and flags regressions before merge.

OrioleDB is a PostgreSQL extension that combines the advantages of both on-disk and in-memory storage engines, using PostgreSQL's pluggable storage to improve performance and reduce costs.

Stroppy v4.0.0 is a major release that adds database transactions, reworks the CLI from the ground up, and ships a built-in help system. If you've been using v3.1.0, this post covers everything that changed and why.

Comparing databases is hard. There are plenty of well-established benchmarking suites out there, and plenty of people with more experience doing it. But at some point, if you need to understand how a database behaves under your workload on your hardware, you end up running the tests yourself. Vendor benchmarks, Stack Overflow threads, and even LLM-generated answers all have their limits — they can point you in a direction, but they can't replace your own measurements.

Stroppy is a tool we've been building to make that hands-on testing more practical. It's built on top of k6, the load testing framework, and focuses on database stress testing with parameterized SQL, data generation, and a pluggable driver model.

This post is a walkthrough of our first attempt at running Stroppy against two databases at once: PostgreSQL and MySQL, both with completely stock configurations, running a TPC-C-derived workload for 30 minutes per test. It's not a proper benchmark — more of a practice run to shake out the tooling. Along the way, we'll cover how we added MySQL support, why running 16 sequential tests by hand quickly becomes impractical, and what happens when your test VM runs out of disk space mid-run.