Protocol conformity · open source
Stroppy Conformity
Does this database really speak PostgreSQL? MySQL? MongoDB? Elasticsearch? Redis? Measured, not claimed.
It runs the same scripted interactions against the real engine and against the engine that claims to be compatible, and compares what each does on the wire. Every verdict carries its evidence: what was sent, what the original answered, what the engine answered, and the frame trace.
- Five protocols
- Expected values from the real engine
- No account needed
- Apache-2.0
- Works the same
- Differs
- Refused
- Breaks
- Not measured
PostgreSQL wire protocol20 engines, best first. Pick an engine for what differs; pick a column for its features.
| Types | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Engine | Score | |||||||||||
| Referencepostgres-13 to postgres-18 | the original | Works the same, 86 of 86 checks matched | Works the same, 72 of 72 checks matched | Works the same, 60 of 60 checks matched | Works the same, 36 of 36 checks matched | Works the same, 96 of 96 checks matched | Works the same, 66 of 66 checks matched | Works the same, 96 of 96 checks matched | Works the same, 522 of 522 checks matched | Works the same, 246 of 246 checks matched | Works the same, 2004 of 2004 checks matched | Works the same, 4380 of 4380 checks matched |
| Refused, 11 of 12 checks matched | Works the same, 12 of 12 checks matched | Works the same, 10 of 10 checks matched | Works the same, 6 of 6 checks matched | Works the same, 16 of 16 checks matched | Works the same, 11 of 11 checks matched | Works the same, 16 of 16 checks matched | Works the same, 87 of 87 checks matched | Works the same, 41 of 41 checks matched | Works the same, 334 of 334 checks matched | Works the same, 735 of 735 checks matched | ||
| Refused, 11 of 12 checks matched | Works the same, 12 of 12 checks matched | Works the same, 10 of 10 checks matched | Works the same, 6 of 6 checks matched | Works the same, 16 of 16 checks matched | Works the same, 11 of 11 checks matched | Works the same, 16 of 16 checks matched | Works the same, 87 of 87 checks matched | Differs, 40 of 41 checks matched | Works the same, 332 of 332 checks matched | Differs, 722 of 727 checks matched | ||
| Refused, 11 of 12 checks matched | Works the same, 12 of 12 checks matched | Works the same, 10 of 10 checks matched | Refused, 5 of 6 checks matched | Works the same, 16 of 16 checks matched | Works the same, 11 of 11 checks matched | Works the same, 16 of 16 checks matched | Differs, 86 of 87 checks matched | Works the same, 41 of 41 checks matched | Works the same, 334 of 334 checks matched | Differs, 732 of 733 checks matched | ||
| Differs, 9 of 12 checks matched | Breaks, 11 of 12 checks matched | Breaks, 8 of 10 checks matched | Differs, 4 of 6 checks matched | Works the same, 16 of 16 checks matched | Works the same, 11 of 11 checks matched | Works the same, 16 of 16 checks matched | Works the same, 87 of 87 checks matched | Works the same, 41 of 41 checks matched | Works the same, 334 of 334 checks matched | Differs, 736 of 737 checks matched | ||
| Breaks, 7 of 15 checks matched | Differs, 11 of 12 checks matched | Refused, 9 of 10 checks matched | Differs, 1 of 6 checks matched | Differs, 13 of 16 checks matched | Differs, 8 of 11 checks matched | Differs, 9 of 16 checks matched | Breaks, 24 of 81 checks matched | Differs, 7 of 41 checks matched | Differs, 87 of 89 checks matched | Breaks, 281 of 332 checks matched | ||
| Breaks, 6 of 12 checks matched | Differs, 8 of 12 checks matched | Differs, 5 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 9 of 16 checks matched | Differs, 8 of 11 checks matched | Differs, 4 of 16 checks matched | Differs, 49 of 81 checks matched | Differs, 12 of 41 checks matched | Differs, 100 of 110 checks matched | Differs, 429 of 517 checks matched | ||
| Differs, 9 of 12 checks matched | Differs, 9 of 12 checks matched | Differs, 4 of 10 checks matched | Differs, 2 of 6 checks matched | Differs, 9 of 16 checks matched | Differs, 4 of 11 checks matched | Differs, 5 of 16 checks matched | Differs, 23 of 80 checks matched | Differs, 12 of 41 checks matched | Differs, 75 of 82 checks matched | Differs, 322 of 403 checks matched | ||
| Differs, 10 of 14 checks matched | Differs, 9 of 12 checks matched | Differs, 6 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 12 of 16 checks matched | Differs, 8 of 11 checks matched | Differs, 2 of 16 checks matched | Differs, 46 of 87 checks matched | Differs, 7 of 41 checks matched | Differs, 98 of 107 checks matched | Breaks, 84 of 698 checks matched | ||
| Breaks, 6 of 14 checks matched | Differs, 6 of 12 checks matched | Differs, 1 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 87 checks matched | Differs, 1 of 41 checks matched | Differs, 68 of 110 checks matched | Breaks, 215 of 479 checks matched | ||
| Differs, 5 of 12 checks matched | Differs, 10 of 12 checks matched | Differs, 2 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 10 of 16 checks matched | Differs, 8 of 11 checks matched | Differs, 3 of 16 checks matched | Differs, 1 of 87 checks matched | Differs, 6 of 41 checks matched | Differs, 87 of 96 checks matched | Differs, 1 of 80 checks matched | ||
| Breaks, 3 of 12 checks matched | Breaks, 3 of 12 checks matched | Breaks, 1 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 1 of 81 checks matched | Differs, 1 of 41 checks matched | Differs, 139 of 141 checks matched | Breaks, 1 of 80 checks matched | ||
| Differs, 5 of 12 checks matched | Differs, 6 of 12 checks matched | Breaks, 1 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 6 of 16 checks matched | Differs, 2 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 80 checks matched | Differs, 2 of 41 checks matched | Differs, 42 of 51 checks matched | Differs, 0 of 80 checks matched | ||
| Breaks, 3 of 12 checks matched | Breaks, 5 of 12 checks matched | Differs, 1 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 8 of 16 checks matched | Differs, 2 of 11 checks matched | Differs, 1 of 16 checks matched | Differs, 1 of 81 checks matched | Differs, 2 of 41 checks matched | Differs, 47 of 50 checks matched | Breaks, 0 of 80 checks matched | ||
| Differs, 5 of 12 checks matched | Differs, 1 of 3 checks matched | Not measured, 0 of 2 checks matched | Differs, 1 of 4 checks matched | Not measured | Not measured | Not measured | Not measured | Differs, 0 of 4 checks matched | Differs, 0 of 2 checks matched | Breaks, 0 of 77 checks matched | ||
| Breaks, 4 of 12 checks matched | Differs, 5 of 12 checks matched | Differs, 1 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 80 checks matched | Differs, 1 of 41 checks matched | Differs, 18 of 27 checks matched | Breaks, 0 of 80 checks matched | ||
| Differs, 6 of 12 checks matched | Differs, 2 of 12 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 80 checks matched | Differs, 1 of 41 checks matched | Differs, 30 of 51 checks matched | Differs, 1 of 80 checks matched | ||
| Breaks, 4 of 12 checks matched | Differs, 1 of 12 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 4 of 16 checks matched | Differs, 3 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 8 of 79 checks matched | Differs, 1 of 41 checks matched | Not measured | Differs, 6 of 80 checks matched | ||
| Breaks, 2 of 12 checks matched | Differs, 1 of 12 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 80 checks matched | Differs, 1 of 41 checks matched | Differs, 23 of 25 checks matched | Differs, 0 of 80 checks matched | ||
| Breaks, 3 of 12 checks matched | Differs, 1 of 12 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Differs, 0 of 12 checks matched | Differs, 0 of 41 checks matched | Not measured | Differs, 0 of 80 checks matched | ||
| Differs, 2 of 12 checks matched | Differs, 1 of 12 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 6 checks matched | Differs, 0 of 16 checks matched | Breaks, 0 of 11 checks matched | Differs, 0 of 16 checks matched | Not measured, 0 of 87 checks matched | Differs, 1 of 41 checks matched | Not measured, 0 of 334 checks matched | Not measured, 0 of 739 checks matched | ||
MySQL client/server protocol14 engines, best first. Pick an engine for what differs; pick a column for its features.
| Engine | Score | ||||||
|---|---|---|---|---|---|---|---|
| Referencemysql-5.7 to mysql-9 | the original | Works the same, 60 of 60 checks matched | Works the same, 52 of 52 checks matched | Works the same, 8 of 8 checks matched | Works the same, 28 of 28 checks matched | Works the same, 380 of 380 checks matched | Works the same, 76 of 76 checks matched |
| Works the same, 15 of 15 checks matched | Works the same, 13 of 13 checks matched | Works the same, 2 of 2 checks matched | Works the same, 7 of 7 checks matched | Works the same, 95 of 95 checks matched | Works the same, 19 of 19 checks matched | ||
| Differs, 4 of 11 checks matched | Works the same, 13 of 13 checks matched | Differs, 1 of 2 checks matched | Differs, 5 of 7 checks matched | Differs, 91 of 95 checks matched | Differs, 18 of 19 checks matched | ||
| Differs, 6 of 15 checks matched | Works the same, 13 of 13 checks matched | Differs, 1 of 2 checks matched | Differs, 5 of 7 checks matched | Differs, 82 of 95 checks matched | Differs, 18 of 19 checks matched | ||
| Differs, 4 of 11 checks matched | Breaks, 6 of 13 checks matched | Works the same, 2 of 2 checks matched | Differs, 2 of 7 checks matched | Differs, 72 of 92 checks matched | Breaks, 12 of 19 checks matched | ||
| Differs, 4 of 11 checks matched | Differs, 4 of 13 checks matched | Differs, 1 of 2 checks matched | Differs, 2 of 7 checks matched | Differs, 85 of 95 checks matched | Differs, 3 of 19 checks matched | ||
| Differs, 3 of 11 checks matched | Breaks, 5 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 2 of 7 checks matched | Differs, 67 of 95 checks matched | Refused, 1 of 19 checks matched | ||
| Differs, 3 of 11 checks matched | Breaks, 3 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 51 of 95 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 4 of 11 checks matched | Differs, 1 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 50 of 95 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 3 of 11 checks matched | Breaks, 1 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 15 of 64 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 2 of 11 checks matched | Differs, 1 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 17 of 65 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 4 of 15 checks matched | Differs, 1 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 1 of 7 checks matched | Differs, 0 of 48 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 1 of 11 checks matched | Breaks, 1 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 95 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 1 of 11 checks matched | Differs, 0 of 13 checks matched | Differs, 0 of 2 checks matched | Breaks, 0 of 7 checks matched | Breaks, 0 of 85 checks matched | Differs, 0 of 19 checks matched | ||
| Differs, 1 of 11 checks matched | Differs, 0 of 13 checks matched | Differs, 0 of 2 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 48 checks matched | Differs, 0 of 19 checks matched | ||
MongoDB wire protocol3 engines, best first. Pick an engine for what differs; pick a column for its features.
| Types | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Engine | Score | |||||||||||
| Referencemongodb-6.0 to mongodb-8.0 | the original | Works the same, 93 of 93 checks matched | Works the same, 12 of 12 checks matched | Works the same, 6 of 6 checks matched | Works the same, 3 of 3 checks matched | Works the same, 18 of 18 checks matched | Works the same, 15 of 15 checks matched | Works the same, 6 of 6 checks matched | Works the same, 9 of 9 checks matched | Works the same, 9 of 9 checks matched | Works the same, 12 of 12 checks matched | Works the same, 36 of 36 checks matched |
| Refused, 29 of 30 checks matched | Works the same, 4 of 4 checks matched | Works the same, 2 of 2 checks matched | Works the same, 1 of 1 checks matched | Works the same, 6 of 6 checks matched | Works the same, 5 of 5 checks matched | Works the same, 2 of 2 checks matched | Works the same, 3 of 3 checks matched | Works the same, 3 of 3 checks matched | Works the same, 4 of 4 checks matched | Works the same, 12 of 12 checks matched | ||
| Differs, 10 of 30 checks matched | Differs, 2 of 4 checks matched | Differs, 1 of 2 checks matched | Works the same, 1 of 1 checks matched | Differs, 5 of 6 checks matched | Differs, 4 of 5 checks matched | Differs, 1 of 2 checks matched | Works the same, 3 of 3 checks matched | Works the same, 3 of 3 checks matched | Differs, 2 of 4 checks matched | Differs, 4 of 12 checks matched | ||
| Differs, 9 of 31 checks matched | Differs, 2 of 4 checks matched | Differs, 1 of 2 checks matched | Refused, 0 of 1 checks matched | Differs, 5 of 6 checks matched | Differs, 4 of 5 checks matched | Differs, 1 of 2 checks matched | Works the same, 3 of 3 checks matched | Works the same, 3 of 3 checks matched | Differs, 2 of 4 checks matched | Differs, 5 of 12 checks matched | ||
Elasticsearch REST API6 engines, best first. Pick an engine for what differs; pick a column for its features.
| Types | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Engine | Score | ||||||||||
| Referenceelasticsearch-7 to elasticsearch-9 | the original | Works the same, 21 of 21 checks matched | Works the same, 21 of 21 checks matched | Works the same, 24 of 24 checks matched | Works the same, 30 of 30 checks matched | Works the same, 30 of 30 checks matched | Works the same, 9 of 9 checks matched | Works the same, 33 of 33 checks matched | Works the same, 21 of 21 checks matched | Works the same, 21 of 21 checks matched | Works the same, 108 of 108 checks matched |
| Differs, 4 of 7 checks matched | Differs, 5 of 7 checks matched | Differs, 6 of 8 checks matched | Differs, 9 of 10 checks matched | Differs, 7 of 10 checks matched | Differs, 2 of 3 checks matched | Differs, 7 of 11 checks matched | Differs, 4 of 7 checks matched | Works the same, 7 of 7 checks matched | Differs, 34 of 36 checks matched | ||
| Differs, 4 of 7 checks matched | Differs, 5 of 7 checks matched | Differs, 6 of 8 checks matched | Differs, 9 of 10 checks matched | Differs, 7 of 10 checks matched | Differs, 2 of 3 checks matched | Differs, 7 of 11 checks matched | Differs, 4 of 7 checks matched | Works the same, 7 of 7 checks matched | Differs, 32 of 36 checks matched | ||
| Differs, 4 of 7 checks matched | Differs, 4 of 7 checks matched | Differs, 5 of 8 checks matched | Differs, 8 of 10 checks matched | Differs, 7 of 10 checks matched | Differs, 2 of 3 checks matched | Differs, 7 of 11 checks matched | Differs, 4 of 7 checks matched | Differs, 6 of 7 checks matched | Differs, 30 of 36 checks matched | ||
| Differs, 2 of 7 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 8 checks matched | Not measured, 0 of 10 checks matched | Not measured, 0 of 10 checks matched | Not measured, 0 of 3 checks matched | Not measured, 0 of 11 checks matched | Not measured, 0 of 7 checks matched | Refused, 0 of 7 checks matched | Differs, 1 of 36 checks matched | ||
| Differs, 2 of 7 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 8 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 3 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 36 checks matched | ||
| Differs, 2 of 7 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 8 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 10 checks matched | Differs, 0 of 3 checks matched | Differs, 0 of 11 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 7 checks matched | Differs, 0 of 36 checks matched | ||
Tier 0 is the wire contract only: frames, types and bulk load. It is not the SQL dialect, performance, durability or feature coverage. A high score means drivers and tools can talk to the engine as they do to the reference; it does not mean your queries behave the same.
Dataset 45a7361f915b, audited 23 October 2026.
What it covers
Five protocols, each against its own releases, at every layer a client touches: connection and authentication, query flows, errors and cancel, the type contract, bulk load, replication, the catalog queries clients send, and the originals’ own test suites.
- PostgreSQL wire13 to 18
- MySQL wire5.7 to 9
- MongoDB wire6.0 to 8.0
- Elasticsearch REST7 to 9
- Redis7.2 and 8.2, RESP2 and RESP3
Why the numbers hold up
A refusal is loud; a different answer is quiet, because the client gets it without an error. The report keeps the two apart, and every row says exactly what was measured.
- Expected values come from the real engine, never written by hand; each original is also run against its own recording.
- A crash is charged only when the server is seen to go down or restart.
- Catalog queries come from the clients themselves: pgx, pgjdbc, psql, pg_dump, the mysql CLI, mongosh, Kibana and others.
compat.json, dataset 45a7361f915b
{
"id": "cockroachdb-pgwire",
"claims": "18.0.0",
"baseline": {"reference": "postgres-18"},
"image_digest": "cockroachdb/cockroach@sha256:c26ade77…",
"measured_at": "2026-10-23T13:09:30Z"
}
Try it
Go 1.27 and Docker with compose v2. Bring up an original and an engine, record what the original answers, run the engine against it, then open out/site/report.html.
Terminal
make build
bin/conformity targets up postgres-18 cockroachdb
bin/conformity golden -t postgres-18
bin/conformity run -t cockroachdb
bin/conformity reports export --format site,html -o out/site
bin/conformity targets down postgres-18 cockroachdb
Measure your build
A branch, a nightly or a server with a compatibility switch: describe the endpoint in a small YAML file and run against it. Without a compose file the tool only connects; it never starts or stops anything. Nothing needs an account or a cloud service.
mine/my-engine.yaml
name: my-engine
engine: my-engine
version: "branch-x"
role: subject
endpoints:
- protocol: pgwire # or mysql, mongo, elastic
host: 10.0.0.5
port: 5432
user: conformity
password: secret
database: conformity
For agents
conformity compare answers a migration question from the measured data: what is lost going from one engine to another, concept by concept, with the checks behind it. An MCP server gives an agent the same answers, and says “not measured” where there is none.
Terminal
conformity compare --from postgres-17 --to cockroachdb \
--concept "Bulk load" --format json
conformity mcp --dataset /path/to/export