You inherited a back end with no reliable tests, and a vendor is proposing a library or framework to “make it testable.” I would refuse to evaluate that proposal by its demo or its promised migration speed, because neither tells you whether the existing service will keep behaving correctly. The vendor’s first job is to help you capture behavior you can rerun without trusting the code that produced it.
The existing service is evidence, not a specification
An untested route can be wrong and still have callers that depend on its behavior. Record what it does before deciding what it should do, because a replacement can break a client even while improving the API on paper. For each candidate change, identify one request, its response, the database effect, and any outbound call; that sequence gives you something more useful than a passing unit test built around a mocked controller.
Start with a small sample rather than a full endpoint inventory. A practical first batch is 12 requests across four routes: a normal result, an authorization failure, and a malformed input for each route. Treat those counts as a starting value to tune, not a coverage target. Choose routes with different side effects so that a framework cannot look successful merely by returning the expected JSON.
Capture status codes, response headers, and bodies, but classify each field before asserting it. A generated ID or timestamp may need a format assertion; an error code consumed by a client may need an exact match. HTTP semantics in RFC 9110 help you challenge surprising statuses, while an OpenAPI 3.1 document can express an agreed future contract; neither proves what current clients actually receive. When observation and the desired contract differ, keep both records and mark the disagreement for a product decision.
I would not generate a large test suite from the current OpenAPI file and call the job done, because an inherited, unverified document can reproduce the same omissions as the code. Instead, compare its examples with captured traffic and client code, then write down which source supports each assertion. That provenance lets you reject an accidental behavior without silently breaking a dependency.
A vendor should survive one ugly route before earning a migration plan
Ask the vendor to work in a disposable branch with one route that uses authentication, PostgreSQL state, and an outbound dependency. The route should include an error path, because a happy-path demo can hide differences in transaction rollback, error serialization, and timeout handling. Give every candidate the same seeded data and the same requests; otherwise, the comparison measures the quality of each demo rather than the quality of the proposed tool.
The first artifact can be a small black-box probe. Save this as probe.mjs and run it with Node.js 22 or a newer release that provides global fetch. Change the path and expected status to match an agreed JSON error response in your service.
import assert from 'node:assert/strict';
const base = process.env.BASE_URL ?? 'http://127.0.0.1:3000';
const path = process.env.QA_PATH ?? '/api/orders/not-a-real-id';
const response = await fetch(new URL(path, base), {
headers: process.env.QA_TOKEN ? { Authorization: `Bearer ${process.env.QA_TOKEN}` } : {}
});
console.log(JSON.stringify({ path, status: response.status }));
assert.equal(response.status, Number(process.env.QA_EXPECT ?? 404));
assert.match(response.headers.get('content-type') ?? '', /json/i);
The probe deliberately logs no response body, because an error response may contain account data or a diagnostic token. It is not a complete test: add assertions for stable error fields only after you have observed them and agreed which are contractual. Run it against the untouched service and the vendor branch, then retain both results with the commit hashes so that a changed outcome has a traceable cause.
For the database check, seed a PostgreSQL 16 container through Testcontainers for Node.js, invoke the route, and query the affected rows afterward. Set an initial tolerance of zero unexpected row changes; that is a deliberate acceptance threshold, not a claim about typical systems. If an outbound call matters, replace it with a local HTTP stub and assert the request it receives, because a response-only test cannot reveal a duplicate charge or an omitted notification.
I would challenge Vet Framework Claims Before You Scope the Build with this narrower demand: make a claim about one failing route, then reproduce the fix without the vendor present. A claim such as “Fastify 5 makes validation automatic” earns no credit until the team shows the schema attached to the relevant route and the resulting response for malformed input. The point is not to make the vendor do free migration work; it is to find out whether your team can maintain the proposed behavior after the demo ends.
Black-box replay wins first; in-process tests win when diagnosis matters
Choose between Schemathesis against OpenAPI 3.1 and Vitest with Supertest according to the question you need answered. Schemathesis wins when you can reach a running service but cannot safely refactor its internals, because it can exercise documented request and response boundaries without importing application code. Its cost is that a weak or inaccurate schema limits what it can detect, and a failure may not identify the responsible handler.
Vitest with Supertest wins when the application exposes an HTTP handler you can instantiate in a test, because you can seed state, exercise the handler, and inspect a failure close to its source. Its cost is setup and coupling: an Express 5 app that starts a server during import, for example, may require a small extraction before tests can run reliably. Neither option replaces the other; use boundary replay to protect behavior during a change, then use focused in-process tests to explain and prevent the failures it exposes.
Measure candidates on the same workstation and fixture rather than quoting a framework benchmark. In a trial, you might record the median of 30 identical requests and set a provisional regression limit of 20% against the untouched branch; both numbers are values to adjust for the route and environment. Publish the raw runs alongside the median, because a candidate that appears faster only after dropping validation or bypassing the database has not met the same conditions.
Also count how many failures yield a useful diagnosis. If a generated test says only “500 instead of 400,” save the request body, correlation ID, and relevant trace so another engineer can reproduce it. OpenTelemetry trace IDs can connect a request to its database and outbound calls, but retain an HTTP-level assertion as well, because instrumentation can change while the client-visible contract remains the same. A useful vendor trial leaves behind fixtures and commands your team can run in CI, not screenshots of green tests.
A pass is worthless if the team cannot reproduce it next week
Set a written acceptance gate before choosing a vendor. Require the candidate to pass the captured boundary tests, preserve agreed database effects, expose a reproducible failure for a deliberately broken change, and run from a clean checkout using documented commands. These conditions are stricter than a feature comparison because they test whether the proposal reduces your uncertainty rather than merely adding dependencies.
Include operational checks in that gate when the candidate introduces a package. Record its exact version and lockfile, inspect its SemVer release history for breaking changes relevant to your integration, and run npm audit –omit=dev as one signal rather than a verdict; an audit result does not show whether a reported vulnerability is reachable in your service. Check the license and the maintainers’ response to issues you can actually reproduce, because a test helper that becomes unmaintained can leave a critical suite tied to an obsolete runtime.
A vendor-published throughput figure belongs in the trial notes, not the acceptance result, because its workload and hardware are unlikely to match your route. A measured baseline from your own checkout is more useful if you preserve the fixture, command, runtime version, and raw output. If the vendor cannot run your probe after a fresh clone, ask them to fix the instructions before debating architecture; reproducibility is a property of the handoff.
I disagree with Scope Exit Costs Before Choosing a Back-End Stack as the first lens for this decision: you cannot price a credible exit while you still lack a way to detect behavior lost during the move. Once the characterization suite runs against both implementations, migration estimates can name the routes, fixtures, and mismatches behind their numbers. Until then, a low estimate is a guess with a spreadsheet attached.
Your first test should pin down an error nobody wants to touch
Tomorrow, pick one route with an authentication or database failure that the team hesitates to change. Capture a redacted request and response, agree which details a client may rely on, and run the probe against a clean checkout. Then ask the vendor to make one small change while keeping that observation intact. You will have a decision artifact even if you reject their tool.
