test_runner: --test-force-exit with concurrency silently loses test verdicts (parent reports fewer tests than ran, exit 0)

[fski]

Version

v26.5.0 (minimal repro below, macOS arm64, Darwin 25.5.0); also observed on v24.18.0 (our CI)

Platform

macOS arm64 (Darwin 25.5.0), also observed on v24.18.0 elsewhere

Subsystem

test_runner

What steps will reproduce the bug?

Run many fast test files under --test-force-exit with --test-concurrency > 1. The parent process silently loses a fraction of the test verdicts: reporters (both built-in tap and a custom one) receive fewer test:pass/test:fail events than tests that actually ran, the parent's own # tests summary counter drops by the same amount, and the run still exits 0. No warning or error of any kind.

Generate 40 files × 50 trivially-passing tests (2000 total):

// gen.mjs
import fs from 'node:fs';
const FILES = 40, TESTS = 50;
fs.rmSync('tests', { recursive: true, force: true });
fs.mkdirSync('tests');
for (let f = 0; f < FILES; f++) {
  const lines = ["import { test } from 'node:test';"];
  for (let t = 0; t < TESTS; t++) {
    lines.push(`test('file ${f} test ${t} — some reasonably long test name padding', () => {});`);
  }
  fs.writeFileSync(`tests/f${f}.test.mjs`, lines.join('\n'));
}

Counting reporter (counts leaf verdicts the parent actually received):

// count-reporter.mjs
import fs from 'node:fs';
export default async function* countReporter(source) {
  let pass = 0, fail = 0;
  for await (const event of source) {
    if (event.type === 'test:pass' || event.type === 'test:fail') {
      if (event.data.details?.type === 'test') {
        if (event.type === 'test:pass') pass++; else fail++;
      }
    }
  }
  fs.writeSync(2, `\nREPORTER-COUNT pass=${pass} fail=${fail} total=${pass + fail}\n`);
}

Run:

node gen.mjs
node --test --test-force-exit --test-concurrency=4 \
  --test-reporter=tap --test-reporter-destination=stdout \
  --test-reporter=./count-reporter.mjs --test-reporter-destination=stderr \
  'tests/*.test.mjs'

How often does it reproduce? Is there a required condition?

Every run in our testing (21/21 runs lost verdicts), with a different loss each time. Three consecutive runs of the exact command above on v26.5.0:

run	exit code	parent # tests summary	custom reporter count	actual tests
1	0	# tests 1835	1835	2000
2	0	# tests 1794	1794	2000
3	0	# tests 1936	1936	2000

Both reporters always agree with each other and with the summary counters — the events never reach the parent at all. Removing --test-force-exit (same command otherwise) reports exactly 2000/2000 every time. Loss magnitude varies (we've seen 25–234 of 2000, ~1–12%) and occurs at every concurrency level we tried on a larger real suite; higher concurrency loses more.

Required conditions in our testing: --test-force-exit + --test-concurrency > 1 (default process isolation).

What is the expected behavior? Why is that the expected behavior?

Either every verdict from every child is reported before the forced exit, or the runner fails loudly when it knows it dropped report data. Silent partial results defeat the purpose of a test run: on our production suite (Node v24.18.0, 1351 tests) the parent reported as few as 1168 — and since exit code stays 0 when the lost fragments contain only passes, CI stays green while a meaningful fraction of the suite is unaccounted for. A failing test whose verdict lands in a lost fragment is never printed, so the failure detail can vanish from logs even when the child's non-zero exit still fails the run. (To be precise about what we measured: in the minimal repro the injected failure's verdict happened to survive all 15 sample runs and exit stayed 1; the silent loss of pass verdicts with exit 0 reproduces on every all-pass run.)

What do you see instead?

Fewer tests reported than ran — # tests 1835 for a 2000-test suite — with exit code 0 and no diagnostic (see table above).

Additional information

Suspected mechanism: --test-force-exit calls process.exit() while child→parent report streams are still flushing, so V8-serialized test-event frames from process-isolated children are truncated/dropped before the parent's FileTest parser consumes them. This is the same failure class that #54327 reported for reporter destination files and #55099 fixed — but that fix only awaits the reporter destinations' close() on the parent side before exiting. The flush guarantee was never extended one layer up, to the child→parent report stream itself, which is where these events are being lost (a reporter writing synchronously on every event still never sees them).

Related, reviewed, and believed distinct:

• Test runner with junit reporter and forceExit flag results in incomplete report #54327 / test_runner: close and flush destinations on forced exit #55099 — same class, parent-side reporter destinations only (fixed).
• test_runner: dont buffer unordered events in process isolation mode #63432 — event ordering in process isolation mode; present in v24.18.0 which still exhibits this loss.
• test_runner: infinite loop in FileTest#drainRawBuffer when child stdout contains FF 0F followed by large size bytes #62693 / test_runner: avoid hanging on incomplete v8 frames #62704 — FileTest frame parser hang on malformed frames (hang, not loss).
• test_runner: allow to force exit after all tests finished #49925 — the original --test-force-exit feature request.

We currently work around this in CI with a committed manifest of expected tests plus a completeness gate that re-runs files whose verdicts went missing — it works, but it amounts to reimplementing "report everything you ran" outside the runner.

[Navaneethp007]

Hi I would like to work on this

[avivkeller]

We accept PRs :-)

[bitpshr]

Confirming this reproduces, and I think it needs specifically the combination of --test-force-exit and --test-concurrency > 1. On v26.5.1 I needed a bit more load than the 2000 in the report to open the race reliably (80 files × 60 tests = 4800, --test-concurrency=8), but then it loses verdicts on every run:

# tests 4783
# tests 4770
# tests 4705   (out of 4800, exit 0)

Two things narrow it down. With --test-concurrency=1 the count is exactly 4800 every time, and without --test-force-exit it is also exactly 4800, so it takes both. And the loss shows up in the built-in # tests summary on its own, not just in a custom reporter, so the events look like they are dropped upstream of the reporters rather than in any one reporter.

It looks like the force-exit path is the culprit. When the root test finishes, the forceExit branch (lib/internal/test_runner/test.js around line 1463) waits for the reporter destinations to unpipe and close, then calls process.exit(). Each child's stdout is parsed into events via child.stdout.on('data', ...) in runner.js, and those parsed events still have to propagate through the parent's stream to the summary counter and the reporters. Under concurrency several children finish close together, so process.exit() can fire while some already-parsed events are still in flight, and they never get counted. Draining the reporter destinations does not cover the events still queued upstream of them.

So I think the fix needs the force-exit teardown to also drain the parent's own event pipeline, not just the reporter destinations, before exiting.