#Test framework direction

#Status

On 2026-09-13 the owner authorized removing Codeception and evaluating the current stable PHPUnit 13 release. PHPUnit 13.3.3 is adopted after the isolated comparison below; composer.json requires ^13.3.3 and the XML configuration uses schema 13.3. The empty HTTP-suite infrastructure is removed, and qa:test runs the existing PHPUnit suite with fresh XML and JSON reports. The tested dependency graph is installed. The four failures recorded in the original 2026-09-13 comparison are historical; after the 0.29.0 test reconciliation, the complete isolated PHP inventory passes (3,135 tests), as do all eight declared theme JavaScript suites (204 tests). PHPUnit still reports 84 non-failing notices about mocks without expectations.

The owner's choice supersedes the former Codeception decision gate and bounds this work to the existing class-based PHPUnit suite. Pest 5 is not selected; there is no syntax conversion or mixed-framework migration. The earlier 2026-08-24/25 experiment and subsequent mock cleanup remain historical evidence below. Their dependency conflicts describe those snapshots, not the resolved dependency graph after Codeception removal.

Pest 4 is excluded as a destination. The disposable Pest 4.7.8 probe proved only that Pest can execute existing PHPUnit-style tests; it was run outside the repository and must not be used to justify an adoption decision. This is a product-maintenance direction, not a claim that PHPUnit 12 or Pest 4 are already unsupported.

#Why the foundation changes in either branch

PHPUnit 13 is the current PHPUnit major and requires PHP 8.4 or later. Pest 5 also requires PHP 8.4 or later and is built on PHPUnit 13. k0smos already requires PHP 8.5, so the language floor does not block either candidate.

The material distinction is therefore not the PHP version. It is whether the suite keeps PHPUnit's class-based authoring surface or adopts Pest's runner, functional syntax and plugin ecosystem on the same PHPUnit 13 generation.

#Original comparison protocol — 2026-08-24

This protocol governed the original two-candidate comparison. The owner's 2026-09-13 decision narrows the current experiment to PHPUnit 12.5 versus current stable PHPUnit 13; the unperformed Pest conversion is not an adoption precondition for that selected direction.

1. Run the complete PHPUnit 12.5 suite with deprecation reporting and remove PHPUnit deprecations first. PHPUnit explicitly recommends this before a major-13 upgrade.
2. Capture the current suite inventory, assertions, groups, skips, warnings, runtime, peak memory, coverage configuration and Codeception compatibility.
3. In one disposable worktree, resolve PHPUnit 13 and update only the test configuration and APIs required by that major. Record dependency conflicts and removed or hard-deprecated behavior.
4. In a second disposable worktree, resolve Pest 5 and run the same existing PHPUnit-style sample before converting any test syntax.
5. Convert a representative 5–10% sample containing unit, integration, temporary-database, data-provider, mock/stub, exception, CLI, HTTP and module lifecycle cases. Preserve behavior, assertions and coverage.
6. Compare authoring/readability review, changed LOC, duplicated setup, focused and full runtime/memory, failure diagnostics, IDE navigation, PHPStan, coverage, parallelism, Windows behavior, Codeception coexistence and the dependency/security surface.

#Historical two-candidate hypotheses and thresholds

Recorded on 2026-08-24, before running either spike. These are the original two-candidate criteria, retained as historical protocol. The owner's later selection of the class-based PHPUnit upgrade uses the explicitly revised method below: complete inventory equivalence plus a sequential representative sample. That method does not establish the original single-process full-suite time and memory budgets.

Original hypotheses:

Original blocking criteria — a candidate failing any was to be rejected regardless of authoring benefit:

• Codeception Api and Acceptance suites still run, with the same result set.
• PHPStan analyses the test tree with no new errors.
• Line coverage for the converted sample stays within ±1% of the 12.5 baseline.
• Full-suite wall time regresses by no more than 15%; peak memory by no more than 20%.
• No converted test loses an assertion or changes behaviour.
• The main workspace's local Composer lock is untouched until a candidate is chosen; composer.lock is gitignored and is not a committed artifact.

Original adoption thresholds:

• Pest 5 is adopted only if it clears every blocking criterion and shows a material authoring benefit on the converted sample: at least a 25% reduction in test lines of code together with a reviewed readability win. A smaller or purely stylistic gain is not sufficient to justify a second authoring surface and a migration schedule.
• PHPUnit 13 is adopted when its major upgrade clears the blocking criteria and Pest 5 does not reach its adoption threshold.
• Neither is adopted if PHPUnit 13 itself fails a blocking criterion; in that case the suite stays on 12.5 and the blocker is recorded with a dated re-evaluation trigger.

#Historical measured snapshot — 2026-08-24/25

The experiment ran in two disposable worktrees of 0.26.5. The main workspace's local Composer lock was not changed. Version and dependency claims in this section describe that snapshot.

#Step 1 — PHPUnit deprecations cleared

All 13 PHPUnit deprecations in the 12.5 suite had a single cause: isType('string') and isType('int'), both removed in PHPUnit 13. They were replaced with isString() / isInt() (12 occurrences in 5 test files). The suite then reports zero PHPUnit deprecations on 12.5, with the assertion count unchanged at 15 391.

This step was mandatory, not cosmetic: the same unfixed file run under PHPUnit 13.0.6 produces 7 errors.

#Step 2 — PHPUnit 13, same code, same machine

1. The 12 missing assertions are two Python-worker tests skipped because the disposable worktree has no python/.venv (it is gitignored). Linking the venv makes that file report 3 tests / 13 assertions with no skips. Not a PHPUnit 13 effect.
2. Pre-existing failures unrelated to the test stack; identical on both sides.
3. All 19 share one cause — “Using with*() without expects() is deprecated and will no longer be possible in PHPUnit 14” — 21 occurrences across 19 tests. Forward-looking work for PHPUnit 14, not a PHPUnit 13 blocker.

Runtime and memory are equal within noise. There is no measured performance argument in either direction.

#Step 3 — dependency reachability

Codeception 5.3.5 accepts phpunit/phpunit ^13.0 but caps phpunit/php-code-coverage at ^13.0. Every PHPUnit from 13.1 onward requires php-code-coverage ^14, and every Pest 5 release requires PHPUnit ≥ 13.2.4. In that snapshot, Codeception pinned the whole stack to the first PHPUnit 13 line, three minors behind the then-current stable release, and excluded Pest 5.

Removing codeception/codeception alone does not lift the cap: the three codeception/module-* packages pull it back in transitively. A full composer update with Pest 5 declared resolves only by removing Codeception and 18 dependencies, including the Api and Acceptance suite infrastructure.

#Step 4 — the sample conversion was not performed

Pest 5 was eliminated at dependency resolution, before authoring quality could matter. Converting a 5–10% sample would have measured a candidate already rejected by a pre-registered blocking criterion. This is a deliberate deviation from the experiment order, recorded here so it is not mistaken for an omission.

#Criterion recorded as not applicable

“PHPStan analyses the test tree with no new errors” cannot be evaluated as written: phpstan.neon targets src only, so the test tree is not analysed at all today. Extending PHPStan to tests/ is separate work and is not a precondition for this decision.

#Former product decision gate — the Codeception suites were empty

The evaluation surfaced a fact that changes what the decision is about.

Codeception executed zero tests. tests/Acceptance/Cest/ and tests/Api/Cest/ contained only a .gitkeep; no Cest file had ever been committed, so none were lost. The stored reports/codeception.json recorded "tests": 0, "assertions": 0 as far back as 2026-04-07.

Two consequences followed at the time:

• composer qa:test was documented as a blocking CI gate and passed vacuously.
• Codeception was nevertheless the single constraint capping PHPUnit at 13.0.x and excluding Pest 5.

The pre-registered blocking criterion “the Codeception Api and Acceptance suites still run, with the same result set” is therefore satisfied by any candidate, because the result set is empty. Applying it literally to reject Pest 5 would be formally correct and substantively misleading, so the verdict is deliberately left open.

The options presented to the product owner in that dependency snapshot were:

The owner selected removal on 2026-09-13 and authorized evaluating updated PHPUnit 13. The blocking question is answered; it must not be requested again.

#Mock expectation compatibility — 2026-09-13

The optional forward-compatibility cleanup is delivered independently of the Codeception decision. An AST inventory of all 662 PHP files under tests/ identified 15 argument-constrained mock configurations lacking expects() in five files. They execute 21 times across 19 tests because the breadcrumb test configures three resolvers and the Turnstile helper is reused by seven tests.

Each now requires expects(self::once()), preserving its original with() constraints and return value. The call paths confirm one settings, ownership, active-task or resolver-support lookup per tested operation. The Turnstile helper is named expectSecretKeyRead() to make its verification role explicit. This follows the PHPUnit mock contract. No diagnostic is suppressed, and no framework dependency, configuration or runtime implementation is changed.

Focused evidence on PHP 8.5.10, using the same five complete test classes:

All four runs exit successfully with no errors, failures, skips or risky tests. The two final runs also enable --fail-on-phpunit-deprecation. The 19 added assertions are mock verifications: of the 21 newly explicit invocation counts, two belong to mocks already verified by another expectation (the unsupported breadcrumb resolver and the same-user email update). Existing value assertions are preserved. The 31 pre-existing notices concern other mocks without expectations and remain visible; this is not a claim that the full suite is diagnostic-free or that either framework has been adopted.

Reproduce the focused baseline runner with:

php vendor/bin/phpunit \
  --bootstrap tests/Support/isolated-tenant-bootstrap.php \
  --order-by=default --fail-on-phpunit-deprecation \
  --display-phpunit-deprecations --display-phpunit-notices \
  tests/Unit/Breadcrumb/BreadcrumbManagerTest.php \
  tests/Unit/Application/Captcha/CaptchaServiceTest.php \
  tests/Unit/Infrastructure/Captcha/CloudflareTurnstileProviderTest.php \
  tests/Unit/Application/Auth/UserSettingsServiceTest.php \
  tests/Unit/Module/Ai/AiChatControllerTest.php

Repeat with vendor/bin/phpunit replaced by an external copy of https://phar.phpunit.de/phpunit-13.0.6.phar. Its measured SHA-256 is b3625c5f81f2a86a2c0661b2b4306e077c4882d72e1b4b45cdf4cfe5830ee205. The PHAR stays outside the repository and the Composer lock remains unchanged. Both runners use the disposable tenant bootstrap; the tested repositories, HTTP clients and Redis interactions are mocks or in-memory fixtures.

Final review found all 115 with*() sites declare expects(), with no detached builder chains. PHP lint and git diff --check pass. The scoped PHP-CS-Fixer check still reports formatting differences in the Auth and Ai test files; comparison against HEAD confirms identical pre-existing differences and no new style violation. PHPStan's configured src scope does not cover this test-only change. The focused runs cover all modified classes; this increment does not repeat the historical full-suite or adoption experiment.

#PHPUnit 13.3.3 comparison and decision — 2026-09-13

The completed experiment compares PHPUnit 12.5.35 with 13.3.3 on PHP 8.5.10. The selected class-based upgrade introduces no new failure in the complete inventory, preserves the measured coverage and static-analysis results, and shows no material regression in the sequential sample. Local report paths below identify evaluation artifacts, not committed fixtures.

reports/p9-dependency-delta.json records 16 removed packages, two added and 19 updated. All nine Codeception packages leave the graph. The candidate updates phpunit/php-code-coverage from 12.5.7 to 14.3.3 and adds sebastian/file-filter and sebastian/git-state. justinrainbow/json-schema 6.12.0 remains an explicit development dependency because Discovery's provider schema test uses it; its marc-mabe/php-enum dependency also remains. These measured counts supersede the earlier hypothetical removal count. The versioned dependency declaration is composer.json; composer.lock remains local and gitignored.

The focused coverage snapshot runs the same nine complete classes on both runners: bootstrap isolation, integration-command execution, PHPUnit reports, the five mock-cleanup classes and Discovery preference interpretation. Both pass 63 tests / 324 assertions, with zero errors, failures, skips or PHPUnit deprecations. The same 49 mock-notice events affect 31 tests.

Source hashes are identical. No shared executable line loses coverage: all 661 surviving covered statement identities match after worktree paths are normalized. Coverage 14 removes 80 structural match endings and two constant match headers; three removed endings were previously counted as covered. It also exposes one previously unreported, uncovered ternary branch in IntegrationClientRegistryService.php:270. AST inspection and the installed coverage analyzer confirm these classification changes. The worktree reports are archived under reports/p9-stack-comparison/{baseline,candidate}/; p9-focused-coverage.xml records coverage and the candidate's p9-focused-coverage-comparison.json records every line delta. This is sample coverage, not whole-application coverage or a module coverage claim.

Full PHPStan analysis of its configured src scope reports 438 findings on each side, with identical file, line, message and diagnostic identifiers after normalizing worktree prefixes in both filenames and messages. No new finding is attributed to the candidate; the existing findings remain open. The test tree is outside the configured PHPStan scope.

The full-inventory comparison completed in four disjoint class shards per runner, covering 630 classes / 3,094 cases (774, 774, 773 and 773). Both sides received the same final application and test files, separate copied vendor and frontend build trees, and the same linked Python virtual environment. Each worker owned its temporary directory, tenant database and blocked Redis endpoint. An earlier baseline attempt was discarded after its test keyring proved misconfigured; it is not comparison evidence. Both sides used the canonical bootstrap's deterministic keyring and the corrected SMTP ciphertext/readback assertion.

Every test identity, status and assertion count matches, with no missing or extra case. Failure details also match after normalizing worktree prefixes and the random ciphertext generated by enc:v2. Notice and warning event details match after normalizing private paths. The warnings come from existing HTML/DOM parsing and unlink calls using @; no new suppression was added.

The four failures remain visible on both runners:

These are pre-existing test-contract mismatches, not newly introduced runner failures. Their later assertions did not execute, so this comparison cannot claim that the full suite passes. The evaluation's comparison.json preserves the case comparison and full-results.json preserves each worker's measurements:

RSS is the operating system's peak resident memory for each worker. Eight workers ran concurrently; these observations do not establish a speed win. Sharding checks the complete case inventory but changes process boundaries and cross-class ordering. This is an explicit deviation from the original method: full-inventory equivalence checks behavior, while a separate sequential representative sample checks timing and memory. Neither concurrent shard wall time nor summed memory can satisfy the original single-process full-suite performance thresholds or establish equivalent cross-class ordering.

The sequential sample completed three paired repetitions, alternating which runner starts first. All six runs pass 70 cases / 408 assertions. It includes the focused unit/CLI/schema/report classes and the isolated mail HTTP integration class; reports/p9-sequential-timing.py records its method and the evaluation's timing-results.json contains every measurement.

The candidate's median wall time is 0.61% higher; maximum peak RSS is 0.77% higher. These small differences do not establish a performance win or a material regression for this workload. They do not prove the original full-suite budgets. Measurements were taken on Linux; Windows execution was not exercised, and the sharded comparison does not validate single-process cross-class ordering.

#Adopted direction and verification boundary

Adopt PHPUnit 13.3.3, retain the existing class-based suite, and remove Codeception's empty suites and dependency family. The decision rests on the explicitly selected method: complete-inventory equivalence, focused coverage, configured PHPStan equivalence and sequential representative timing. It does not assert that the historical full-suite performance thresholds were proved. The four failures and diagnostics recorded below describe the comparison-time baseline. The subsequent 0.29.0 reconciliation ran the complete PHP inventory as one Unit suite, 79 individually isolated Integration files and a separate Seed suite; all tests pass. That verification does not claim a single-process Application run or remove the existing mock notices.

The main workspace now has PHPUnit 13.3.3, coverage 14.3.3 and JSON Schema 6.12.0 installed, with no Codeception package or codecept binary. Its local lock was copied from the tested candidate and its content hash refreshed; the package and development-package entries remain identical to the evaluated graph.

Main-workspace composer qa:test passes the ten-class sample with 70 tests / 408 assertions, fresh reports/phpunit.xml and reports/phpunit.json, and zero PHPUnit deprecations. Its 31 existing notice-affected tests remain visible in reports/p9-main-qa.txt. The direct Composer integration alias also passes the selected mail-default test (one test / seven assertions), captured in reports/p9-main-integration-alias.txt. Both commands complete with COMPOSER_PROCESS_TIMEOUT=1, verifying that long-running test execution is not cut off by Composer's timeout.

Composer audit reports no advisories. Strict manifest validation exits 1 for the same two pre-existing warnings on baseline and main: the explicit version field and aldas/modbus-tcp-client wildcard. There is no stale-lock warning. The scoped style check passes for eight infrastructure and fixture files. These focused checks verify installation and QA wiring; they do not replace the complete inventory comparison or remove its four existing failures.

Reproduce the final focused selection with:

composer qa:test -- --order-by=default --fail-on-phpunit-deprecation \
  tests/Unit/Database/IsolatedTenantBootstrapTest.php \
  tests/Unit/Command/TestIntegrationCommandTest.php \
  tests/Unit/Quality/PhpunitReportsTest.php \
  tests/Unit/Application/Auth/UserSettingsServiceTest.php \
  tests/Unit/Application/Captcha/CaptchaServiceTest.php \
  tests/Unit/Breadcrumb/BreadcrumbManagerTest.php \
  tests/Unit/Infrastructure/Captcha/CloudflareTurnstileProviderTest.php \
  tests/Unit/Module/Ai/AiChatControllerTest.php \
  tests/Unit/Module/Discovery/PreferenceInterpreterTest.php \
  tests/Integration/MailSettingsApiTest.php

The complete local evidence is archived in reports/p9-stack-comparison/, including discovery inventories, shard assignments, per-case results, dependency snapshots, coverage, PHPStan and timing. The archived scripts record the original disposable-worktree paths; recreate those isolated projects before repeating the comparison. Normal composer qa:test still runs both configured suites in one process and reports the existing failures.

Reopen Pest 5 only when its measured benefit is material and there is no significant coverage, debugging, static-analysis, runtime or cross-platform regression. If selected, keep one PHPUnit-13-compatible configuration and use a bounded migration schedule; do not leave an indefinite mixed-style suite.

Do not select Pest 4, do not compare only trivial tests, and do not infer a speed or readability win from runner output alone.

#Test-tree PHPStan scope — 2026-09-14

The production phpstan.neon still analyses src at level 5 with strict rules. A separate complete-tree probe of tests/ found 942 findings in 179 files under those rules: 576 staticMethod.dynamicCall findings mostly reflect instance-style PHPUnit assertions, and 177 nullsafe.neverNull findings. A second probe at level 5 without the production strict-rule include found 288 findings in 85 files; 177 are nullsafe.neverNull. Neither result is suppressed or presented as a passing whole-tree gate.

phpstan-tests.neon starts with six related unit-test directories: architecture, access control, database, HTTP errors, kiosk and OPC UA. These contain 40 PHP files and pass at level 5 with zero findings. composer qa:phpstan:tests writes reports/phpstan-tests.json and runs before the existing production PHPStan step in the aggregate QA scripts. The dedicated config keeps the test gate independent of production strict-style findings and uses its own cache. It does not run PHPUnit or access a tenant database.

Reproduce the full test-tree level-5 inventory with vendor/bin/phpstan analyse --configuration=phpstan-tests.neon tests --error-format=json --no-progress; passing tests overrides the six configured paths for this advisory command. Expand the blocking test gate by one related directory at a time only after that directory reaches zero genuine findings, its focused PHPUnit tests pass with the disposable bootstrap, and the existing 40-file gate remains green. Review the full inventory again after each expansion. Do not generate ignore-baseline entries or relax production analysis to absorb test-only debt. The measured 288-finding inventory is the starting point for such bounded fixes, not a promise that the whole tree is clean.

#Scheduling

The owner authorized this isolated test-stack change on 2026-09-13. It does not reopen P4 cleanup, P5 settings migration or signage. Re-evaluate Pest 5 on 2027-02-05, or earlier only for a concrete authoring/maintenance problem that justifies repeating the representative conversion and its original thresholds. No performance or readability win is inferred from dependency reachability.

#Primary references

• PHPUnit 13 announcement and upgrade cautions: https://phpunit.de/announcements/phpunit-13.html
• Pest support policy: https://pestphp.com/docs/support-policy
• Pest 5 upgrade guide: https://pestphp.com/docs/upgrade-guide/
• Pest installation: https://pestphp.com/docs/installation