benord-labs/frink is a project where AI coding agents operate based on tickets, errors, and schedules, and before this change it provided open-source, local automation that asks about approval first. The October 26, 2026 commit b5b3178 adds a common test grid that validates five providers (Claude Code, Codex, OpenCode, GitHub Copilot, Cursor) against the same scenario, while also cleaning up the problem of permission approvals leaking between test cases.
Each row of the grid passes through handleRemoteExecute, through which real chat and Flow stages run, and the row names follow the format conformance: <scenario> x <runtime>, so the results read as a conformance matrix. There are only two scenarios. A simple turn, in which a text response reaches the chat and the turn runs to completion, passes for all five providers, but the scenario in which the editor tab refreshes after writing a file passes only for Claude. This is because only Claude's turn loop sends file-changed, and because the edit card for Codex does not carry file_path. The current gaps are enumerated in conformance/baseline.ts and marked with vitest's fails option; since both a failure that is not on the list and a success that is on the list would break the test, the list is prevented from only shrinking.
The same commit also corrected the state of the earlier tests. Of the 490 cases in executor.test.ts, 237 had a permission card left pending by an earlier case, and 479 inherited the card sender installed by an earlier case. Of those, 470 were the auto-approval implementation. The reason was vi.clearAllMocks, which left the implementation in place even after deletion. Now every case starts with no pending card and both numbers are 0. The drain test was also fixed, so that the check runs inside the fake CLI and exceptions are no longer swallowed as failed turns.
That said, this change does not touch product behavior. Since there are not yet any permitted write runs recorded with the providers' actual binaries, all fixtures are scripts and are distinguished only as RECORDED and SCRIPTED. The unit suite runs only on Linux CI and is not Windows validation. When the editor-refresh effect is moved out of Claude's loop into shared code in the next PR, the remaining rows are expected to switch to passing. As of now, the editor tab refresh after a file write reaches only the Claude path, and the grid does not hide that difference but exposes it.