ADR 0014: Risk-based testing strategy

Decision: Automate the deterministic and security-critical surfaces (conversion pipeline, SVG sanitisation, file path safety); validate visual UX manually. Why: Line coverage and bug coverage are weakly correlated — over-testing the visual layer produces ceremony tests that catch shape changes, not behaviour.

Context

The test suite expresses what kind of failures the architect is willing to ship, and which kinds they want to be told about before they reach a user. For Vektory, those choices were made deliberately, shaped by the part of my background most relevant to this question: I have spent years in QA before pivoting to architecture.

A monochrome PNG-to-SVG converter has three surfaces where bugs are expensive and detection delays them:

  1. The conversion pipeline. Sharp preprocessing + Potrace tracing + pathBounds geometry. Pure deterministic transforms; output should be byte-stable for a given input. A regression here ships visibly broken SVGs, the entire product output is wrong.
  2. The security boundary. SVG sanitisation against XSS payloads (ADR 0002), file path validation against traversal/null-byte/DoS attacks. A regression here is a security failure with privileged-process consequences.
  3. The visual UX. Gradients, spacing, slider feel, drag-and-drop, dark/light theme integrity, batch progress. None of this is unit-testable in any meaningful way without ceremony tests that catch shape changes rather than behaviour. Visual UX is validated by looking at it. And let’s be honest, sometimes something mathematically completely in the middle, but for a human eye - it looks misaligned, just because of the shape of the logo.

These three surfaces want three different testing strategies. A uniform approach (e.g. “everything has unit tests”) would over-test the third and under-test the first.

Options considered

Decision

Vitest as the test framework. Three test suites covering the surfaces where automation pays the most:

  1. ConversionService.test.ts: pure-function unit tests for pathBounds plus integration tests using real Sharp and real Potrace against generated test images.
  2. svgSanitizer.test.ts: XSS attack-payload tests in a jsdom environment.
  3. FileService.test.ts: security-pattern tests against path traversal, null-byte injection, and DoS-by-length, plus directory-sandboxing logic.

Plus format.test.ts for trivial utility coverage. Total: 56 test cases.

Coverage instrumentation is configured to report only on these three modules (ConversionService.ts, svgSanitizer.ts, format.ts).

Manual testing is the strategy for everything else: UI flows, drag-and-drop, theme switching, batch UX feel, real-image variety beyond the synthetic test fixtures, cross-platform installation behaviour.

Rationale

Why risk-based, not coverage-based

The QA-trained instinct is that bugs do not distribute uniformly across a codebase. They cluster at boundaries, at parameter handling, at I/O, at concurrency, at security surfaces, and they avoid getters, framework wiring, and pure rendering of static state. A test budget spent on the second category produces test files that turn green on every commit and catch nothing. The same budget spent on the first category catches real regressions.

For Vektory, the bug-prone surfaces are easy to name because they are documented in other ADRs. The Sharp preprocessing pipeline order (ADR 0012) was empirically reverse-engineered from three independent failure modes; a regression that re-introduces one of those bugs would silently produce wrong output. The cubic-Bezier extrema math (ADR 0010) replaced a convex-hull approximation that over-estimated viewBoxes; a regression to convex hull would silently re-introduce the wrong-icon-size bug. The renderer SVG sanitiser is the second defensive layer of ADR 0002’s threat model; a regression here is a security failure.

Each of those is exactly the kind of bug that re-occurs without a regression test. So that is where the tests are.

Tests as regression contracts on architectural decisions

The test suite is structured around the ADRs:

Essentially: the ADRs say what the architecture decided, the tests prove it stays decided.

Why integration tests use real Sharp and real Potrace

The integration tests in ConversionService.test.ts generate real PNG fixtures with Sharp itself (a 50×50 white square with a 20×20 black centre, and the inverse) and feed them through the actual pipeline. No mocking of Sharp; no mocking of Potrace.

The QA-known reason: when the bugs being defended against are bugs in how those libraries compose (alpha behaviour, threshold semantics, byte-perfect input shape), mocked tests verify the mock, not the real behaviour.

Why Vitest

The project is already Vite-based for the renderer (ADR 0004). Vitest reuses the same config, the same module resolver, the same TypeScript setup, the same alias paths etc. For a single-developer codebase already on Vite, Vitest is the path of least configuration that gives the most actual testing capability.

Why no E2E, no visual regression, no CI matrix, for now

Each of these has a real future-cost trade-off:

Consequences

Positive

Negative

Follow-up

Code: vitest.config.ts · src/main/services/ConversionService.test.ts · src/main/services/FileService.test.ts · src/renderer/utils/svgSanitizer.test.ts · src/renderer/utils/format.test.ts