Design Quality Benchmark (Beta)

How the provisional Design Quality assessment works, what it measures, and what it does not.

Draft / Beta. Criteria, weights, and thresholds are provisional. Calibration against independent expert review is in progress. Scores are versioned and meaningful only relative to their benchmark version.

What Mateo measures

Mateo evaluates the observable execution quality of a specific web page - the pages, states, and viewports it actually inspected - across seven areas: visual hierarchy, layout & spacing, typography, components & consistency, accessibility, color, and motion & interaction. Some findings are grounded in formal standards (WCAG 2.2). Others use research-informed or expert-defined criteria, some informed by ISO 9241 interaction principles without claiming clause-level conformance. Every criterion's source basis is recorded in the benchmark registry, and every deduction carries evidence and provenance (screenshot, extracted CSS, or both).

What Mateo does not measure

The assessment is not a measure of product-market fit, content strategy, conversion, task success, brand reputation, or aesthetic preference as a universal truth. It does not certify WCAG conformance: automated, page-scoped evidence supports human evaluation but cannot by itself determine that content is accessible. It says nothing about routes, states, or devices that were not inspected, and a high result is not proof of usability outcomes.

How evidence is collected today

  • Computed CSS and design-token extraction from the live page
  • Full-page and scroll screenshots at a desktop viewport, plus a 375px mobile capture with measured horizontal overflow
  • A real keyboard focus pass: genuine Tab key events with rendered-style diffs, so native focus rings and suppressed outlines are observed, not guessed
  • An accessibility-tree snapshot: programmatic headings, landmarks, and control/image naming coverage
  • Rendered bounding boxes for sampled interactive targets, evaluated against the WCAG 2.5.8 24x24 CSS px minimum and its spacing exception
  • A live runtime animation sample (real timing from the Web Animations API) and pointer hover sampling

Planned and labeled as such until implemented: tablet and intermediate viewports, zoom and text-spacing fixtures, interaction-state discovery (disabled, selected, loading, error, empty), reduced-motion testing, RTL and localization fixtures. When required evidence is missing, the affected finding is unknown - never silently assumed to pass.

The four outputs

  1. Design Quality - the provisional estimate of observed execution quality (0-100).
  2. Evidence coverage - how much of the required evidence was actually collected.
  3. Likely range - a deterministic interval: the low end resolves every unknown against the page, the high end resolves them for it. Incomplete evidence widens the range; it never inflates the score.
  4. Critical issues - confirmed failures on normative-critical criteria (contrast, keyboard focus, target size, image alternatives), listed separately so they can never disappear into an average.

These are never collapsed into a single number.

Finding statuses

Pass is earned with affirmative evidence, never a default. Partial means met with a concrete exception. Fail is a supported material weakness. Unknown means required evidence is missing - it lowers coverage and widens the range, and can only ever hurt a result. Not applicable is used only when a construct genuinely does not apply; it never substitutes for positive evidence, so a simple page is not penalized for simplicity but also cannot earn credit for mastery it never demonstrated.

Why 100 is exceptionally rare

100 means a verified ceiling against this benchmark version: every applicable criterion passed with affirmative evidence, coverage is complete, nothing is unknown, and no critical finding exists. It does not mean perfect design - it means nothing inspectable was found wanting under this version's criteria.

Limitations

Evidence is bounded by the captures listed above; judgment-based criteria carry model variance; category weights, status points, and caps are expert-proposed and uncalibrated until the formal calibration program (independent expert panels, seeded-defect fixtures, and a held-out validation corpus) completes. Documents assessed before this Beta used older methodologies and are labeled accordingly.