Design Assay Bench v0 (draft)
An open evaluation of how well models produce production-grade design. The bench reuses the fitness functions from 02-FITNESS-FUNCTIONS.md, so a model's score means the same thing as an assay score.
Why
Model progress follows what can be measured. Code has strong benchmarks. Design has screenshots and opinions. The bench gives labs and builders a reproducible way to measure design ability.
Task suites
| Suite | Input | Output | Count (v0) |
|---|---|---|---|
S1 Screen |
Target profile plus genome | One production screen in Next.js and Tailwind | 30 |
S2 Extend |
Existing design system plus a new component brief | Component that matches the system | 30 |
S3 Repair |
A screen with seeded defects | Fixed screen | 30 |
S4 System |
Brand brief | Token set plus 5 core components | 10 |
Scoring
- Every output runs through the same gates, measured and judged functions.
- Bench score per suite is the mean composite across tasks.
- Headline score is the mean across suites, with gate pass rate reported alongside.
Integrity
- A public dev set and a held-out test set. The test set is never published.
- Seeded defects in
S3are rotated each release. - Judge models are calibrated per release and listed in the results.
- Every submission includes model name, version, date, prompt and harness version.
Release rules
- The bench opens publicly only after at least two published assays have validated the fitness functions.
- Results page lists every model run, including poor results. No cherry-picking.
Harness
- Runs locally or in CI with Playwright and axe-core.
- One command per suite:
assay bench run --suite S1 --model <id>. - Outputs a run record in the same format as
01-PROTOCOL.md.