Benchmark · 2026-10-12

Coding agents with and without onsystem

In the run of 2026-10-12, design-system rule violations per run went from 0.85 to 0.00 with onsystem's skill, MCP server and hook, with no change in tsc pass rate or cost per run (1,200 runs, 6 models, 3 providers, 5 repositories).

Each model wrote the same 20 UI components in five open-source repositories, once with no onsystem and once with each of its parts: the skill, the MCP server, the hook, and all three. A rule violation is an error from onsystem check that the run added to the repository. tsc type-checks each file in its project's own setup, and a sample was labelled blind by an AI reviewer.

No onsystem → skill, MCP server and hook, all 6 models pooled, with 95% intervals.

Rule violations per run: 0.85→0.00; difference −0.85 [−1.38, −0.40]; without onsystem [0.40, 1.38]
tsc passes: 93%→92%; difference 0 points [−4, +3]
Cost per run: $0.0464→$0.0454; without onsystem [$0.0432, $0.0497]

Results

Every arm, pooled over the six models. Rates have 95% Wilson intervals; means and differences have 95% intervals from a bootstrap that resamples tasks. Pooled by tier, rule violations per run went from 0.92 to 0.00 for the small models and from 0.62 to 0.00 for the mid models.

Results by arm, pooled over all 6 models, with 95% intervals
Arm	Runs	Rule violations / run	onsystem clean	`tsc` passes	Cost / run	Hook blocks / run	MCP calls / run
noneNo onsystem	240	0.85 [0.40, 1.38]	64% 153/240 [57%, 70%]	93% 222/240 [88%, 95%]	$0.0464	0.00	0.0
skillSkill	240	0.42 [0.17, 0.72]	76% 183/240 [70%, 81%]	91% 218/240 [87%, 94%]	$0.0486	0.00	0.0
mcpMCP server	240	0.01 [0.00, 0.04]	93% 224/240 [89%, 96%]	91% 218/240 [87%, 94%]	$0.0467	0.00	7.6
hookHook	240	0.05 [0.01, 0.10]	93% 223/240 [89%, 96%]	92% 220/240 [87%, 95%]	$0.0508	0.35	0.0
allSkill, MCP server and hook	240	0.00 [0.00, 0.00]	95% 229/240 [92%, 97%]	92% 221/240 [88%, 95%]	$0.0454	0.11	7.5

Each arm against no onsystem

Paired on model, task and repeat. Accepted means the file was created, passes tsc and has no onsystem error.

Each arm against no onsystem, paired on model, task and repeat, all models, with 95% intervals
Arm	Pairs	Δ rule violations / run	Δ onsystem clean (points)	Δ `tsc` passes (points)	Δ accepted (points)
skillSkill	240	−0.42 [−0.69, −0.20]	+13 [+5, +20]	−2 [−5, +1]	+12 [+4, +20]
mcpMCP server	240	−0.83 [−1.35, −0.40]	+30 [+15, +45]	−2 [−5, +2]	+28 [+14, +42]
hookHook	240	−0.80 [−1.29, −0.38]	+29 [+16, +44]	−1 [−4, +2]	+27 [+14, +41]
allSkill, MCP server and hook	240	−0.85 [−1.38, −0.40]	+32 [+17, +48]	0 [−4, +3]	+29 [+15, +44]

Every model and arm (45 rows)Show

Results for every model and arm, then each tier and all models pooled, with 95% intervals
Model	Arm	Runs	Rule violations / run	onsystem clean	`tsc` passes	Accepted	Created	Cost / run	Calls / run	Hook blocks / run	MCP calls / run
Claude Haiku 5.5	none	60	0.58 [0.20, 1.07]	70% 42/60 [57%, 80%]	97% 58/60 [89%, 99%]	68% 41/60 [56%, 79%]	100%	$0.0041	6.2	0.00	0.0
Claude Haiku 5.5	skill	60	0.13 [0.02, 0.27]	88% 53/60 [78%, 94%]	100% 60/60 [94%, 100%]	88% 53/60 [78%, 94%]	100%	$0.0045	8.0	0.00	0.0
Claude Haiku 5.5	mcp	60	0.05 [0.00, 0.15]	98% 59/60 [91%, 100%]	93% 56/60 [84%, 97%]	92% 55/60 [82%, 96%]	100%	$0.0045	6.5	0.00	7.5
Claude Haiku 5.5	hook	60	0.12 [0.00, 0.32]	95% 57/60 [86%, 98%]	98% 59/60 [91%, 100%]	93% 56/60 [84%, 97%]	100%	$0.0044	6.8	0.32	0.0
Claude Haiku 5.5	all	60	0.00 [0.00, 0.00]	100% 60/60 [94%, 100%]	97% 58/60 [89%, 99%]	97% 58/60 [89%, 99%]	100%	$0.0041	5.9	0.05	7.0
Claude Sonnet 5.5	none	20	0.40 [0.00, 0.90]	85% 17/20 [64%, 95%]	100% 20/20 [84%, 100%]	85% 17/20 [64%, 95%]	100%	$0.0375	5.3	0.00	0.0
Claude Sonnet 5.5	skill	20	0.15 [0.00, 0.40]	90% 18/20 [70%, 97%]	100% 20/20 [84%, 100%]	90% 18/20 [70%, 97%]	100%	$0.0476	6.4	0.00	0.0
Claude Sonnet 5.5	mcp	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	95% 19/20 [76%, 99%]	95% 19/20 [76%, 99%]	100%	$0.0403	4.6	0.00	3.5
Claude Sonnet 5.5	hook	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100%	$0.0394	5.5	0.20	0.0
Claude Sonnet 5.5	all	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	95% 19/20 [76%, 99%]	95% 19/20 [76%, 99%]	100%	$0.0425	4.2	0.10	4.0
Gemini 3.5 Flash-Lite	none	60	1.48 [0.70, 2.38]	60% 36/60 [47%, 71%]	95% 57/60 [86%, 98%]	58% 35/60 [46%, 70%]	100%	$0.0607	22.1	0.00	0.0
Gemini 3.5 Flash-Lite	skill	60	1.07 [0.40, 1.82]	68% 41/60 [56%, 79%]	93% 56/60 [84%, 97%]	65% 39/60 [52%, 76%]	98%	$0.0617	22.9	0.00	0.0
Gemini 3.5 Flash-Lite	mcp	60	0.00 [0.00, 0.00]	97% 58/60 [89%, 99%]	93% 56/60 [84%, 97%]	93% 56/60 [84%, 97%]	97%	$0.0638	24.4	0.00	6.3
Gemini 3.5 Flash-Lite	hook	60	0.03 [0.00, 0.10]	98% 59/60 [91%, 100%]	93% 56/60 [84%, 97%]	93% 56/60 [84%, 97%]	100%	$0.0710	25.1	0.48	0.0
Gemini 3.5 Flash-Lite	all	60	0.00 [0.00, 0.00]	100% 60/60 [94%, 100%]	93% 56/60 [84%, 97%]	93% 56/60 [84%, 97%]	100%	$0.0575	22.4	0.32	5.6
Gemini 3.8 Flash	none	20	0.90 [0.05, 1.90]	15% 3/20 [5%, 36%]	35% 7/20 [18%, 57%]	15% 3/20 [5%, 36%]	35%	$0.2716	37.3	0.00	0.0
Gemini 3.8 Flash	skill	20	0.40 [0.00, 1.20]	15% 3/20 [5%, 36%]	20% 4/20 [8%, 42%]	15% 3/20 [5%, 36%]	20%	$0.2724	38.1	0.00	0.0
Gemini 3.8 Flash	mcp	20	0.00 [0.00, 0.00]	35% 7/20 [18%, 57%]	35% 7/20 [18%, 57%]	35% 7/20 [18%, 57%]	35%	$0.2537	38.0	0.00	10.5
Gemini 3.8 Flash	hook	20	0.10 [0.00, 0.30]	35% 7/20 [18%, 57%]	30% 6/20 [15%, 52%]	25% 5/20 [11%, 47%]	40%	$0.2891	38.6	0.25	0.0
Gemini 3.8 Flash	all	20	0.00 [0.00, 0.00]	45% 9/20 [26%, 66%]	40% 8/20 [22%, 61%]	40% 8/20 [22%, 61%]	45%	$0.2567	39.0	0.00	11.0
GPT-6 Luna	none	60	0.70 [0.20, 1.32]	68% 41/60 [56%, 79%]	100% 60/60 [94%, 100%]	68% 41/60 [56%, 79%]	100%	$0.0032	6.4	0.00	0.0
GPT-6 Luna	skill	60	0.32 [0.10, 0.57]	80% 48/60 [68%, 88%]	97% 58/60 [89%, 99%]	77% 46/60 [65%, 86%]	100%	$0.0039	7.3	0.00	0.0
GPT-6 Luna	mcp	60	0.00 [0.00, 0.00]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100%	$0.0043	8.7	0.00	9.3
GPT-6 Luna	hook	60	0.00 [0.00, 0.00]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100%	$0.0036	7.2	0.35	0.0
GPT-6 Luna	all	60	0.00 [0.00, 0.00]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100% 60/60 [94%, 100%]	100%	$0.0037	8.9	0.03	9.5
GPT-6.1 Sol	none	20	0.55 [0.15, 1.15]	70% 14/20 [48%, 85%]	100% 20/20 [84%, 100%]	70% 14/20 [48%, 85%]	100%	$0.0433	5.5	0.00	0.0
GPT-6.1 Sol	skill	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100%	$0.0533	6.3	0.00	0.0
GPT-6.1 Sol	mcp	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100%	$0.0486	6.3	0.00	8.2
GPT-6.1 Sol	hook	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	95% 19/20 [76%, 99%]	95% 19/20 [76%, 99%]	100%	$0.0442	6.2	0.25	0.0
GPT-6.1 Sol	all	20	0.00 [0.00, 0.00]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100% 20/20 [84%, 100%]	100%	$0.0491	6.5	0.00	8.8
mid models	none	60	0.62 [0.12, 1.25]	57% 34/60 [44%, 68%]	78% 47/60 [66%, 87%]	57% 34/60 [44%, 68%]	78%	$0.1175	16.0	0.00	0.0
mid models	skill	60	0.18 [0.00, 0.52]	68% 41/60 [56%, 79%]	73% 44/60 [61%, 83%]	68% 41/60 [56%, 79%]	73%	$0.1244	17.0	0.00	0.0
mid models	mcp	60	0.00 [0.00, 0.00]	78% 47/60 [66%, 87%]	77% 46/60 [65%, 86%]	77% 46/60 [65%, 86%]	78%	$0.1142	16.3	0.00	7.4
mid models	hook	60	0.03 [0.00, 0.10]	78% 47/60 [66%, 87%]	75% 45/60 [63%, 84%]	73% 44/60 [61%, 83%]	80%	$0.1242	16.8	0.23	0.0
mid models	all	60	0.00 [0.00, 0.00]	82% 49/60 [70%, 89%]	78% 47/60 [66%, 87%]	78% 47/60 [66%, 87%]	82%	$0.1161	16.6	0.03	7.9
small models	none	180	0.92 [0.46, 1.46]	66% 119/180 [59%, 73%]	97% 175/180 [94%, 99%]	65% 117/180 [58%, 72%]	100%	$0.0227	11.6	0.00	0.0
small models	skill	180	0.51 [0.22, 0.83]	79% 142/180 [72%, 84%]	97% 174/180 [93%, 98%]	77% 138/180 [70%, 82%]	99%	$0.0233	12.7	0.00	0.0
small models	mcp	180	0.02 [0.00, 0.05]	98% 177/180 [95%, 99%]	96% 172/180 [91%, 98%]	95% 171/180 [91%, 97%]	99%	$0.0242	13.2	0.00	7.7
small models	hook	180	0.05 [0.00, 0.12]	98% 176/180 [94%, 99%]	97% 175/180 [94%, 99%]	96% 172/180 [91%, 98%]	100%	$0.0263	13.1	0.38	0.0
small models	all	180	0.00 [0.00, 0.00]	100% 180/180 [98%, 100%]	97% 174/180 [93%, 98%]	97% 174/180 [93%, 98%]	100%	$0.0218	12.4	0.13	7.3
all models	none	240	0.85 [0.40, 1.38]	64% 153/240 [57%, 70%]	93% 222/240 [88%, 95%]	63% 151/240 [57%, 69%]	95%	$0.0464	12.7	0.00	0.0
all models	skill	240	0.42 [0.17, 0.72]	76% 183/240 [70%, 81%]	91% 218/240 [87%, 94%]	75% 179/240 [69%, 80%]	93%	$0.0486	13.8	0.00	0.0
all models	mcp	240	0.01 [0.00, 0.04]	93% 224/240 [89%, 96%]	91% 218/240 [87%, 94%]	90% 217/240 [86%, 94%]	94%	$0.0467	14.0	0.00	7.6
all models	hook	240	0.05 [0.01, 0.10]	93% 223/240 [89%, 96%]	92% 220/240 [87%, 95%]	90% 216/240 [86%, 93%]	95%	$0.0508	14.0	0.35	0.0
all models	all	240	0.00 [0.00, 0.00]	95% 229/240 [92%, 97%]	92% 221/240 [88%, 95%]	92% 221/240 [88%, 95%]	95%	$0.0454	13.4	0.11	7.5

Each arm against no onsystem, by model (36 rows)Show

Each arm against no onsystem for every model, each tier and all models, with 95% intervals
Model	Arm	Pairs	Δ rule violations / run	Δ onsystem clean (points)	Δ `tsc` passes (points)	Δ accepted (points)
Claude Haiku 5.5	skill	60	−0.45 [−0.82, −0.15]	+18 [+8, +30]	+3 [0, +10]	+20 [+8, +33]
Claude Haiku 5.5	mcp	60	−0.53 [−0.95, −0.18]	+28 [+13, +45]	−3 [−8, 0]	+23 [+10, +38]
Claude Haiku 5.5	hook	60	−0.47 [−0.80, −0.18]	+25 [+12, +40]	+2 [0, +5]	+25 [+12, +40]
Claude Haiku 5.5	all	60	−0.58 [−1.07, −0.20]	+30 [+13, +48]	0 [0, 0]	+28 [+12, +47]
Claude Sonnet 5.5	skill	20	−0.25 [−0.65, 0.00]	+5 [0, +15]	0 [0, 0]	+5 [0, +15]
Claude Sonnet 5.5	mcp	20	−0.40 [−0.90, 0.00]	+15 [0, +30]	−5 [−15, 0]	+10 [−10, +30]
Claude Sonnet 5.5	hook	20	−0.40 [−0.90, 0.00]	+15 [0, +30]	0 [0, 0]	+15 [0, +30]
Claude Sonnet 5.5	all	20	−0.40 [−0.90, 0.00]	+15 [0, +30]	−5 [−15, 0]	+10 [−10, +30]
Gemini 3.5 Flash-Lite	skill	60	−0.42 [−1.07, +0.02]	+8 [+2, +17]	−2 [−8, +5]	+7 [0, +15]
Gemini 3.5 Flash-Lite	mcp	60	−1.48 [−2.38, −0.70]	+37 [+20, +55]	−2 [−12, +7]	+35 [+18, +53]
Gemini 3.5 Flash-Lite	hook	60	−1.45 [−2.37, −0.67]	+38 [+20, +57]	−2 [−10, +7]	+35 [+18, +52]
Gemini 3.5 Flash-Lite	all	60	−1.48 [−2.38, −0.70]	+40 [+22, +58]	−2 [−13, +8]	+35 [+15, +55]
Gemini 3.8 Flash	skill	20	−0.50 [−1.40, +0.15]	0 [−20, +20]	−15 [−35, +5]	0 [−20, +20]
Gemini 3.8 Flash	mcp	20	−0.90 [−1.90, −0.05]	+20 [−5, +45]	0 [−25, +25]	+20 [−5, +45]
Gemini 3.8 Flash	hook	20	−0.80 [−1.85, +0.05]	+20 [0, +45]	−5 [−25, +15]	+10 [−15, +35]
Gemini 3.8 Flash	all	20	−0.90 [−1.90, −0.05]	+30 [+5, +55]	+5 [−15, +25]	+25 [0, +50]
GPT-6 Luna	skill	60	−0.38 [−0.85, 0.00]	+12 [−7, +30]	−3 [−10, 0]	+8 [−12, +28]
GPT-6 Luna	mcp	60	−0.70 [−1.32, −0.20]	+32 [+13, +52]	0 [0, 0]	+32 [+13, +52]
GPT-6 Luna	hook	60	−0.70 [−1.32, −0.20]	+32 [+13, +52]	0 [0, 0]	+32 [+13, +52]
GPT-6 Luna	all	60	−0.70 [−1.32, −0.20]	+32 [+13, +52]	0 [0, 0]	+32 [+13, +52]
GPT-6.1 Sol	skill	20	−0.55 [−1.15, −0.15]	+30 [+10, +50]	0 [0, 0]	+30 [+10, +50]
GPT-6.1 Sol	mcp	20	−0.55 [−1.15, −0.15]	+30 [+10, +50]	0 [0, 0]	+30 [+10, +50]
GPT-6.1 Sol	hook	20	−0.55 [−1.15, −0.15]	+30 [+10, +50]	−5 [−15, 0]	+25 [+10, +45]
GPT-6.1 Sol	all	20	−0.55 [−1.15, −0.15]	+30 [+10, +50]	0 [0, 0]	+30 [+10, +50]
mid models	skill	60	−0.43 [−0.82, −0.12]	+12 [0, +23]	−5 [−12, +2]	+12 [0, +23]
mid models	mcp	60	−0.62 [−1.25, −0.12]	+22 [+7, +38]	−2 [−10, +7]	+20 [+3, +37]
mid models	hook	60	−0.58 [−1.23, −0.10]	+22 [+7, +37]	−3 [−10, +3]	+17 [+2, +32]
mid models	all	60	−0.62 [−1.25, −0.12]	+25 [+10, +42]	0 [−8, +8]	+22 [+7, +38]
small models	skill	180	−0.42 [−0.69, −0.18]	+13 [+3, +22]	−1 [−4, +3]	+12 [+2, +22]
small models	mcp	180	−0.91 [−1.42, −0.46]	+32 [+17, +48]	−2 [−6, +2]	+30 [+16, +45]
small models	hook	180	−0.87 [−1.35, −0.45]	+32 [+17, +47]	0 [−3, +3]	+31 [+17, +46]
small models	all	180	−0.92 [−1.46, −0.46]	+34 [+18, +51]	−1 [−4, +3]	+32 [+16, +48]
all models	skill	240	−0.42 [−0.69, −0.20]	+13 [+5, +20]	−2 [−5, +1]	+12 [+4, +20]
all models	mcp	240	−0.83 [−1.35, −0.40]	+30 [+15, +45]	−2 [−5, +2]	+28 [+14, +42]
all models	hook	240	−0.80 [−1.29, −0.38]	+29 [+16, +44]	−1 [−4, +2]	+27 [+14, +41]
all models	all	240	−0.85 [−1.38, −0.40]	+32 [+17, +48]	0 [−4, +3]	+29 [+15, +44]

From results.json and results.md; one row per run in runs.csv.

Blind labels

A stratified random sample of 60 files (12 per arm) was labelled blind by an AI reviewer (Claude Opus 5.5). It saw the task, the design system and the file, not the model, the arm or onsystem's verdict, and counted invented APIs, hardcoded values, native elements and unnamed controls. Two of the six models (Claude Haiku 5.5 and Claude Sonnet 5.5) are from the same family as the labeller, so they may share its blind spots. With 12 files per arm, the intervals are wide. Who labelled, what it could see and the rules it applied are in LABELLER.md.

The blind-labelled sample by arm, as the labeller counted it, with 95% intervals
Arm	Files labelled	No violations (labeller)	Violations / file (labeller)	Would merge (labeller)
noneNo onsystem	12	83% 10/12 [55%, 95%]	0.42 [0.00, 1.20]	92% 11/12 [65%, 99%]
skillSkill	12	100% 12/12 [76%, 100%]	0.00 [0.00, 0.00]	100% 12/12 [76%, 100%]
mcpMCP server	12	100% 12/12 [76%, 100%]	0.00 [0.00, 0.00]	100% 12/12 [76%, 100%]
hookHook	12	83% 10/12 [55%, 95%]	0.67 [0.00, 2.15]	100% 12/12 [76%, 100%]
allSkill, MCP server and hook	12	83% 10/12 [55%, 95%]	0.92 [0.00, 2.54]	92% 11/12 [65%, 99%]

Per file, onsystem and the labeller both found a problem in 1, only onsystem in 3, only the labeller in 5 and neither in 51.

Of onsystem's 22 findings on these files, the labeller judged 3 correct, 0 wrong and 19 debatable.

Method

The tasks

20 tasks, 4 per repository, each a new component file a product team would write. Each names something the design system has to answer: a status color, a destructive action, an icon-only button, a dialog, a form control. Every prompt asks the agent to use the project's components and tokens, to create only that file and to run nothing.

The 20 promptsShow

chatbot
- Create components/chat/model-settings-card.tsx exporting ModelSettingsCard: a card with a title, a description, an API key text input, a Save button and a small green 'Connected' status label.
- Create components/chat/rate-limit-alert.tsx exporting RateLimitAlert: an inline red error alert telling the user they were rate limited, showing when they can retry, with a 'Try again' button.
- Create components/chat/message-toolbar.tsx exporting MessageToolbar: a row of icon-only buttons under an assistant message: copy, regenerate, thumbs up and thumbs down.
- Create components/chat/share-chat-dialog.tsx exporting ShareChatDialog: a dialog showing the share link in a read-only input, with an icon-only copy button next to it and a secondary 'Make private' action.
dub
- Create apps/web/ui/links/link-expiry-banner.tsx exporting LinkExpiryBanner: a dismissible warning banner shown when a short link expires within 24 hours, with an 'Extend expiration' button and an icon-only close button.
- Create apps/web/ui/workspaces/delete-workspace-modal.tsx exporting DeleteWorkspaceModal: a confirmation modal for deleting a workspace that asks the user to type the workspace slug to confirm, with a Cancel button and a red, destructive 'Delete workspace' button.
- Create apps/web/ui/analytics/stat-card.tsx exporting StatCard: a small card with a metric label, a large number, and a change indicator that is green for an increase and red for a decrease, with an info icon that shows a tooltip explaining the metric.
- Create apps/web/ui/links/copy-short-link.tsx exporting CopyShortLink: a short link shown in a read-only field next to an icon-only copy button, which shows a 'Copied' state for two seconds after copying.
saas-starter
- Create app/(dashboard)/dashboard/billing/plan-summary-card.tsx exporting PlanSummaryCard: a card showing the current plan name, the monthly price, the next renewal date, a small green 'Active' status label and a 'Manage subscription' button.
- Create app/(dashboard)/dashboard/team/invite-member-form.tsx exporting InviteMemberForm: a form to invite a team member, with a labelled email input, a choice between the Member and Owner roles as radio buttons, and an 'Invite member' submit button.
- Create app/(dashboard)/dashboard/settings/danger-zone-card.tsx exporting DangerZoneCard: a card with a red 'Delete team' heading, a short warning that this removes every member and all data, and a destructive 'Delete team' button.
- Create components/user-menu.tsx exporting UserMenu: a round avatar button showing the user's initials that opens a dropdown menu with 'Settings', 'Billing' and 'Sign out' items, with 'Sign out' shown in red.
openstatus
- Create apps/dashboard/src/components/monitors/monitor-status-card.tsx exporting MonitorStatusCard: a card for one uptime monitor with its name, its URL, a status label that is green when up, red when down and yellow when degraded, the time of the last check, and a 'Pause' button.
- Create apps/dashboard/src/components/incidents/resolve-incident-dialog.tsx exporting ResolveIncidentDialog: a dialog to resolve an incident, with a textarea for the final update, a Cancel button and a 'Resolve incident' button.
- Create apps/dashboard/src/components/status-pages/subscribers-empty-state.tsx exporting SubscribersEmptyState: an empty state for a status page without subscribers, with an icon, a title, a short description and an 'Add subscriber' button.
- Create apps/dashboard/src/components/monitors/region-select.tsx exporting RegionSelect: a labelled select to choose a monitoring region (Amsterdam, Frankfurt, Virginia, Singapore, Sydney), with helper text under it.
shadcn-website
- Create apps/v4/components/feedback-widget.tsx exporting FeedbackWidget: a small card asking 'Was this page helpful?' with icon-only thumbs-up and thumbs-down buttons; after a vote, a textarea for optional comments and a 'Send feedback' button appear.
- Create apps/v4/components/release-badges.tsx exporting ReleaseBadges: a list of component names, each with a status label: 'New' in green, 'Updated' in blue and 'Deprecated' in amber.
- Create apps/v4/components/newsletter-signup.tsx exporting NewsletterSignup: an inline form with an email input and a 'Subscribe' button, muted helper text under it, and a success message in place of the form once it is submitted.
- Create apps/v4/components/shortcuts-table.tsx exporting ShortcutsTable: a table of keyboard shortcuts with an 'Action' column and a 'Keys' column that shows each key as a small styled key label (Search ⌘K, Toggle theme ⌘J, Copy code ⌘C).

The repositories

Repository	Commit	App	Design system	TypeScript
vercel/chatbot	c2f8235	.	shadcn/ui in components/ui: 23 components, 31 color tokens. The v1 benchmark's repository.	5.8.2
dubinc/dub	70c96f9	apps/web	Workspace package @dub/ui, a custom (not shadcn) system: 430 components and icons, no color tokens found, so the color rules stay quiet here and the component, prop and variant rules do the work.	5.4.4
nextjs/saas-starter	6e33e58	.	shadcn/ui in components/ui: 7 components, 40 color tokens. Next.js, Tailwind v4.	5.8.3
openstatusHQ/openstatus	e36446d	apps/dashboard	Workspace package @openstatus/ui (shadcn-based) plus app components: 132 components, 51 color tokens. TypeScript 7.	7.0.2
shadcn-ui/ui	0132174	apps/v4	The shadcn/ui registry (registry/new-york-v4/ui): 68 components, 40 color tokens. The ui.shadcn.com site.	5.9.2

The arms

Each arm adds shipped onsystem files to the same loop, unchanged:

none
What the agent gets on top of the file tools
Nothing. The prompt still asks it to use the project's components and tokens, as v1's did.
skill
What the agent gets on top of the file tools
The Claude Code plugin's SKILL.md, the way Agent Skills work: its name and description in the system prompt and a load_skill tool that returns the whole file. The skill names the MCP tools, which this arm does not have; its other instructions (tokens, variants, native elements, icon buttons) still apply.
mcp
What the agent gets on top of the file tools
The onsystem MCP server (this checkout's dist/cli.js), started over stdio in the run's copy of the repository. Its five tools, with their own descriptions and schemas, as mcp__onsystem__<tool>, and its instructions in the system prompt, as MCP clients do.
hook
What the agent gets on top of the file tools
The plugin's hook script run after every write or edit, with the PostToolUse input Claude Code gives it. On exit 2 its stderr goes back to the model with the tool result; on exit 0 its additionalContext is added as a note. Its message points at get_component and get_tokens, which this arm does not have.
all
What the agent gets on top of the file tools
All three: the Claude Code plugin's setup.

Arm	What the agent gets on top of the file tools
`none`	Nothing. The prompt still asks it to use the project's components and tokens, as v1's did.
`skill`	The Claude Code plugin's `SKILL.md`, the way Agent Skills work: its name and description in the system prompt and a `load_skill` tool that returns the whole file. The skill names the MCP tools, which this arm does not have; its other instructions (tokens, variants, native elements, icon buttons) still apply.
`mcp`	The onsystem MCP server (this checkout's `dist/cli.js`), started over stdio in the run's copy of the repository. Its five tools, with their own descriptions and schemas, as `mcp__onsystem__<tool>`, and its instructions in the system prompt, as MCP clients do.
`hook`	The plugin's hook script run after every write or edit, with the PostToolUse input Claude Code gives it. On exit 2 its stderr goes back to the model with the tool result; on exit 0 its `additionalContext` is added as a note. Its message points at `get_component` and `get_tokens`, which this arm does not have.
`all`	All three: the Claude Code plugin's setup.

skill and hook each mention tools only the MCP arm has. That is how the files ship; the arms measure what each part does on its own, not a redesigned part.

The models

Six models from three providers (Anthropic, Google, OpenAI), in two tiers. Every model runs in the same agent loop, with the same file tools and no shell, at the same nominal reasoning level. Prices as read on 2026-10-12.

Model	ID	Provider	Tier	Runs
Claude Haiku 5.5	claude-haiku-5-5	Anthropic	small	300
Gemini 3.5 Flash-Lite	gemini-3.5-flash-lite	Google	small	300
GPT-6 Luna	gpt-6-luna	OpenAI	small	300
Claude Sonnet 5.5	claude-sonnet-5-5	Anthropic	mid	100
Gemini 3.8 Flash	gemini-3.8-flash	Google	mid	100
GPT-6.1 Sol	gpt-6.1-sol	OpenAI	mid	100

Scoring

Three measures, two of them independent of onsystem:

tsc (independent). Each changed TypeScript file is type-checked in its project's own setup: a temporary tsconfig extends the project's, lists only the changed files (plus the project's ambient .d.ts files), and runs with the repository's own TypeScript. Only diagnostics in the changed files count. Invented components, props and variant values are type errors when the design system is typed, which all five are; hardcoded colors and spacing are not. A run passes when its file exists and has no type error. dub's @dub/ui and @dub/utils export a build output that an install without scripts lacks, so their imports are mapped to their sources for tsc.
onsystem check (the tool's own). The changed UI files, against a baseline of the untouched repository, so only findings the run added count. A run is clean when its file exists and has no error.
Blind labels (independent of onsystem). A stratified random sample of files (2 per model and arm by default), blinded: the labeller, a person or an AI reviewer, recorded in labels/LABELLER.md, sees the task, the design system and the file, not the model, the arm or onsystem's verdict. They count invented APIs, hardcoded values, native elements and unnamed controls, and say whether they would merge it. Then, separately, they judge each onsystem finding on those files as correct, wrong or debatable. LABELLING.md is the guide.

prepare checks that the scorers can be trusted in each repository before anything runs: a correct use of the design system's Button must pass both tsc and onsystem, the same component with an invented variant (variant="warning") must fail both, and two or three existing files must type-check. All five pass (tsc: 0 errors on the correct use, 1 on the invented one; onsystem: 0 and 1, no-unknown-variant).

Also recorded per run: whether it changed files other than its own, requests, tokens, cost, time, hook blocks, MCP calls and skill loads.

Statistics

Outcomes, fixed before any run:

Primary, independent: tsc pass rate per model and arm.
Primary, onsystem's own: onsystem-clean rate per model and arm.
Composite: accepted = file created, type-checks and onsystem-clean.
Blind labels: would-merge rate, files without violations, violations per file, and the share of onsystem's findings the labeller judges correct.
Secondary: onsystem errors per run, cost, requests, tokens, hook blocks, MCP calls, skill loads.

Rates get 95% Wilson score intervals. Means and every difference between arms get 95% percentile intervals from a cluster bootstrap that resamples tasks (10,000 draws, seed 1): runs of the same task are not independent, since a hard task is hard in every arm. Each arm is compared with none on the same model, task and repeat. There are no p-values. The tables pool the models of each tier, and all models together. The headline is the comparison pooled over a tier's models for each arm (4 per tier and outcome); per-model comparisons (up to 36) are reported too, and about 1 in 20 of those intervals would exclude zero by chance alone. The tables are computed by lib/stats.ts, which test/bench-v2.test.ts covers.

Cost and time

API cost: $57.08 for 1,200 runs, under a cap of $60
Wall-clock time: 120 minutes, runs in parallel
Time per run: median 33 s, from 8 s to 427 s
How runs ended: 1,122 finished, 78 stopped at a limit (max-calls); a run that stops at a limit counts, with what it wrote

Threats to validity

The labels are an AI reviewer's. The sample was labelled blind by an AI reviewer (Claude Opus 5.5), not by people. LABELLER.md records what it could see and how it decided.
The labeller's family is under test. Two of the six models (Claude Haiku 5.5 and Claude Sonnet 5.5) are from the same family as the labeller, so they may share blind spots.
onsystem is one of the scorers. The headline counts onsystem's own errors. tsc and the blind labels are the independent checks: tsc does not see hardcoded values, and the labels cover 60 files.

Not a product test. The loop is not Claude Code, Codex or Gemini CLI. Real clients add their own prompts and tools, which can change how much onsystem helps.
One reasoning level. medium means different things to different providers; other levels may change the picture.
The prompt asks for the design system. Every arm is told to use the project's components and tokens. A prompt that does not ask would likely show larger effects; this one is the conservative case, and the same as v1.
The scorers' blind spots. tsc sees invented APIs, not hardcoded values. onsystem sees both but is the tool under test; its precision on the corpus is in corpus/labels.json. Neither judges whether the UI does what was asked; only the blind labels do, and only on a sample.
Repositories and tasks. Five open-source React and Tailwind projects and twenty single-file tasks. Editing existing screens, other frameworks and multi-file work are not covered. The repositories and their tasks may be in some models' training data.
The skill and hook arms mention the MCP tools they do not include, as shipped.
Prices and models change. The cost table is for 2026-10-12. Gemini 3.8 Flash's price doubles on 2027-01-01, and Gemini 3.1 Pro is a preview.
Run-to-run variance is measured only where a matrix has repeats.

Reproduce it

Requirements: macOS or Linux, Node.js 22.18 or later (the harness's TypeScript runs through Node's type stripping; 24 is what CI uses), pnpm 10 or later and git on the path, about 8 GB of disk for the prepared repositories, and pnpm build in this checkout (the MCP server, the hook and the scoring all use dist/cli.js).

Shell

pnpm install && pnpm buildnode bench/v2/run.mjs plan                       # matrices, runs and cost; no network, no keynode bench/v2/run.mjs prepare                    # fetch, install (scripts off), baseline, scorer controlsnode bench/v2/run.mjs run --matrix mock --out /tmp/bench-mock   # offline, scripted model, no key

The work folder ($TMPDIR/onsystem-bench-v2, or --work, or BENCH_WORK_DIR) must sit outside any project, since onsystem looks upwards for node_modules and workspace roots.

Once the spend is approved, with ANTHROPIC_API_KEY, OPENAI_API_KEY and GEMINI_API_KEY exported in the shell:

Shell

node bench/v2/run.mjs run --matrix canary   --out bench/v2/results/canary   --max-cost 5node bench/v2/run.mjs plan --matrix standard --usage-from bench/v2/results/canarynode bench/v2/run.mjs run --matrix standard --out bench/v2/results/standard --max-cost 60 --concurrency 8node bench/v2/run.mjs labels --out bench/v2/results/standard   # then label, see LABELLING.mdnode bench/v2/run.mjs report --out bench/v2/results/standard   # rebuilds the tables, labels included

Keys are read from those environment variables only, by name, and never printed or written; a missing one stops the run before anything starts, naming the variable. Child processes (the MCP server, the hook, tsc, installs) never receive them. A matrix needs only its providers' keys; mock needs none.

run resumes: finished runs are skipped, and spending so far (ledger.jsonl) counts against the cap. v1's runs took 22 seconds to 6 minutes each; --concurrency sets how many run at once.

Every request's usage and every tool call are on the bench-v2-2026-10-12 release; the files each run wrote, its scores and the labels are in bench/v2/results/standard. The full method is in bench/v2/README.md.

The earlier benchmark

v1 was one repository, Claude only, 40 runs, and onsystem scored its own output. It showed a direction, not a result.

Agent
v1 (2026-10-08)
Claude Code headless, with the user's own settings loaded
v2
One neutral agent loop over each provider's API, nothing else loaded
Models
v1 (2026-10-08)
Claude Haiku 4.5, Claude Opus 5
v2
Small and mid tiers from Anthropic, OpenAI and Google; frontier tier optional
Repositories
v1 (2026-10-08)
1 (vercel/chatbot)
v2
5 from corpus/, pinned, installed with scripts off
Tasks
v1 (2026-10-08)
10
v2
20 (4 per repository), fixed prompts
Conditions
v1 (2026-10-08)
base, plugin
v2
none, skill, mcp, hook, all
Repeats
v1 (2026-10-08)
1
v2
1 to 3 per cell
Scoring
v1 (2026-10-08)
onsystem check
v2
tsc in each project's own setup, onsystem check, and blind labels on a stratified sample
Statistics
v1 (2026-10-08)
counts
v2
Wilson intervals for rates, cluster bootstrap for means and paired differences

	v1 (2026-10-08)	v2
Agent	Claude Code headless, with the user's own settings loaded	One neutral agent loop over each provider's API, nothing else loaded
Models	Claude Haiku 4.5, Claude Opus 5	Small and mid tiers from Anthropic, OpenAI and Google; frontier tier optional
Repositories	1 (vercel/chatbot)	5 from `corpus/`, pinned, installed with scripts off
Tasks	10	20 (4 per repository), fixed prompts
Conditions	base, plugin	none, skill, mcp, hook, all
Repeats	1	1 to 3 per cell
Scoring	`onsystem check`	`tsc` in each project's own setup, `onsystem check`, and blind labels on a stratified sample
Statistics	counts	Wilson intervals for rates, cluster bootstrap for means and paired differences

Claude Haiku 4.520 runs

Components with no design-system errors

Without6 / 10

With plugin10 / 10

Design-system errors, all ten tasks

Without13

With plugin0

Cost: −11%; $1.31 → $1.17
Avg. time: 46 → 42 s
Tool calls: 5.6 per task

Claude Opus 520 runs

Components with no design-system errors

Without8 / 10

With plugin10 / 10

Design-system errors, all ten tasks

Without5

With plugin0

Cost: +7%; $7.74 → $8.31
Avg. time: 108 → 141 s
Tool calls: 7.3 per task

Every task, every runShow

Design-system errors per task, for each model, without and with the plugin
Task	Haiku 4.5without	Haiku 4.5with plugin	Opus 5without	Opus 5with plugin
Usage bannera dismissible warning banner shown when the user is close to their monthly message limit ("You've used 90% of your messages this month"), with an Upgrade button and an icon-only close button.	1 error1 × icon-button-accessible-name	clean	clean	clean
Delete dialoga confirmation dialog for deleting all chats, with a Cancel action and a red, destructive confirm action.	clean	clean	clean	clean
Settings carda card with a title, a description, an API key text input, a Save button and a small green 'Connected' status label.	2 errors2 × no-hardcoded-color	clean	4 errors4 × no-hardcoded-color	clean
Empty historyan empty state for the chat history sidebar with an icon, a short message and a 'New chat' button.	clean	clean	clean	clean
Message toolbara row of icon-only buttons under an assistant message: copy, regenerate, thumbs up and thumbs down.	clean	clean	clean	clean
Upgrade carda pricing card for the Pro plan with the price, a list of four features with check icons, a 'Most popular' label and an 'Upgrade to Pro' button.	clean	clean	clean	clean
Rate limit alertan inline red error alert telling the user they were rate limited, showing when they can retry, with a 'Try again' button.	9 errors9 × no-hardcoded-color	clean	clean	clean
Share dialoga dialog showing the share link in a read-only input, with an icon-only copy button next to it and a secondary 'Make private' action.	1 error1 × prefer-design-system-component	clean	clean	clean
Profile carda small card with the user's avatar initials in a colored circle, their name and email, a 'Free plan' label and a 'Sign out' button.	clean	clean	1 error1 × no-hardcoded-color	clean
Shortcuts dialoga dialog listing keyboard shortcuts (New chat ⌘K, Toggle sidebar ⌘B, Copy last answer ⌘⇧C) in two columns, with styled key labels.	clean	clean	clean	clean

Claude Code built ten components for vercel/ai-chatbot, once as it ships and once with the plugin, and onsystem check scored every file. Without the plugin, the misses were raw colors for things the prompt described, a native <label> where the project has Label, and an icon button nobody could name with a screen reader. With it, the agent looked components and tokens up before writing (5.6 to 7.3 design-system tool calls per task).

It is one project and 40 runs, one per task, model and condition, so read it as a direction rather than a rate; Claude Haiku 4.5 cost less with the plugin than without. The method, per-run results and every generated file are in the repository.