Skip to content

<design-systems> By Ben Voran

Component Contracts and the Self-Healing Loop for Design Systems

A lot of what agents get wrong in a design system is silent. The code is valid and the page looks fine. A self-healing loop catches it before it ships.

In the last several projects here at Southleft, we’ve been exploring this idea of a self-healing loop for agents building with a design system. What makes this idea possible is having a single source of truth for components, namely contracts, which are captured in this repo and mentioned in this post.

On one of these projects I decided to test the effectiveness of contracts and, more specifically, of using those contracts to check the agentic output when asking agents to build with the design system components.

The “check” piece is where it gets interesting, because we are deterministically checking the agent’s work, which is more stable and more portable than just trying to correct an agent with documentation. This idea is already popular in AI-assisted software development, and I think this video does a great job of explaining the benefits of such an approach.

The test

I set up a test where I would run different agents using Opus 5.5, Sonnet 5.5, and GPT-6.1 Sol (Codex). Each of these agents was tasked with rebuilding the same six real pages from screenshots, using the components that were in the design system. Just one shot with no follow-ups.

Then what I did was basically have a validator CLI function that would run and check the output that the agent gave against the contracts or sources of truth that were set up for each component. These sources of truth included things like which components could be nested inside of another component, which tokens to use for styling, including override styling for a theme, and just basic best practices of how to do things like accessibility and composing with the components.

Diagram of the self-healing loop. The contract feeds the agent. The agent sends code to the validator, the validator sends the fix back to the agent, and code that passes comes out as on-system UI.

Each agent built each page twice, once with the validator and once without the validator. The docs and the contracts stayed the same in both. We just deleted the validator. That ended up being 48 builds. Now you are probably checking the math on that and wondering why I didn’t come up with 36. That’s because I had Opus run each build twice.

How the test was set up. With the validator, the agent had the docs, the contracts and the validator. Without the validator, it had the same docs and contracts, and the validator was removed.

Lastly, I used both Opus and Sol to blindly judge the validated and non-validated output on visual fidelity and code quality. And so the whole concept behind this post is exploring the efficacy of a self-healing loop (validator) with agents composing with your design system.

What the validator actually caught

So here’s my reading of the results. Basically, the validator really helps with things like CSS tokens and the styling hooks that are being used in the components. Because the components are built with TypeScript, things like props and other types that are outlined in the components themselves have a built-in natural validator. 87% of what the unvalidated builds shipped came from rules that read our hook manifest, which basically lists every styling hook for every component, variant and CSS property. And about half of everything that was caught was repetition, the same override copied into a bunch of different places. That doesn’t break the page, but it’s still a defect, because the next person to make a change has to find every copy.

And the scary part is a lot of that is silent. A token that doesn’t exist, or a hook the variant never actually reads, is still valid code. The browser console shows no red, and the page usually looks fine because the agent made up for it somewhere else.

Part of a dashboard rebuilt by GPT-6.1 Sol without the validator: four stat tiles for new leads, unread messages, tasks and bookings, above Create new, Automations and Leads cards. It looks finished, but the validator found 8 errors and 58 warnings in the code behind it.
GPT-6.1 Sol without the validator: 8 errors and 58 warnings
The same part of the dashboard rebuilt by GPT-6.1 Sol with the validator. It looks nearly identical and passed with no errors or warnings.
GPT-6.1 Sol with the validator: 0 issues

I went in thinking that visuals, accessibility and what children or even what parent a component is attached to would be where the contracts and the validator really help. That’s not really what the data shows. The validator did catch some of that, things like illegal nesting, invented components and missing accessibility props, but only 18 times and only on the agents’ first runs. And the accessibility audit found the same number of serious issues with and without the validator (probably a sign that the accessibility contracts need work).

The self-healing part

This is the part that really sold me on the self-healing loop. The first time each agent ran the validator, it found about as much as the unvalidated agents shipped, a median of 20.5 issues against 22.5. They just kept running it and fixing things until it passed. Most of that happened in the very first round of fixes, and the typical session hit its first clean run by the third time it ran the validator. In at least six sessions, new code brought in new problems after the count had already dropped, and the validator caught those before the page shipped. Every single validated build shipped clean. All 24 of the unvalidated builds shipped something a validated run would have caught.

Shipped issues. Without the validator, 24 of 24 builds shipped issues. With the validator, 0 of 24 builds shipped issues.

Not every model needs it the same

A real benefit of using this validator and the self-healing loop concept is that we can get better quality code and design system adherence no matter which model is doing our agentic composition with the design system. But it doesn’t help every model the same. And it’s not really about how smart the model is. It’s more about what kind of mistakes each model makes.

Opus, without the validator, mostly left behind repetition, the same styles getting set over and over in different places. 18 out of 24 judgments preferred the Opus code that had the validator. Sonnet and GPT-6.1 Sol left behind more of the silent and off-system stuff. All six Sonnet pages moved in favor of the validator, and Sonnet’s code improved the most. GPT-6.1 Sol came out 8 to 4 in favor of the validator, and it fixed the most issues out of all of them, about 52 per page, compared to about 29 for Sonnet and about 21 for Opus.

Bar chart of the average issues the validator caught and the agent fixed, per page: about 21 for Opus 5.5, about 29 for Sonnet 5.5 and about 52 for GPT-6.1 Sol. These are findings removed between validator runs, and a finding that came back and was fixed again counts twice.

The reviewers preferred the validated code in 38 out of 48 judgments, but the visual scores were basically the same. This isn’t to say the contracts can’t help with visual fidelity. However, since the north star was a screenshot, the agents were smart enough to get visual parity with and without the validator. For a deeper dive into agents and visual judgment, see this post.

The validator also costs very little for at least two of the models. On Opus it was about 2% more per build, which is inside the noise between two runs of the same build, and GPT-6.1 Sol didn’t take any longer with the validator. Sonnet averaged about 15% more per build, but two of its builds were significantly slower. Removing those outliers brought the other Sonnet tests to within a negligible difference.

Bar chart of which code the blind judges preferred, by model. Opus 5.5: 18 of 24 judgments preferred the build with the validator and 6 the build without. Sonnet 5.5: 12 of 12 with, 0 without. GPT-6.1 Sol: 8 of 12 with, 4 without. Two blind judges per pair made 48 judgments in all, with repeated overrides counted as a defect, and 38 of 48 preferred the build with the validator.

AI grading AI

Another thing too: while doing this experiment, we found that the agents running the experiment, Opus 5.5 and GPT-6.1, were able to give me pointers on where to go next with the validator, based on what they were finding and on independently reviewing the quality of the output from each agent’s tests. The blind judges came from two different model families, and an independent Codex pass even double-checked the numbers. So it was basically using AI to grade AI’s results against my intent in the experiment, and it provided some helpful next steps with where to take the validator.

Where it gets tricky

All in all, the validator seems like something that can really benefit large-scale systems, not only for code quality, but also for making sure the components are being used the way they’re meant to, at least as far as the contracts can describe it. And this is where it gets a little tricky. I built the validator for this purpose to be pretty simple. It’s really a thin layer over the contracts, so what it can catch, what it gets wrong, and what it misses all come from the contracts. Which means most of the ways to make the validator better are really ways to make the contracts better.

But we can continue to improve what goes into the validator based on the needs of individual design systems. And so this is where it gets really interesting because you can set up more of those golden paths where you have a specific deterministic output that you’re wanting. And so you allow an agent to fine-tune your validator to make sure that you get closer and closer to that output that you want. So if you have those golden paths, you can make the right kind of loops in a system to help an agent better steer the validator and get you to the results that you want.

Why anyone should care

Agents are going to write more and more of the UI that ships on top of our design systems, and in this test every one of the 24 unvalidated builds shipped issues. Most of them you’d never see by looking at the page. Better docs and smarter models don’t necessarily solve that. The unvalidated agents had the same docs and contracts as the validated ones, and each model still drifted in its own way. What solved it was letting the agent check its own work against the contracts. So I think the contracts are where the real investment belongs, because without something that can tell an agent it’s wrong, the agents end up deciding what your design system is without you even knowing.

<cta>

Is your design system AI-ready?

A 30-minute discovery call is the fastest way to find out where your leverage is.

<theme.console>

// every theme is solved to WCAG AA before it's allowed out. color, type, shape, texture, elevation, motion — 44 tokens, the whole site, no rebuild.