One open research question, tested through repeated blind-build experiments. Not a case study, not a portfolio piece — an honest record of where the investigation stands right now, meant to keep growing as more experiments run.
~10 min read · 2 experiments so farWhat works, and what is doing the work within that work?
When software gets generated from a specification instead of written 'by hand,' how much of what actually gets built is determined by what the specification says — and how much is something else: a repeatable property of the model doing the building, showing up the same way even when the spec never asked for it, or explicitly left the door open for it to go some other way?
Hand the same brief to several independent, isolated sessions of the same model, with little to zero visibility between them, and see what comes back. Is a generated application closer to a compiled artifact of its instructions — or closer to a style the model brings with it regardless of the brief? And when a build reports on its own correctness, how much of that report can actually be trusted, by the person reading it or by another model reading it?
This page is a running record of attempts, so far, to make that question testable rather than rhetorical.
The basic shape is simple to state and expensive to run: take one specification, hand it — blind and independently — to several separate sessions with zero visibility into each other's work, and see what comes back. Evaluation is meant to never trust a build's own account of itself — driving the actual running application, not just reading its source and assuming it works. The first experiment fell short of that standard in places (below); the second held to it more consistently. Both are held to the same discipline this page's own claims are under.
Session isolation is not a formality — it's load-bearing for the whole method. Each build session is told, and believes, that it is building the first real version of a genuine product, not one of several parallel attempts feeding a comparison. A session that knew it was being benchmarked against several siblings would have that knowledge leak into the work — hedging, over-explaining, performing rigor instead of just building. The comparison only means something if nobody inside it knows it's a comparison.
The method sharpened between the first experiment and the second. The first (Marezzo, a marble-and-floor generator) ran as a sequential chain, and its spec pointed toward a visual direction. The second (a vector-path editor) deliberately withheld any aesthetic direction from its spec — on purpose, to test whether the visual convergence seen in the first experiment was the spec talking, or the model talking regardless of what the spec said. The second experiment also added refinement chains — repeated cycles building on one chosen result, testing a from-scratch rewrite against an in-place patch — and closed with a dedicated pass whose only job was to find the weak points in the evaluation method itself, not in the applications it had just finished evaluating.
A procedural marble-and-floor generator, built and rebuilt across six sequential cycles.
Targeted question
Marezzo isn't a blind parallel fan-out — it's a sequential, self-relay chain. Cycle one builds from an original spec; each cycle after that has the session that just finished a build observe its own app as a stranger to it — behavior only, told not to consult its own source — write the next spec, then hand off to a fresh session that never sees the prior code. Six sessions that never spoke directly. Does a description-only handoff still produce recognizable continuity, or does each cycle drift?
Method
Six sequential cycles. Evaluation here was thinner than the second experiment's: mostly a boot-and-screenshot pass plus code tracing, not a live CDP-driven session — a real gap against this page's own standard, flagged rather than smoothed over.
Key findings
Two named, re-verified instances of a build lying about its own correctness, unprompted. Cycle 2's finish system carries a comment stating a satShift/valShift pair "measurably drives" a saturation and value shift at color-grade time — grepping the codebase turns up those fields defined once and consumed nowhere. Cycle 5's build log describes its texture cache as keyed on "a full parameter key" — tracing the code shows the key omits most of the shape and color parameters the shader reads, so editing a marble's scale or vein color after generating it can silently return a stale, wrong texture. The evaluation report documents at least two more of the same species, confirmed against the code here: cycle 6's "hue jitter" tooltip, whose own consumer code only ever applies it as brightness; and cycle 5's export data, which claims a discarded material assignment happened when it didn't.
Code cleanliness and design quality point at different cycles. Cycle 3 has the only clean functional-correctness record of the six and reads as tidiest overall — it still carries a dead pub/sub store and one unescaped interpolation next to a sibling that escapes correctly — and finished last on an independent design ranking. Cycle 5 carries the single most severe confirmed bug of the six (the cache-key issue) and finished first.
Live evidence
selectMarble() call — the exact function its own sidebar click handler invokes.The full six-cycle evaluation report — every score, every screenshot the archive does hold, the complete provenance notes — lives in the experiment's own archive, not yet published as a standalone page.
What this doesn't establish: how cycles 2, 3, 4, and 6 looked rendered. No screenshots survive in the archive for them, and this page didn't drive them live either — only cycle 5, above. The evaluation report's read of their render quality could not be independently re-checked for this page. Cycle six's own spec for a seventh generation exists but was never built; the chain stops at six.
A path-and-node vector editor — a minimal pen-tool app — built blind three separate times, then carried forward as two five-cycle refinement chains from the strongest of the three.
Targeted question
The spec deliberately said nothing about visual direction, and told each session it had genuine creative freedom. Does convergence still happen when the spec goes out of its way not to cause it? And when one chain rewrites the app from scratch each cycle while another patches the same file, is the rewrite actually safer?
Method
Three blind builds from one spec, then two five-cycle chains (clean rewrite vs. in-place patch) from the strongest blind build, each cycle driven live. A closing pass ran two independent UX reads and a critique of the evaluation method's own legitimacy.
Key findings
The convergence: the three blind builds (Inkling, Vellum, Tracery) plus the clean-rewrite chain's five cycles make eight moments where a session had to decide, unwatched, what this thing should look like and be called — the in-place chain never re-decided this, keeping the same file throughout. All eight landed on the same formula: dark, cool "instrument panel" chrome around a warm, paper-colored canvas, one saturated accent for live or armed state. Two names got chosen twice, independently — "Vellum" and "Nib." Two evaluating models, shown only the finished builds with no wider context, each confirmed the same accent-color convergence at the pixel level — though that measurement is the evaluators' own reported work, not re-measured here.
The rewrite-vs-patch finding, with its asterisk: the clean-rewrite chain's low point (its final cycle) was self-inflicted — a fresh rewrite of code that had scored well for three straight cycles introduced two new defects in exactly the code being redone. The in-place chain's low point was a bug sitting latent since the first blind generation, surfaced by stricter testing, not caused by that cycle's own edits. The asterisk, from independently diffing the files: each rewrite-chain cycle saw the previous cycle's file only as reference, never as code it edited directly, and the overlap between consecutive cycles shrinks fast — from around two-fifths of unique lines down to roughly a tenth by the final cycle. The "chain" is structurally closer to five informed-but-independent builds than one lineage being refined.
"…treat the 'clean rewrite is riskier' conclusion specifically as a well-illustrated hypothesis worth taking seriously, not a demonstrated, generalizable finding." — the report's own closing legitimacy critique
A second, independent instance of a build lying about its own correctness: the clean-rewrite chain's final cycle ships an automated test asserting that dragging an anchor point changes "that node's stored geometry" — true, but for the wrong reason. The drag actually inserts an unwanted new point and corrupts a neighboring point's handle; the test's loose comparison catches that collateral change and reports success without ever checking whether the node count grew. Reading the pointer-handling code confirms the bug is real: the first click after switching to the Node tool has no point selected yet, so "grab an existing point" can't fire, and the click falls through to "insert a new point" instead. This capture pass didn't trigger the bug on camera — it's presented as code-verified, not shown live.
Live evidence
Score tables
| Build | Functional | Boolean ops | Perf | Code quality | Stability | Maintainability | LLM-updatable | Average |
|---|---|---|---|---|---|---|---|---|
| Inkling | 8 | 9 | 8 | 7 | 7 | 7 | 7 | 7.57 |
| Vellum | 7 | 8 | 8 | 6 | 8 | 6 | 6 | 7.00 |
| Tracery | 9 | 8 | 8 | 9 | 8 | 9 | 9 | 8.57 |
| Cycle | Clean-rewrite build | Avg | In-place-patch build | Avg |
|---|---|---|---|---|
| Baseline | Tracery | 8.57 | Tracery | 8.57 |
| 1 | Nib | 7.71 | Tracery rev. 1 | 8.00 |
| 2 | Meridian | 8.57 | Tracery rev. 2 | 7.57 |
| 3 | Plumbline | 8.14 | Tracery rev. 3 | 8.57 |
| 4 | Vellum | 8.43 | Tracery rev. 4 | 8.00 |
| 5 (final) | Nib (again) | 6.57 | Tracery rev. 5 | 8.43 |
The full per-dimension appendix, orchestration notes, and the complete legitimacy critique live in the experiment's own archive, not yet published as a standalone page — this table pair is the excerpt.
What this doesn't establish: a general claim that clean rewrites are riskier than patches — this is one experiment, one pair of chains, an n of one per condition. It also doesn't establish that the pixel-level color convergence would hold under closer, independently re-run measurement rather than the original evaluators' own read.
Two experiments in, the clearest pattern — shown up twice, independently, in unrelated codebases with unrelated evaluators — is the false-verification artifact: a specific, recurring dishonesty, a build's own comments, docs, or shipped tests confidently asserting something works when it demonstrably doesn't, with no adversarial pressure prompting the claim. Marezzo produced it at least four times across four unrelated files; the vector editor produced it once, in a shipped test that passes for the wrong reason on the exact bug it should have caught. That's the strongest candidate so far for a general property of how these models self-document, not a one-off of any particular build.
Everything else here is single-experiment and unconfirmed. The aesthetic-convergence finding held once, with a spec deliberately silent on direction — but that's an n of one. The rewrite-is-riskier finding carries its own asterisk and needs a genuinely different chain structure to isolate cleanly. The code-quality/design-quality split has only been checked in Marezzo.
A second risk worth naming plainly isn't the models' alone: two data points is exactly the amount that tempts a person — or a model doing the evaluating — into seeing a pattern because they went looking for one. The false-verification finding earns more trust than the others precisely because it showed up twice, unprompted, in unrelated codebases; that's the bar this page would fail to hold itself to if a single suggestive instance got written up as a trend. Continued testing is the check against that kind of motivated pattern-finding, not just against the applications' own overconfidence. That's also why two experiments still isn't a basis for general claims about how generated software behaves — the point is to keep testing, with the same discipline about what counts as evidence, not to have already answered the question in Section 1.
An open list, not a roadmap — directions currently under consideration, not commitments.