Specification-Driven Development
I've been maintaining a personal codebase (workspaces, boilerplates, packages, scripts) and developing personal projects in parallel — I believe having your own arsenal of everyday tools saves a lot of work. In this post, I decided to check on the state of agent autonomy — and, as we saw in the first post, not every agent was designed with autonomy native to its philosophy. Below is a survey of the techniques and frameworks used for this purpose.
The test runs on a fixture: a minimal blog in pure Node, with posts served by REST routes over the native http module and an already established response and error convention. The task given to each framework was to add a comment system to this blog — not a greenfield scenario, but a feature built on top of conventions that already exist. That's what the test measures: does each framework respect the existing pattern, or does it ignore it and reinvent everything from scratch?
(Repository with the seven worktrees: github.com/vitorinoguilherme/sdd-frameworks-compared.)
Fundamental Concepts
Before comparing the frameworks, it's worth aligning on some concepts that appear throughout the post:
Prompt Engineering: the technique of structuring the instruction sent to the model to get better responses. On its own, it doesn't solve problems of memory, continuity between sessions, long-running execution, or state maintenance.
Context Engineering: organizing documentation, architecture, rules, specifications, and conventions so the agent works with relevant, well-structured information. Post 1 shows an example of how I organize all of this in an
.ai/folder.
Spec-Driven Development: development guided by specifications — you define pages, states, entities, flows, rules, contracts, and acceptance criteria before execution. Goal: reduce ambiguity, drift, inconsistency, and wrong decisions during implementation.
Workflow-Driven Development: separates execution into distinct phases. Goal: avoid context overload, loss of focus, and chaotic execution.
Runtime State: operational memory persisted between executions. Important because the LLM forgets, summarizes incorrectly, and loses the original objective over long sessions.
Checkpoints: formal stops between phases — for example, validating, summarizing, persisting state, and only then moving on to the next phase. Goal: avoid cognitive degradation of the model over the course of execution.
Orchestration: overall coordination of execution. Can include planning, retries, validation, task splitting, and checkpoints.
| Concept | Focus | Mitigates |
|---|---|---|
| Prompt engineering | prompt | instruction |
| Context engineering | context | consistency |
| Spec-driven | specification | ambiguity |
| Workflow-driven | execution | loss of focus / chaotic execution |
| Runtime state | memory | continuity |
| Checkpoints | transition between phases | context degradation |
| Orchestration | coordination | autonomy |
Where the frameworks come in
The concepts above describe problems; the frameworks below are attempts to solve them systematically, each with a different philosophy. The problems that motivate them keep recurring: context drift, loss of objective, bad summaries, recency bias, diluted attention — and the hypothesis this post puts to the test is that they all share a common denominator: the lack of persistent state between executions. There's also the simplest option of all: use the LLM directly, without any framework — that's what the Baseline represents here, the control condition. And there's also a parallel category emerging, focused specifically on the memory problem — tools like agent-memory — that don't try to orchestrate the entire flow like the frameworks below, only solve the persistent-state part. That's outside the scope of this test, but it's another angle attacking the same root cause.
| Framework | Real focus |
|---|---|
| Baseline | control without a framework |
| Ralph Loop | execute → verify (stateless loop) |
| OpenSpec | specs + contracts |
| Superpowers | methodology enforcement |
| BMAD-METHOD | multi-agent workflow |
| Open GSD | orchestration + decomposition |
| GitHub Spec Kit | TDD + spec contracts |
Running the experiment
The idea here isn't to compare the frameworks on paper, but to run each one end-to-end on the same real task — the full comment system from TASK.md, not a minimal example. Each framework received the entire task, in its own isolated worktree, and ran until it finished or got stuck. Even an incomplete result — "stopped at X" — already reveals something about how the framework drives execution.
Underneath the whole comparison lies a tradeoff: giving the agent more autonomy gains execution speed, but costs context control at every step.
Before running any framework, I set aside a few design decisions — because without that, the comparison becomes just "which tool seemed cooler," not data.
The first was to isolate everything in a disposable fixture, unrelated to any
real project. A fixture is a minimal blog application created just for this
test — pure Node, no external dependencies, post routes using only native
http. Anything beyond that (Express, an ORM, a test framework) would
make the agent spend part of the decision on "which lib to use," which isn't
what I want to measure.
The task is a comment system — but not greenfield in disguise. The fixture already comes with an established convention (a simple REST route, a 404 error pattern, response format), and the question the test asks is whether each framework respects that or ignores it and reinvents it. I deliberately placed two traps: the comment body must not allow arbitrary HTML/script (this tests whether the framework thinks about security without me explicitly asking), and deleting a comment must preserve the reply thread (this tests whether it thinks about cascading effects, not just surface-level CRUD). I also defined an explicit "out of scope" — without that, the more "completist" frameworks tend to expand the task on their own (notifications, moderation, likes), and then the comparison between them stops being fair.
Each framework runs on a separate branch, all born from the same commit via
git worktree — one repo, not seven loose folders. This guarantees an
identical starting point across the tests: any difference in outcome comes
from the framework, not from which context each one had available.
The evaluation rubric has two dimensions. The first is how each framework handled the decisions the task deliberately left open: decided silently and baked the value into the code, decided and documented the rationale, proposed and waited for approval, or asked one by one. The second is the quality of the result beyond "it finished": whether the generated tests actually run and cover the cases the TASK.md asked for — rate limiting, thread preservation on soft-delete, character limit — or whether the suite passes green while leaving exactly those cases out.
Baseline
The baseline is the control: the same agent, the same model, with no framework at all — just the TASK.md pasted in as input. Zero install, zero configuration. The only question it answered was the task itself.
The agent explored the codebase before coding (six reads: server.js, posts.js,
package.json, the protocol files). It then declared all four ambiguous decisions
as a block of text at the start of its response — it didn't ask anything. Character
limit: 2000. Edit window: 15 minutes. Response when the parent is removed:
the entry remains, text and name become null, deleted: true flag. Rate limit: 5
comments per IP every 10 minutes. Reasonable values, in line with what you'd expect from
a quick human choice — but the user wasn't consulted on any of them.
One decision was entirely off the radar: persistence. The agent didn't ask, didn't
declare, didn't raise it — it simply used an in-memory array (const comments = []) and
a Map for rate limiting. package.json ended up with no dependencies at all. This kept
the fixture's zero-deps convention, but it means the data disappears on every restart. GSD,
by contrast, raised persistence as a decision to be made before starting.
TDD: no — the transcript confirms the order: comments.js first, server.js next, tests last. The 10 new tests covered the 5 mandatory cases from TASK.md plus four extras (parentId, nonexistent post 404, listing GET, rejection of text above the limit); 12/12 pass. Zero human intervention, zero planning artifacts, zero external dependencies — all in a single turn.
The baseline reveals the floor: given enough context and permission to decide, the agent completes the feature — but the trail of decisions stays invisible in the code, not in a document.
Full implementation: worktrees/00-baseline.
Ralph Loop
Ralph isn't a package — it's a technique. The premise: a ~100-line bash loop calls
claude -p without --continue. The model's context resets on every iteration; all state
persists on disk — src/, test/, a LEARNINGS.md that the agent maintains as
inter-iteration memory, and a .ralph-done sentinel. The loop stops when the sentinel exists and
npm test returns zero. The idea, as described by creator Geoffrey Huntley, is that
each iteration is an agent with completely zeroed context — the only continuity is
what's in the files.
In this test, the loop converged in a single iteration, in about 2 minutes. The only
human intervention was bash launch.sh. The Docker container came up with only the worktree
mounted, a non-root user, and an isolated HOME — no access to the host's environment variables or
to the real ~/.claude.
What happened next was completely autonomous: the agent read TASK.md, created
DECISIONS.md with 6 documented decisions (the 4 ambiguities from the spec plus 2 extras about
escaping strategy and how to represent the soft-delete), implemented src/comments.js with
an in-memory store, sliding-window rate limiting, and an injectable _now = Date.now()
parameter to test time windows without sleep, and wrote 19 tests. During this
pass, it detected that the edit-window check used > where it should have been >= — the
boundary test failed, the agent fixed it in a single Edit and ran again. The same bug
exists in GSD, where it was never tested.
TDD: not in the strict sense — the code came before the tests within the same iteration,
confirmed by the transcript. What happened was detection and fixing within a single
~2-minute pass, not a planned RED→GREEN cycle. One real gap remains: none of the
21 tests verify the 429 rejection behavior, despite rate limiting being
implemented and documented in DECISIONS.md.
Ralph's tradeoff is that --dangerously-skip-permissions doesn't come for free. The native
bubblewrap sandbox failed inside Docker — the agent itself experienced this: the
first tool call returned exit 1 with bwrap: No permissions to create new namespace.
It added dangerouslyDisableSandbox: true to all 6 subsequent Bash calls.
The real containment was Docker + HOME override. The LEARNINGS.md written at the end with 4
notes for future iterations was never re-read — the loop didn't need a second pass.
On tasks where iteration 1 fails, it's this file that keeps iteration 2 from repeating the
same mistakes.
The full transcript has 57 JSONL messages and 16 tool calls, with zero thinking blocks. BMAD produced 976 messages for the same task. Ralph used about 6% of the message volume of the heaviest framework — with zero human intervention after the trigger.
Full implementation: worktrees/06-ralph-loop.
OpenSpec
OpenSpec decides on its own — but leaves a trail: every ambiguous choice goes into a
design.md with a rationale, instead of staying invisible in the code. It's the first
framework in the batch with a real installation: npm install -g @fission-ai/openspec
followed by openspec init. When
selecting "Claude Code" during init, five skills and five commands are created in the worktree's
.claude/ — no touching the global ~/.claude.
The first difference from the baseline shows up right at the entry point: pasting the
TASK.md and waiting wasn't enough. The framework requires explicit invocation — /opsx:propose @TASK.md —
and imposes a pipeline of artifacts with gates between stages. The proposal must exist
before design and specs can be written; tasks is only possible afterward; and implementation
(/opsx:apply) is only unlocked once everything is complete.
The agent explored the codebase before producing any artifact — the same reads of
server.js, posts.js, and package.json as the baseline, but here as part of the framework's protocol. The
proposal.md stated right in the Impact field that the system would be in-memory and
have no external dependencies — without asking. Next, design.md and spec.md were
generated in parallel. It was in design.md that the 4 ambiguities appeared: in an
"Unspecified decisions resolved" section, with a table listing the value and rationale for each
one — 2000 chars, 15-minute edit window, replies preserved with the soft-deleted parent,
and 10 comments per IP per 60 minutes for rate limiting. Everything decided
unilaterally. The framework has an explicit internal instruction: "prefer making
reasonable decisions to keep momentum."
TDD: no — the tests were written after comments.js and server.js. 18/18 pass,
zero-deps maintained, zero human intervention besides confirming /opsx:apply.
The contrast with the other frameworks is telling. GSD asked about the four ambiguities,
one by one, with recommended defaults. The baseline decided them implicitly in the code —
invisible without reading the implementation. OpenSpec landed in the middle: it decided on its own, like the
baseline, but documented each choice with justification in design.md. The user wasn't
consulted, but can read the reasons afterward. With 2 commands and 237 lines of artifacts, it's the
lightest framework with any structure at all — and one of the best at separating decision from
implementation without asking for approval.
Full implementation: worktrees/01-openspec.
Superpowers
Superpowers is the only one of the seven that runs an autonomous self-review before
moving on to the plan — the agent itself re-reads and fixes what it just wrote, with no
human input. It isn't a CLI — it's a plugin installed from within the session, via
/plugin marketplace add obra/superpowers-marketplace followed by
/plugin install superpowers@superpowers-marketplace. With HOME override, it ended up in
the worktree's .claude/plugins/ without touching the real ~/.claude. The operating model is an
explicit sequence of skills with delimited scopes: brainstorming → design →
self-review → plan → execute → finish. Each transition is deliberate; each skill knows
what it can and can't do in that phase.
The brainstorming skill explored the codebase before asking anything (5 files
read, git log) and then asked 2 questions — but neither was about the 4
ambiguities in TASK.md. The questions were about technical design: how to sanitize (strip all
tags or use an external lib?) and which route convention (nested REST or flat?). The 4 ambiguities
came later, compiled into an approval table with proposed values and rationales, and
a single checkpoint was requested before the 132-line design doc. It's the opposite of GSD,
which interrogated each ambiguity individually; Superpowers consolidated everything into one block
and asked for a "yes."
Before generating the plan, the framework ran an autonomous self-review of the spec: the same
agent that wrote the design doc read it and fixed 3 problems — the route count was
wrong (3→4), the behavior when parentId points to a soft-deleted comment wasn't
specified, and DELETE needed to be declared idempotent. It fixed everything in a separate
commit, with no human input.
TDD: yes — confirmed by the transcript. The tests were written before src/comments.js,
and the RED→GREEN cycle of the initial unit test was executed: the agent confirmed
Cannot find module before creating the implementation file. The 566-line plan
embedded TDD steps with the expected output of each node --test, and the execution followed that
script. 18/18 pass, zero external dependencies, about 3 human interventions throughout
the whole session.
Superpowers' distinguishing feature is its constitutive self-review — the agent edits its own artifact before moving forward. That it identified 3 problems in the spec it had just written, and fixed them autonomously, says something about how much the phase structure forced a critical re-read that linear execution wouldn't produce.
Full implementation: worktrees/03-superpowers.
BMAD-METHOD (BMAD)
BMAD is the most ceremonious of the seven — and the only one that exhausted the 5h usage window
of Claude Code during the run. The install (npx bmad-method install) creates 46 skills
locally in the project's .claude/skills/, without touching the global ~/.claude. The structure
is of named role-agents: John (PM), Winston (Architect), and Amelia (Dev) are invoked
in sequence, and the handoff between them happens exclusively via files on disk in
_bmad-output/. There's no shared session; the next agent starts by reading what the
previous one wrote.
John handled the 4 ambiguities with a specific approach: he proposed each value with a rationale in a formatted table and waited for a single approval checkpoint before writing it into the PRD. One "ok" was enough to move forward. The final PRD had 166 lines and 17 scenarios with stable IDs (T-1..T-17).
Winston scanned the existing codebase before any architecture draft and asked
only 2 questions — neither reopened the PRD's ambiguities. A detail from AD-8 revealed
the value of the pipeline: the initial draft assumed a direct reset on the Map; the user requested
explicit exports (resetComments(), resetRateLimit()). Winston updated, sealed it —
and that architectural decision enabled the backdating of timestamps in tests T-8 and T-14.
create-story worked as an adversarial gate: before any code, it caught 6
problems in the specification — ?? {} in the rate limiter would lose state on the first request,
rateLimitStore missing from exports would block T-14, a collision of parsed.body with the
HTTP variable, match→postMatch shadowing, missing finally in the teardown, hardcoded IP.
All fixed before Amelia started.
TDD: no — Epic 1 was code (src/comments.js, src/server.js), Epic 2 was tests.
The framework's first real npm test happened at
Story 2.1, after all the logic was implemented. 19/19 pass; each test() names the
PRD scenario it verifies — T-1 through T-17, 1:1 traceability.
BMAD required about 25–30 human interactions — 10 skill invocations in the dev cycle (5× create-story + 5× dev-story, no batching) plus the phase checkpoints. That the 5h window ran out before the end isn't a bug: it's the most honest metric of the framework's cost to drive.
Full implementation: worktrees/04-bmad.
Open GSD (GSD)
GSD closed with the thinnest test suite of the seven — 7/7, missing exactly the most
important case from TASK.md — after being the framework that asked the most and
documented the most before coding. The installation already sets the measure of that
excess: GSD (@opengsd/gsd-core) is the only one of the seven that doesn't install in the project — it installs
globally in $HOME/.claude with hardcoded paths. Without a disposable HOME, the
installation contaminates the real global config. HOME isolation was used across all seven
worktrees as generic precaution; what's unique to GSD is that it goes from precaution
to a technical requirement of the installer itself. With 69 skills and hooks all hardcoded in
$HOME/.claude/, there's no way to isolate it per project.
Before any spec, /gsd-explore generated 7 documentation files in
.planning/codebase/ — stack, integrations, architecture, structure, conventions, tests, and
a CONCERNS.md that already flagged XSS risk before any code was written. Next,
/gsd-new-project configured workflow preferences over 6 rounds of questions
(execution mode, phase granularity, parallelism, git tracking of artifacts,
per-phase research, plan verification), generating PROJECT.md, REQUIREMENTS.md, ROADMAP.md,
and CLAUDE.md — 12 artifacts, 1148 lines before the first line of implementation.
The 4 ambiguities were asked one by one, with a recommended default for each.
On top of that, GSD raised a fifth question absent from TASK.md: persistence. This
led to adopting better-sqlite3 via npm — the only framework among the seven to break the
fixture's zero-deps convention, instead of using Node 24's native node:sqlite. The codebase
map that GSD itself had generated recorded zero-deps as a project characteristic,
but it didn't factor into the implementation decision.
Each phase had a plan-checker subagent that validated the plan before an executor
ran. TDD: no — code throughout phases 1 and 2, tests written in phase 3, about
an hour after the last implementation commit. The mechanism that explains the main
coverage gap: phase 3's plan-checker instructed exporting resetRateLimit() so that
the beforeEach would never hit the rate limit. As a result, RATE-01 — documented in
REQUIREMENTS.md, implemented in server.js — never appears in any test. The suite
ended with 7 cases and 7/7 pass.
GSD generated the most complete specification of the seven, asked more than any other, and produced the smallest test suite — because the isolation mechanism that the planner itself created for the tests also eliminated the most important case to verify.
Full implementation: worktrees/05-open-gsd.
GitHub Spec Kit (Spec Kit)
Spec Kit is the only one of the seven where TDD isn't a planning intent — it's a
"NON-NEGOTIABLE" clause in the project constitution. tasks.md instructs confirming RED before
implementing each user story. Before any code: 9 artifacts, 1254 lines — business
spec, requirements checklist, technical plan, data model, complete HTTP contract, and
tasks per user story.
The 4 ambiguities were resolved silently by /speckit-specify, in an "Assumptions"
section with product justification for each one. One detail: the rate limit was
set to 5 per IP per 60 seconds — the only framework that used seconds instead of
minutes, which is more restrictive. The user wasn't consulted on any of the four.
/speckit-clarify did a coverage scan and found 2 points of ambiguity — but
neither was among the 4 from TASK.md. It asked about the listing endpoint, authorName
in the tombstone, and ID format. Clarify finds ambiguities that the spec process created,
not the ones the task intentionally left out.
TDD: yes — mandated by the constitution. The implementation confirmed 15/17 tests failing before
any code. A bug surfaced during this process: stripHtml with only /<[^>]*>/g left
the contents of <script> intact — <script>alert(1)</script>Hi became alert(1)Hi.
The fix added two steps before the regex: removing complete <script> blocks,
then <style>. None of the other six frameworks caught this behavior — the contract-based TDD
forced the check before implementation. There was context compaction during
the session; the agent resumed from the artifacts on disk without human intervention.
19/19 pass, zero external dependencies, ~2 human interventions. With 1254 lines of
pre-code artifacts, Spec Kit did more verification than GSD, with less paperwork than
either of the two. The difference wasn't the amount of documentation — it was what tasks.md
mandated doing with it.
Full implementation: worktrees/02-speckit.
Results
Having run all seven tests, the numbers side by side make the contrast clear.
Total counts include artifacts generated across phases (GSD: 17 files, 2483 lines; BMAD: 11 files, 2372 lines). The central irony of the experiment: GSD generated the largest volume of planning and the thinnest suite — 7/7 tests, none verifying the RATE-01 documented across three artifacts.
| Framework | Artifacts | Artifact lines | Manual turns | Zero-deps | Tests | TDD |
|---|---|---|---|---|---|---|
| Baseline | 0 | 0 | ~1 | ✅ | 12/12 | No |
| Ralph Loop | 0 | 0 | 1 | ✅ | 21/21 | No |
| OpenSpec | 4 | 237 | ~1 | ✅ | 18/18 | No |
| Superpowers | 2 | 698 | ~2 | ✅ | 18/18 | Yes |
| BMAD-METHOD | 11 | 2,372 | ~25–30 | ✅ | 19/19 | No |
| Open GSD | 17 | 2,483 | ~6 | ❌ | 7/7 | No |
| Spec Kit | 9 | 1,254 | ~2 | ✅ | 19/19 | Yes |
No framework fell into the bottom-right quadrant (asked, but left no trail) — whoever asked also documented. The only entry in the bottom-left is the Baseline: it decided everything and left no artifact besides the code.
More paperwork didn't buy more verification. BMAD (2,372 lines, 27 turns) and GSD (2,483 lines, 6 turns) produced the largest volumes of planning; OpenSpec (237 lines, 1 turn) ended up with the broadest suite in relative coverage, and Ralph (0 lines, 1 turn) was the only one to test the exact boundary of the edit window.
What the research revealed in my own practice
In my most recent projects, I've been using my own approach inside the
.ai/ folder, instead of any of the seven frameworks above. This convention predates
this research; nothing here was "applied as a result of the study." It was the opposite: I looked at what I'd
already been doing by hand and later recognized the same mechanisms that the frameworks formalize.
And the overlap is real, even if unintentional. My runtime/current-state.md serves the
same function as BMAD's status tracking and Ralph's on-disk memory — the agent itself
updates it after each phase, following an instruction in CLAUDE.md, not a built-in tool
mechanism. My .ai/decisions/*.md are informal ADRs: the same idea as the assumptions that
Winston seals in BMAD's AD-8, without the formal template. And my tasks/phase-*.md, with
checkbox gating, are the same phase-gating as GSD's roadmap phases or BMAD's sprint stories —
except enforced by convention and prose, not by a plan-checker subagent. I arrived at
similar forms on my own because the underlying problem is the same: context drift and phase memory.
Convergence, not copying.
What genuinely interested me was applying the same research yardstick to my own projects — and concrete gaps showed up there, not theoretical ones:
- Doc drift. Outdated docs, describing an old phase of the project, still get auto-loaded as context before the correct runtime doc. None of the seven has exactly this flaw: they either regenerate docs per phase (GSD) or gate on a fresh artifact (OpenSpec, Spec Kit).
- Redundancy. The same execution rules appear copied across some files — a pattern designed for unattended loops, where each file needs to
carry enough context on its own. But it becomes pure overhead in incremental
development, where
CLAUDE.mdis always already loaded. - Gate only in prose. My phase-gating exists as a written instruction ("check the
current-state.mdbefore opening a task file"), not as an enforced check — unlike GSD's plan-checker or BMAD's IR gate, which verified 24/24 requirements before releasing implementation.
None of these gaps is a reason to adopt an entire framework — most are cheap fixes for a
solo maintainer. But the pattern is clear: I had given agents autonomy without
building the brakes the frameworks embed. The .ai/ folder gave me the autonomy; what
the research exposed was where the control was missing.
Of those controls, the only one with an obvious place to fit into what I already have is an enforced gate between planning and coding: the flow I already use blocks every commit with lint, typecheck, and tests; what's missing is the equivalent over the plan and spec, not just over the finished code. The rest is ceremony billed upfront: BMAD's named role-agents, GSD's global install, the 1148 lines of artifacts before the first line of implementation — and whether that cost pays off depends on the project and who's on it.
On a project with a single maintainer, that volume tends toward waste; on a team, it's a different
story. A state file like current-state.md is already, in practice, a handoff:
it hands back context when the project is reopened after days, and it would serve the same role
passing the baton to someone else. That's where BMAD's and GSD's heavy trail pays off —
continuity across executions when there's a single dev, onboarding and shared history
when there are more people. It's worth remembering where this reading comes from: the research ran on a
disposable fixture, a single task, one turn. The failure mode that the enforced gate targets —
doc drift, phase memory lost over weeks and many sessions — a fixture like this doesn't
reproduce. Whether closing that gap is worth the friction of one more moving part, a
one-turn test doesn't answer; a real, multi-phase project would.
One lesson runs through all seven: amount of spec isn't synonymous with rigor. The two largest planning volumes produced the thinnest test suite and the most expensive one to run. What predicted the quality of verification was one single thing — the process requiring a check between plan and code; documenting more, earlier, didn't move that needle. Choosing a framework means deciding how much control you want back and how much friction you accept paying for it — a calculation that changes with the task, with no single answer.