Methodology — Felan Extension Benchmarks

1. Study design

Each benchmark compares a feature-off baseline with a feature-on candidate on the same tasks. The study estimates extension-specific effects; it is not an all-extensions-on evaluation.

Controlled within each benchmark Prompt, workspace source, setup, provider, model, thinking level, timeout, and grading stay fixed between the two arms.

One treatment at a time The candidate enables the named extension and its required feature configuration; the matching baseline leaves that extension off.

Repeated, then quality-gated Every arm runs three times per case. Resource outcomes are interpreted only after the benchmark's candidate pass-rate gate is evaluated.

Most pairs keep Felan's Codex, Context, and Tasks extensions enabled in both arms while disabling unrelated built-ins. MarkItDown is the intentional exception: both arms use the specialized ordinary-read profile with Codex, Context, and Tasks disabled, and differ only in automatic MarkItDown conversion. Profiles are explicit rather than inherited.

Prompts request user-facing outcomes rather than instructing the agent to use the feature under test. Feature-use assertions then confirm organic activation where it is part of the hypothesis: successful Prewalk entry, explorer-subagent delegation, or converted-document reads. The corresponding control arm must not make the treatment tool call.

2. Benchmark matrix

All root sessions used the openai-codex provider. Models, thinking levels, and Felan releases are held constant within a row, but not across rows.

Published benchmark arms, cases, attempts, execution identity, and objectives
Comparison and treatment	Published cases	Attempts	Root execution	Objectives
Subagents `felan-no-subagents` → `felan-delegated-exploration` `subagents: false` → `true`	`subagents-codebase-exploration`	61 case × 2 arms × 3 trials	Felan 0.21.1gpt-5.6-sol · max children: gpt-5.6-luna	CostPrompt-token result omitted after descendant-cost correction
MarkItDown `felan-no-markitdown` → `felan-markitdown` `markitdown: false` → `true`	`markitdown-document-configuration` `markitdown-docx-extraction` `markitdown-pptx-extraction` `markitdown-xlsx-extraction`	244 cases × 2 arms × 3 trials	Felan 0.19.0gpt-5.6-terra · low	CostPrompt tokens
Concise output `felan-no-output-style` → `felan-concise` `outputStyle: false` → `concise`	`output-style-planning` `output-style-review` `output-style-support`	183 cases × 2 arms × 3 trials	Felan 0.20.2gpt-5.6-sol · max	CostOutput tokens
Prewalk `felan-no-prewalk` → `felan-prewalk` `prewalk: false` → `true`	`prewalk-checkout` `memory-summary-links`	122 cases × 2 arms × 3 trials	Felan 0.20.2gpt-5.6-sol · max planner: gpt-5.6-luna · medium	Cost
RTK `felan-no-rtk-optimizer` → `felan-rtk-optimizer` `rtkOptimizer: false` → `true`	`prewalk-checkout` `rtk-felan-project-instructions`	122 cases × 2 arms × 3 trials	Felan 0.20.0gpt-5.6-sol · max	CostPrompt tokens
Codebase Memory `felan-cbm-off` → `felan-cbm-on` `codebaseMemory: false` → `true` · mode `curated`	`cbm-extension-architecture` `cbm-extension-config-scope` `cbm-project-instructions`	183 cases × 2 arms × 3 trials	Felan 0.23.2*gpt-5.6-sol · medium	CostAgent-step duration
Session Compaction `felan-no-session-compaction` → `felan-inherit-compaction` native Pi → verified extension; `model: inherit`	`session-compaction-continuation`	61 case × 2 arms × 3 trials	Felan 0.24.7gpt-5.6-sol · low local pre-release extension artifact	CostPrompt tokens

3. Workspaces and checks

Every attempt starts from a copied, isolated workspace. Public repositories are pinned to full commits; authored document and response tasks use versioned local fixtures.

Workspace source and verification approach for each published case group
Case or group	Pinned source	Principal checks
`subagents-codebase-exploration`	Felan `5e67c921794c00ad23e0a223299a2a1fc8a0f3fd`	Organic explorer delegation, repository coverage, no edits
Four MarkItDown cases	`evals/fixtures/markitdown/v1/source`	Expected document facts, converted reads, exact runtime JSON, no unintended edits
Three output-style cases	`evals/fixtures/output-style/v1/source`	Required information retained, no edits; no LLM judge
`prewalk-checkout`	Storzy `9cb213463222732cb955067953220d665b2f561b`	Type-check, both feature-flag builds, hidden checkout and API tests; Prewalk entry in its benchmark
`memory-summary-links`	Felan `0b461533f86c4d58521cdd7da7d9f298da03b123`	Hidden behavior verifier, workspace boundary, Prewalk entry in its benchmark
`rtk-felan-project-instructions`	Felan `104faa5559029c8be9e8a1eb504d87974a5864e9`	Agent Core type-check, focused public and hidden tests, build
`session-compaction-continuation`	`evals/fixtures/session-compaction/v1/source`	Sanitized 234-entry real session; native versus inherited extension ownership, summary fact ledger, continuation fidelity, no tools
`cbm-extension-architecture`	Felan `e5867637569bd1c7ad08420b79ec4031a5733f57`	Read-only file boundary, 150-word minimum, directional five-fact coverage check
`cbm-extension-config-scope`	Felan `51a18d8f0c853a06867ddbd48046ad4a84307058`	Workspace boundary, package build, existing and hidden behavior tests
`cbm-project-instructions`	Felan `104faa5559029c8be9e8a1eb504d87974a5864e9`	Agent Core type-check, focused public and hidden tests, build

Dependency-bearing cases install from their committed lockfile with pnpm install --frozen-lockfile. Network access is enabled only for that setup operation; setup runs before the measured agent attempt.
MarkItDown fixture setup runs the same installed converter in both arms with network disabled.
Verifier commands run with network.mode: none. Hidden grading assets are mounted for the verifier and are not exposed to the evaluated agent.
Whole-run infrastructure time is retained in detailed reports but is not the Codebase Memory duration objective.

4. Quality and eligibility

Correctness is a guardrail, not a resource metric. Every benchmark requires the candidate aggregate pass rate to be at least 0.8.

A run's pass status reflects execution status, required assertions, and any verifier process. The project score weights verifier reward at 1 and gives assertion-pass-rate and LLM-judge scores weight 0; required assertions still determine whether their run passes. No published comparison uses an LLM judge.

Baseline and candidate pass counts with quality-gate outcome
Benchmark	Baseline	Candidate	Gate result
Subagents	3/3	3/3	Passed
MarkItDown	12/12	12/12	Passed
Concise output	9/9	9/9	Passed
Prewalk	6/6	6/6	Passed
RTK	6/6	5/6	Passed at 83.3%
Session Compaction	2/3	3/3	Passed at 100%
Codebase Memory	9/9	9/9	Passed
Descriptive total	47/48	47/48	Gates are evaluated per benchmark

RTK timeout and resource observations

One candidate prewalk-checkout attempt timed out. It remains a failed attempt in the 5/6 aggregate pass rate, while its recorded cost and token observations remain in the three-trial median. The absolute 0.8 gate passed, despite a 16.7 percentage-point pass-rate decrease from the 6/6 baseline.

Codebase Memory pass status versus verifier reward

All 18 attempts reported status: passed and pass: true. Two baseline architecture attempts received reward 0.8 after matching four of five directional facts; every other reward was 1.0, including all three candidate architecture attempts. Reward magnitude and binary framework pass rate are therefore not interchangeable. Four architecture attempts, two per arm, also recorded a non-required clean-tool-execution assertion failure, which did not affect pass status.

Feature-use eligibility

A Prewalk candidate attempt without a successful enter_prewalk call is invalid. A Subagents candidate must make a successful Agent call with subagent_type: explore. Their baselines must make no corresponding call.

5. Metrics and aggregation

Every declared objective is minimized. A positive gain means the candidate used less of the measured resource; a negative gain means it used more.

cost.total: Provider-reported billed model cost in US dollars. This is the primary objective for all seven comparisons.

usage.promptTokens: Provider-reported prompt tokens. Secondary for MarkItDown and RTK.

usage.outputTokens: Provider-reported output tokens. Secondary for concise output style.

duration.stepsMs: Sum of measured agent-step durations in milliseconds. It excludes workspace setup, image probes, and verifier time; secondary for Codebase Memory.

Headline estimator

For each case and arm, take the median of its three numeric trial observations.
Sum those case medians separately for the baseline and candidate.
Calculate the goal-aware change from the two sums.

m_arm,case = median(trial 1, trial 2, trial 3)

gain = (Σ m_{baseline,case} − Σ m_{candidate,case}) ÷ Σ
              m_{baseline,case} × 100

This ratio of reduced sums is the published headline. It weights cases by the magnitude of their baseline resource value. The equal-weight macro mean of case gains and the case minimum/maximum remain diagnostics in detailed JSON; raw attempt totals are not the headline estimator. If a baseline sum is zero, the percentage is unavailable.

Published estimator inputs

Median-reduced baseline and candidate cost sums and resulting gains
Benchmark	Baseline cost sum	Candidate cost sum	Cost gain	Secondary reduced sums
Subagents	$1.285667	$0.98112684	23.687%	Not published
MarkItDown	$0.0554932	$0.0382732	31.031%	21,569 → 18,603 prompt tokens; 13.751% gain
Concise output	$0.229548	$0.196258	14.502%	4,187 → 3,500 output tokens; 16.408% gain
Prewalk	$8.52425	$2.899686	65.983%	None declared
RTK	$6.259809	$4.596907	26.565%	7,240,443 → 4,302,107 prompt tokens; 40.582% gain
Session Compaction	$0.49551	$0.27131	45.246%	83,304 → 39,891 prompt tokens; 52.114% gain
Codebase Memory	$5.727142	$4.135741	27.787%	1,167,282 → 1,076,719 step ms; 7.758% gain
Cost-weighted portfolio	$22.5774192	$13.11930204	41.892%	Seven independent primary comparisons

6. Evidence and provenance

The package combines direct runs, verifier-only reprocessing of retained workspaces, and one separately supplied redacted batch. Source relationships remain explicit.

Evidence type, source batch, and selected batch for each benchmark
Benchmark	Evidence type	Source batch	Selected batch	Reverification
Subagents	Direct, cost recalculated	`20260903-052334-7809`	`20260903-052334-7809`	—
MarkItDown	Offline verifier replay	`20260902-112808-33e8`	`20260903-045947-5857`	`20260903-045947-56b5`
Concise output	Offline verifier replay	`20260902-121757-89d1`	`20260903-045947-ba0e`	`20260903-045947-56b5`
Prewalk	Offline verifier replay	`20260902-121757-b16d`	`20260903-044517-8452`	`20260903-044517-d1f1`
RTK	Offline verifier replay	`20260902-121847-5fa4`	`20260903-045947-e11d`	`20260903-045947-56b5`
Session Compaction	Direct local pre-release	`20260911-154148-3bc0`	`20260911-154148-3bc0`	—
Codebase Memory	Derived from three local case batches	`20260908-090907-5a66` `20260908-114759-951e` `20260909-123711-ff11`	`20260908-090907-5a66` `20260908-114759-951e` `20260909-123711-ff11`	—

Offline reprocessing reran only the current network-isolated verifiers against retained workspaces. It made no agent or provider call, did not overwrite the source runs, and preserved source-run IDs. These derived batches carry publicationEligible: false because they are not independent provider-backed evidence; this package includes them only with their source relationship.

The RTK source batch contained a third memory-summary-links case. The selected publication reprocessed only prewalk-checkout and rtk-felan-project-instructions, matching the two-case benchmark definition. Runs from the unselected case do not contribute to the RTK figures.

Repository freeze

Commit: f7530cac88fb0d86e95bb713040d66df6abcadd3
Clean tree: true
Config SHA-256: 7a847181e54b618d92aa672bf7a5a6c29eceecacda0af7c4600ddbdfd600c170
Freeze-time path: harness-evals.yaml

Harness at freeze

Dependency spec: ^0.2.15
Local link: active
Linked version: 0.2.13
Linked commit: d5aaab2afc5f9536a86bbf8ffefd67def1c5c002

Session Compaction evidence qualification

The selected six-attempt batch used a local pre-release image containing an unreleased 0.1.2 artifact. The candidate used extension-owned compaction in all three attempts and passed its quality gate at 100%. The native baseline passed two of three because one continuation omitted the required black-box end-to-end publication plan. The later registry 0.1.2 tarball is not byte-identical; this provenance limitation is independent of candidate quality. See the full evidence.

Subagents descendant-cost correction

Native root-session telemetry omitted the candidate's child-session spend. For each candidate attempt, every child assistant message's usage.cost.total was summed and added to the native root cost before the median was taken. The correction covers nine child sessions and 75 child assistant requests. It changes the candidate median to $0.98112684 versus $1.285667 for the baseline, a 23.687% gain.

Only numeric cost aggregates, session/model identity, and source hashes were retained for that correction. Prompts, responses, tool output, and reasoning were excluded. Because an equivalent descendant-inclusive token aggregation was not established, the configured Subagents prompt-token objective is omitted from the publication.

Provenance manifestSelections, limitations, hashes, and package inventory Selected evidenceRun IDs, batches, quality, and source relationships Aggregation evidencePer-case medians and ratio-of-reduced-sums calculations Subagents cost evidenceDescendant-inclusive attempt costs and source hashes Codebase Memory evidenceAll 18 retained attempts in the final three-case scope Session Compaction evidenceSix retained attempts, ownership, quality, and pre-release qualification Frozen project configByte-preserved configuration at the repository freeze

7. Limitations

Three trials per arm and case are enough for a median, not for a confidence interval or a claim of statistical significance.
Results apply to the selected prompts, fixtures, repository commits, models, thinking levels, Felan releases, and run conditions. They do not establish universal extension performance.
Models, thinking levels, releases, task counts, and batch concurrency differ across benchmark rows. Compare candidate with baseline within a row; do not use the table to rank extensions against one another.
The 41.9% portfolio result is cost-weighted and dominated by higher-cost benchmark cases. It is not an equal-weight average and not an all-extensions-enabled measurement.
The RTK resource outcomes remain eligible under the configured absolute 0.8 floor even though candidate quality was 83.3% versus the baseline's 100%.
Subagents cost uses a documented manual recalculation. Its descendant-inclusive prompt-token result is unavailable.
Session Compaction candidate quality was 100% (3/3); the native baseline passed 2/3. The benchmarked local pre-release artifact is not byte-identical to the later npm 0.1.2 artifact.
The Codebase Memory result comes from three separately launched case batches without a benchmark stamp; its figures are recomputed from recorded run metrics, and it ran on Felan 0.23.2 while the other extensions ran on earlier releases.
Credentials, authentication and model stores, raw workspaces, full transcripts, and full session JSONL are excluded. The preserved files support aggregate audit, not full conversation replay.

8. Artifacts and integrity

Every benchmark has a linked HTML detail page. MarkItDown, concise output, Prewalk, RTK, Codebase Memory, and Session Compaction also include report-local JSON and CSV evidence; Subagents' corrected evidence is the dedicated provenance JSON.

The canonical top-level JSON records methodology, arms, cases, execution identity, quality, objectives, exact values, and selections. The top-level CSV is rectangular. Detailed benchmark CSV attempt rows retain the renderer's historical missing final empty column; those files remain standards-parseable.

SHA256SUMS covers every package file except itself. From this report directory, verify the complete package with:

shasum -a 256 -c SHA256SUMS