Accuracy, measured

Section extraction and diffing are evaluated against a public golden set of real EDGAR filings, including amended filings and known-messy layouts. The dataset and harness are in the open repo; every number below is reproducible with node eval/run.mjs.

Last run 2026-08-25 · parser 2026-08-25.1 · JSON at /eval.json

Section boundaries

cases                    88
passed                   80
pass rate                90.9%
  non-amended filings    90.8%
  amended filings (/A)   91.7%

A case passes only if the extracted section starts at the real item heading, contains none of the table-of-contents signature, does not run into the next item, stays inside a plausible length band, and contains the hand-verified anchor phrases for that filing. Any single failed check fails the case.

Diff precision and recall

perturbation trials      45
sentence precision       100.0%
sentence recall          100.0%
false-positive reformats 0

Method: real extracted sections are perturbed with known edits (sentences removed, sentences inserted, and cosmetic reformatting that must NOT register), then the diff's reported changes are scored against the known edit set. Ground truth is constructed, so precision and recall are exact, and a reformat that shows up as a change counts against us.

Open failures

Case	Why it fails
jpm-0001628280-26-008131-7	extraction returned null
jpm-0001628280-26-054343-2	item title not at the start of the section
intc-0000050863-26-000011-1A	implausibly short (35 chars)
intc-0000050863-26-000011-7	extraction returned null
intc-0000050863-25-000009-1A	implausibly short (35 chars)
intc-0000050863-26-000157-2	item title not at the start of the section
ibm-0000051143-26-000010-7	extraction returned null
wfcf-0001493152-22-009182-1A	expected no section for this filing, but text was returned

Failures stay listed until fixed; they are the to-do list, not a secret.

Reproduce it

git clone https://github.com/hgenix20/signalnodus
cd signalnodus && node eval/run.mjs

The harness fetches the pinned accessions straight from SEC EDGAR and runs the same parser the live service runs. Found a filing we get wrong? Email hgenix@agentmail.to with the accession; confirmed misparses join this set.