Walk-forward backtest — 927 tickers (500 SP500 + 501 Russell 1000 supplement)

Headline numbers (monthly forward hold)

Cumulative monthly group means (vs benchmark)

Compounded equal-weight monthly mean returns for each cohort, anchored at 2020-12-31. The three curves move together — the headline paired t-test confirms the visual: no persistent separation. Note near_critical exhibits the largest drawdowns (2022 tightening; H1 2025 vol shock).

Raw data: /api/backtest-cumulative · /api/backtest-result

Pre-registered limitations

These caveats were known before the test ran. They define what this result can and cannot say.

1. Label leakage risk. All 500 Wave 1 StructTuples were classified once at 2026-05-14, then applied retroactively to every snapshot from 2020-12 onward. The model may have used information unavailable at the historical snapshot date. A per-snapshot label refresh (re-classify each company as of each T) is the strict-causality version of this test and remains the single most expensive — and most likely to flip the result — experiment we have not yet run.
2. Russell 1000 supplement is a curated 501-name sample, not the official R1000. FTSE Russell licenses membership; we cannot redistribute the full list. Our supplement is sector-balanced and large/mid-cap, but it is not point-in-time R1000 reconstitution. True point-in-time membership would mitigate this.
3. Survivorship bias. 74/1001 tickers were dropped due to delisting / yfinance edge cases. We did not include historically-listed-then-delisted tickers in our universe, biasing benchmark return upward by ~1–2% annually.
4. Static labels on R1000 supplement (no labels at all). Only the 500 SP500 tickers carry LLM critical_point_state labels. The heuristic price-regime cohort uses a 90-day vol + 180-day drawdown placebo — a baseline, not a Phase Detector test.
5. Horizon sweep was minimal. 1-month and 6-month forward holds reported; no full 3/12/24-month grid. Any post-hoc sweep (horizon × threshold × sector × dynamics-family) needs Benjamini-Hochberg correction.
6. No transaction costs, borrow costs, or sector neutralisation. Pure equal-weighted long-only cohort means. Real portfolio implementations would face additional decay.

What’s next

The null result is informative but specification-bound. Per W7-D, we proceed with structured-research-narrative positioning. Backlog (priority-ranked, cheapest first) for v0.2 if positioning traction warrants:

1. Sector-stratified test (cheap, 1 day). Run the paired t-test within each GICS sector. The signal may live in healthcare or tech and be diluted in financials/utilities. Benjamini-Hochberg correction over 11 sectors.
2. dynamics_family stratified test (cheap, 1 day). 9 families × 2 cohorts = 18 tests, BH-adjusted. The signal may live in motter_lai_cascade or scheffer_fold companies specifically.
3. Confidence-filtered cohort (cheap, 0.5 day). Restrict near_critical to confidence ≥ 0.8 labels only (54% of universe). Reduces cohort size but may sharpen signal.
4. Per-snapshot label refresh (expensive, $500–$1k LLM cost, 2–3 days). Re-classify each company at each historical T using only information available at that T. Removes the leakage risk noted above. This is the single most likely experiment to flip the result.
5. Russell 2000 universe (medium cost). True small-cap is where phase transitions should bite. R2000 = 2000 names, fetch cost ~10 min, label cost ~$2–$5k if we LLM-classify the new 2000 tickers.