End-to-end testing loop

v1.6 · 18 Oct 2026 — Shared A+ setup library (before any A+ ships)

Research only. Added per David (voice, 18 Oct 2026, 3:19 PM AEST). This sits in the procedures-before-testing area: no A+ definition ships until the library check is done and logged.

Current library version: see _setup_library/VERSION. Sibling page: Testing Processes carries the same rule.

v1.5 · 18 Oct 2026 — Loop first: nothing runs until this is in place

Research only. Added per David (voice, 18 Oct 2026). Source: /workspace/specs/farm_scheduler_spec_2026-10-18.md. Older content below is kept; the $5 commission rule is replaced by real fees from /workspace/farm_fees/. The gated sequence, stage-one matrix and top-20 ranking follow in the next section.

Stage One: testing instruments and timeframes · power loop

7 instruments × 4 timeframes = 28-combination grid, Hunter 7 sifted by Guardian, in the farm context.

Note: US30 and YM are both the Dow — keep both or drop one is pending David.

Flag — timeframes: the stage-one matrix below lists 1m · 5m · 15m · 1h · daily, but the Stage One grid is 1 · 3 · 5 · 15 minute. David to confirm.

Flag — test periods not locked: lanes use IS 2025 / OOS Jan–Jun 2026 / HO Jul–5 Oct 2026; older doc text said Jan–Apr / May–Jul / Aug–9 Oct; guardian.py splits on 1 Jun 2026; the scheduler spec proposes OOS 1 Apr–5 Oct 2026. David to confirm one split.

• Bias combos: B1–B5.
• Regime tagging: method TBD (pending David).
• Session mix: results split and compared by session.

Core cycle (David, 18 Oct 2026)

Test → Test more → Optimise → Backtest → Review the features summary (what to add, what's missing, how to perfect results) → back to Test. This cycle repeats inside the six-stage flow below.

The testing loop (v1.5 order)

1. Data readiness + gap check — the day's data download confirmed complete, quality-checked, known gaps excluded. Incomplete data = no tests start that day.
2. Python grid — fast parameter/instrument/timeframe grid, real fees applied.
3. Validation on farm data — rerun survivors on the farm's NinjaTrader data.
4. Hunter 7 + Guardian hybrid — Hunter 7 entries sifted by Guardian.
5. MT5 real-tick tester — straight tester confirmation, real ticks.
6. Forward demo — demo only, never live.

Daily top 10 / top 20 per playbook, net of real fees, each with its curve. A new version becomes a candidate only after beating the current one twice; nothing swaps until David says “promote”.

Pre-test protocols checklist

• Real broker fees only, stated on every result: NinjaTrader Lifetime ES/NQ $4.34 RT, MES/MNQ $1.28 RT · IBKR MGC $1.92 RT, GC $5.02 RT · Global Prime XAUUSD $7/lot RT ($3.50/side). No placeholder fees.
• Every stat shown with its equity curve.
• Known data gaps excluded.
• In-sample / out-of-sample split defined before the run.
• Minimum trade floor met (value pending David).
• Both halves (IS and OOS) profitable.
• Overfit checks: random-entry comparison and neighbour-setting stability.
• Every job logged on the Queued Jobs page when it starts and marked when it ends.
• Research only until David says “promote”.
• Back up before any page edit.
• Shared A+ library (v1.6): every A+ definition checked against /workspace/nitro-setups/top_traders/_setup_library/ current VERSION before publish / entry-contract handoff / preset change / test lane; check logged in CHANGELOG.md. One library for all builders — never a separate copy.

Standing rules (every stage)

Testing sequence and matrix for Stage One: testing instruments and timeframes (v1.5 · David, 18 Oct 2026)

Plan only: writing this down started no runs. Each step is gated. A step does not start until the one before it is done.

Matrix for Stage One: testing instruments and timeframes

This layers on the instrument levels below (Stage 1 = trader defaults, normally NQ + GC · Level 1 = same defaults with matrix + in-loop self-learning · Level 2 = futures → forex → crypto → stocks, missing data skipped and logged). The dimensions are crossed together:

The sequence at a glance

↺ Stage 6 feeds back: the next trader's Stage 1 starts from the updated A+ candidates and farm memory.

Instrument levels (David lock, 6 Oct 2026)

Level 1 is solid (automatic, no human call) when forensic 2b = PASS, the learner stopped cleanly on a stall guard, and the best config has IS net > 0 and locked OOS net > 0 with ≥ 60 trades across IS+OOS. If it is not solid, Level 2 is skipped with that reason and the run closes DO NOT PROMOTE.

Level 2: the Level-1 locked configs carry over unchanged to every selected symbol, with an optional per-group IS-only re-tune. Level-2 results are cross-instrument robustness evidence and never change the Level-1 promote call.

Instrument coverage, data and costs

Box inventory 6 Oct 2026. Windows (farm map): IS 2026-01-01 → 04-30 (stocks 03-12 → 04-30) is the only tuning stage · OOS 05-01 → 07-31 locked · HO 08-01 → 10-09 locked, clipped to data end · 21-day warm-up.

Other FILLED cross-asset items (switchable, tagged): non-NQ/ES price thresholds scaled by IS ATR ratio vs NQ; SMT pairs GC↔SI, stocks↔QQQ, QQQ↔SPY (SMT lineups skipped when the pair is missing); stock size = floor($500 ÷ stop) shares per leg, with ranking also shown in R.

The six stages

1. Ingest rules and code (NinjaTrader + Python). Done when: the trader's locked rules template and entry plugin / preset exist on both sides, for the trader's default instruments only (normally NQ + GC). The other groups are only inventoried for Level 2; their bars are not loaded. A smoke failure is a hard fail.
   • Locked rules template, versioned and named for the trader's main setup (e.g. CRT / ORM / iFVG).
   • Verified trade list beside the master rules (video ID@timestamp, instrument, date, entry time ET, price, direction; stop/target only if stated).
   • Python preset + entry plugin under smt-python; defaults match the locked template; new switches default OFF.
   • NinjaTrader: same parameters in the current All-in-One NT file; backup before edit (.bak_pre_<trader>).
   • Port map / FILLED tags updated; smoke test — both sides load, the default-off path does not change other traders.
   Writes: locked template path, NT .cs path, Python preset/plugin paths, verified-trades file.
2. Baseline reproduction (in-range set). Done when: every in-range verified trade is classified, with a blocking rule on misses. Default instruments only. No verified trades → research mode (figures labelled RESEARCH; promote forced to DO NOT PROMOTE).
   • Run code as-is (locked defaults, judgement OFF) on clean data for each trade date (ET, tick-rounded, roll weeks handled). No tuning.
   • Match each trade on entry time (within one bar), entry price (stated tick tolerance) and direction.
   • Classify: REPRODUCED · PARTIAL (say which of time/price/dir is off) · MISS (name the gate) · DATA_GAP / untestable.
   • Report reproduced / in-range, the full trade table and the data-gap note.
   • 2b Forensic pass (gate into Level 1, automatic): every lineup component fires on ≥ 1% of IS days; entry = next 1m open; no trade outside the window or after 15:55; SL/TP1/TP2 = stop-ref / 1R / 2R; no fills on roll-drop or missing-bar days; ET clock sanity; net_usd = gross − $5 RT; day cap respected. If it fails, Level 1 is NOT SOLID, Level 2 is skipped, and the run still finishes.
   Writes: REPRO_<TRADER>.md, repro_<trader>_trades.csv, baseline run JSONs (signals + closed trades with net_usd).
3. Parameter / judgement sweeps + in-loop self-learning. Done when: try-all and per-switch results are recorded, the self-learning loop stopped on a named stall guard (3a), and Level 2 ran or was skipped with its reason (3b).
   • 3a Level 1 (default instruments): round 0 = FILLED trigger matrix (76 lineups, or a named focused set) × default instruments, with static CDX exits (TP1 1R / TP2 2R, 1+1, no BE, no trail, same-bar = stop first, flatten 15:55, $5 RT). Then self-learning rounds use IS only (see Self-learning). Then the top-5 configs get locked OOS + HO once.
   • 3b Level 2 (after Level 1 solid): futures → forex → crypto → stocks, farm-selected symbols; Level-1 configs carried unchanged; optional per-group IS re-tune; missing data = logged skip.
   • Identify judgement / contradiction switches (default OFF).
   • Run try-all ON, then each switch alone to attribute unlocks and regressions.
   • Table: switch → unlocked trade IDs → regressions → keep default OFF? yes/no.
   • Never promote a switch that raises one trade but regresses a locked reproduce without an explicit David decision. Data-gap note on the sweep report.
   Writes: *_JUDGEMENT_TRYALL_*.md, per-tag run JSONs, score JSON (repro/partial IDs per tag).
4. Dollar PnL from run logs. Done when: dollars are attached to hits, partials and any standalone profitable engine trades — not score alone.
   • From each run JSON read closed trades: opened_at, entry, exit, reason (stop / target / be_stop), net_usd (already net of $5 RT).
   • Map scorecard signal bars to closed trades (±1–2 min); note signal-only with no closed trade.
   • Report whole-run book (count, total net_usd, winners vs losers), per REPRODUCED and per PARTIAL net USD, and any standalone profitable engine trades.
   • Never invent PnL for a miss from an unrelated same-day fill.
   • Equity curve (mandatory — every result): running equity, max DD from peak, longest losing streak, win clusters, largest trade $ and % of total, top-2 winners' share, date range — labelled smooth vs lumpy. Deliver in the same report as the nets. Baseline, every sweep, matrix cell, OOS/HO, experiment.
   Writes: PnL section (markdown or CSV) linked from the scorecard, equity-curve table/JSON, paths to the run JSONs used.
5. Scorecard package (paths ready). Done when: one folder holds everything Stage 6 will consume.
   • Collect baseline repro report + CSV, try-all / per-switch report, PnL extract, data-gap notes, NT/Python commit or Drive paths.
   • Confirm in-range count and ID lists (repro / partial / miss / gap) agree across files.
   • Name the promote candidate config tag (usually baseline locked defaults; sweeps are diagnostic unless David locks a switch ON).
   Writes: scorecard_inputs/ (or dated lane folder) with a short MANIFEST.md listing paths.
6. Scorecard + Hermes. Done when: the scorecard is written with a promote call and Hermes has run on it and its revisions are committed to the A+ setup library and entry-plugin lane.
   • 6a Scorecard: trader, date, template version, NT + Python paths; baseline score; sweep unlocks/regressions; dollar PnL; equity curve; data gaps; instrument levels (Level 1 solid? Level 2 groups run/skipped); automatic promote call PROMOTE_CANDIDATE (a candidate file for David, never live) · DO NOT PROMOTE, with a one-line reason. “HOLD — needs David” is now a non-blocking open question.
   • 6b Hermes (inside this stage, never a "stage 7"): reads the scorecard, PnL, per-switch results and rule-match details; returns revised A+ definitions, entry-param suggestions, a do-not-enable list and open questions for David — as concrete artifacts.
   • Autonomous form: the loop runs the local Hermes learner, autocommits candidates (a_plus_gallery/_hermes/<run_id>/, A_PLUS_LIBRARY_candidates/, presets/hermes_candidates/, farm_memory/), and sends the Hermie agent an async copy without waiting. Live presets, the A+ library v1 and gate.yaml are never auto-edited.
   • Commit accepted A+ updates (/workspace/nitro-setups/a_plus_gallery/) and entry-param updates before the next trader is ingested. FILLED switches stay default OFF unless David locks one ON. Record the Hermes run id + artifact paths on the scorecard footer.
   Writes: SCORECARD_<TRADER>_<DATE>.md, Hermes return bundle path, updated A+ defs, updated entry-plugin params, footer link on the trader Dev page (Dev only).

Self-learning loop (in-loop, no manual step)

1. Round 0. Every lineup × every ready symbol at locked defaults gives an IS objective per config, pooled across the level's symbols.
2. Rounds 1…N. The Hermes-local proposer reads IS results only and proposes new variants: lineup beam (add one component to the top IS lineups, max 5), parameter mutation (window bars/start/end, CISD series/look-ahead, iFVG age, SL buffer, gap and risk-cap scale, trades/day), and a surrogate model that ranks about 10× the candidates and keeps the best 24 per round. Each variant runs and is scored automatically.
3. Objective (FILLED, farm judgement): IS net $ − 0.5 × IS max DD; ≥ 30 IS trades; bonus if both IS halves are positive. Exits stay fixed at the CDX standard.
4. Locked evaluation. The top-5 configs get OOS and HO once. The KEEP gate mirrors hermie_loop/gate.yaml: IS beats base; OOS and HO net > 0; avg trade ≥ base − 10%; ≥ 100 trades; ≥ 90 days; max DD ≤ 1.2 × base. Locked numbers never reach the proposer.
5. Permanent memory. Every trial key and IS score goes to farm_memory/trials.jsonl, so later runs never repeat a config and warm-start from the best priors.

Stall guards: what stops it from stalling

Thorough result set complete = round-0 matrix on every ready Level-1 symbol × every lineup; learner stopped on a named guard; top-K locked-evaluated once; every book has trades + equity curve + summary; Level 2 run for every ready selected symbol (or skipped with reason); every skipped symbol listed with its gap; ranked report, scorecard, Hermes input/return and manifest written.

How the stages chain (autonomous loop)

# end-to-end testing loop — one command, no human checkpoints
run_e2e_loop.py --config config.json --execute
for trader in build_order:
    Stage1  ingest                 # trader DEFAULT instruments only (normally NQ + GC)
    Stage2  baseline repro         # default instruments; research mode if no verified trades
    Stage2b forensic               # automatic; PASS gates Level 1
    LEVEL 1:
        Stage3a matrix round 0 -> self-learning rounds (IS only) -> stall-guard stop
        Stage4  PnL + equity curve every book
        locked OOS + HO once for top-K -> KEEP / REJECT
    if Level1 solid:
        LEVEL 2: Stage3b futures -> forex -> crypto -> stocks (farm-selected; gaps skipped + logged)
    Stage5  package
    Stage6  scorecard (auto promote call) + Hermes (local learner, candidate autocommit, async Hermie copy)
    # next trader Stage1 starts from updated A+ candidates + farm_memory priors

Stops only on a hard fail:

Everything else is logged and the run finishes with a written call: missing data, research mode, forensic fail, no KEEP config. An explicit “stop” from David kills the process; the loop does not wait for it as a checkpoint.

Proceedings — what gets written at each stage

Every result above carries the data-gap note and, from Stage 2 on, the equity curve. Net figures are after $5 round-turn commission.