kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5b8a10c730f8fd6b98d32f77a0bd4828f62fbef2
set50-system/.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md

18 KiB
Raw Blame History

Event-Driven PIT Backtest Implementation Plan

For Hermes: Execute task-by-task. Use TDD for accounting/event logic, keep one task in progress, commit by phase, and run requesting-code-review before delivery.

Goal: Replace the current calendar-rebalance/backtest proxy with a strict point-in-time, event-driven portfolio simulation that reinvests sale proceeds and paid dividends, respects 100-share lots, records realized/unrealized P&L, and reconciles final equity exactly.

Architecture: Build a deterministic event calendar from data releases, execution dates, dividend entitlement/payment dates, and the end valuation date. Feed those events through a portfolio ledger that owns cash, holdings, average cost, dividend receivables, orders, realized P&L, and costs. The API derives readiness/default dates from actual PIT coverage and blocks rather than silently falling back to current data.

Tech Stack: Python 3.11, Flask, filesystem PIT stores, Vue/Vite, unittest/pytest.

Confirmed Decisions

Decision Chosen default Rationale
Backtest readiness Strict PIT coverage for every required scoring input; otherwise block and report missing coverage Prevents mixed-PIT results from looking valid
Signal execution Freeze signal on release date D; execute at next available trading-day close after D Avoids same-day close look-ahead
Dividend timing Entitlement at ex-date; cash becomes available exactly 30 calendar days after ex-date User-approved deterministic timing assumption; expose it as ex_date_plus_30d rather than claiming an observed payment date
Cost basis Average cost Simple, auditable, common portfolio accounting
Fees/taxes All-in fee 0.3% of notional on every buy and every sell; do not add VAT/tax again Explicit user decision; deterministic and auditable
Same-day releases Coalesce all releases known by the execution cutoff into one frozen signal and one rebalance Avoids churn from multiple events on the same day
End date on holiday/weekend Value at latest trading close on or before end date Deterministic and realistic

These decisions were confirmed by the user and are implementation requirements.

Current-State Gap Matrix

Requirement Current behavior Gap
Default start = first date advice is genuinely available API/UI hardcode 2024-06-01 No readiness calculation; current disk has no factor vintages or Siamchart vintage manifest
Default end = yesterday API/UI hardcode 2026-06-01 On 2026-08-28 +07, expected default is 2026-08-27
Recompute only when new information arrives _rebalance_dates() emits monthly/quarterly dates (backend/app/backtest.py:103-125) Calendar-driven, not release/event-driven
Strict PIT Optional score_fn; current data fallback exists Current local data has 0 factor-vintage files and 0 Siamchart vintage snapshots, so no strict PIT-ready start exists
Execute after signal is known _latest_close(..., d) uses close on rebalance date Can use the same close that occurred before/while data was released
Dividend entitlement and cash timing credit_dividends() is called only at end, using final holdings (backtest.py:253-268) Wrong holdings-at-ex-date, no mid-run cash credit, no reinvestment, historic dividends can be overcounted
Sale P&L Sell proceeds update cash only (backtest.py:232-241) No cost basis, realized P&L, fees, or per-trade ledger
Reinvest gains/dividends Sale cash can fund buys in the same rebalance; dividends are added only after all rebalances Dividends cannot fund later buys; profit/cash trace is absent
100-share lots allocate_capital() floors to 100 shares Already present; must preserve during target reconciliation
Final report final holdings, total value, combined price P&L, dividend total Must split realized/unrealized P&L, cash, receivables, costs, trades, and prove reconciliation
Current data coverage Price snapshot: 9 symbols, 2024-01-03 to common 2026-07-17; no factor vintages/Siamchart manifest on disk Strict PIT backtest should currently block, not manufacture a default start

Target Event Flow

Coverage/readiness
  → earliest date where ALL PIT inputs + executable prices exist
  → initial recommendation event at start
  → freeze target signal
  → next trading day execution (100-share lots)
  → for each data release:
       coalesce same-day changes
       recompute recommendation from PIT state
       if target changes: sell first → credit proceeds/realized P&L → buy using current cash
  → for each dividend:
       ex-date: record entitlement using shares held before ex-date
       ex-date + 30 calendar days: credit cash; available to next rebalance
  → end date:
       mark holdings to latest close ≤ end
       report cash + market value + dividend receivable (separate)
       reconcile all P&L components

Accounting invariant:

ending_equity - initial_capital
  = realized_trading_pnl
  + unrealized_trading_pnl
  + dividend_cash_received
  + accrued_dividend_receivable
  - transaction_costs

price_pnl = realized_trading_pnl + unrealized_trading_pnl for compatibility.

Task 1: Add Strict Backtest Coverage and Default-Date Readiness

Objective: Derive valid start/end defaults from actual PIT stores and block when coverage is incomplete.

Files:

  • Create: backend/app/backtest_readiness.py
  • Modify: backend/app/factor_vintages.py
  • Modify: backend/app/siamchart_vintages.py
  • Modify: backend/app/prices.py or add a read-only adapter for price coverage
  • Modify: backend/app/__init__.py
  • Test: backend/tests/test_backtest_readiness.py

Steps:

  1. Write failing tests for:
    • no factor vintages → ready=false, missing factor list
    • no Siamchart snapshot → blocked
    • price coverage starting after PIT factors → recommended start equals latest of first-ready dates
    • recommended end = yesterday in Bangkok, bounded by latest price date
    • user start earlier than readiness → HTTP 400 with exact missing coverage
  2. Add store coverage APIs:
    • FactorVintageStore.first_release(factor_key) and coverage summary
    • SiamchartVintageStore.first_retrieved_at()
    • price coverage: earliest/latest executable trading dates by required universe
  3. Implement BacktestReadiness response:
    • ready, recommended_start, recommended_end, missing, coverage, timezone
  4. Add GET /api/v1/backtest/readiness.
  5. Change POST /api/v1/backtest defaults to readiness-derived dates; never hardcode dates.
  6. Verify targeted tests and API smoke.

Acceptance: Current local dataset returns ready=false because factor/Siamchart PIT history is absent; it does not start a purported PIT run.

Task 2: Build the Unified Backtest Event Calendar

Objective: Rebalance on information changes, not arbitrary monthly/quarterly dates.

Files:

  • Create: backend/app/backtest_events.py
  • Modify: backend/app/factor_vintages.py
  • Modify: backend/app/siamchart_vintages.py
  • Modify: backend/app/dividend_ledger.py
  • Test: backend/tests/test_backtest_events.py

Steps:

  1. Define typed/dataclass events:
    • SignalReleaseEvent(released_at, sources, factor_keys)
    • ExecutionEvent(signal_date, execution_date)
    • DividendEntitlementEvent(ex_date, symbol, per_share)
    • DividendPaymentEvent(assumed_payment_date=ex_date+30d, entitlement_id, timing_method="ex_date_plus_30d")
    • EndValuationEvent(end_date)
  2. Add chronological iterators to factor and Siamchart vintage stores.
  3. Build release calendar from all PIT changes within [start, end].
  4. Coalesce releases on the same date.
  5. Map each signal date to the next available trading date strictly after the release date.
  6. Include dividend entitlement/payment events and final valuation event.
  7. Create an assumed payment event exactly 30 calendar days after ex-date. Keep the dividend as a receivable until that event, then credit spendable cash. Expose the timing assumption in every event/result.
  8. Test ordering, holiday/weekend handling, coalescing, and no-look-ahead execution.

Acceptance: A factor release three months after start generates one new frozen signal and, only if targets differ, one rebalance on the next trading day.

Task 3: Implement the Portfolio Accounting Ledger

Objective: Own all cash, holdings, cost basis, trades, dividends, and P&L in one auditable component.

Files:

  • Create: backend/app/portfolio_ledger.py
  • Test: backend/tests/test_portfolio_ledger.py

Data model:

Position(symbol, qty, average_cost)
Trade(date, signal_date, symbol, side, qty, price, notional,
      cost_basis_released, realized_pnl, fees, cash_after, reason)
DividendReceivable(symbol, ex_date, assumed_payment_date,
                   timing_method="ex_date_plus_30d", qty_entitled,
                   per_share, amount, status)
PortfolioState(cash, positions, receivables, realized_pnl,
               dividend_cash, fees, trades)

Steps:

  1. Write RED tests for average-cost buys, partial sells, full sells, realized P&L, and no negative cash.
  2. Implement buy():
    • qty positive and multiple of 100
    • all-in fee = notional * 0.003
    • cost + fee <= cash
    • recompute weighted average cost
  3. Implement sell():
    • qty positive/multiple of 100 and <= position
    • all-in fee = notional * 0.003
    • realized P&L = proceeds - released average cost - sell fee
    • credit net proceeds (notional - fee) to cash before later buys
  4. Implement dividend entitlement at ex-date using quantity held before ex-date.
  5. Implement payment-date cash credit; paid dividend can fund later orders.
  6. Keep unpaid dividends as receivables, not spendable cash.
  7. Add mark-to-market and reconciliation methods.
  8. Test cash conservation and accounting identity after mixed buys/sells/dividends.

Acceptance: Profitable sales and paid dividends increase cash available to subsequent 100-share-lot purchases; unpaid receivables do not.

Task 4: Reconcile Target Portfolio with Lot and Cash Constraints

Objective: Convert each frozen recommendation into executable sell/buy orders using current total equity and current cash.

Files:

  • Modify: backend/app/simulation.py
  • Create or modify: backend/app/portfolio_rebalancer.py
  • Test: backend/tests/test_portfolio_rebalancer.py

Steps:

  1. Preserve canonical 50/20/30 allocation rules and 100-share lots.
  2. Compute target quantities from current equity (cash + market value + paid dividends; exclude receivables).
  3. Generate sells first, execute them, then recompute available cash.
  4. Generate buys from remaining target deficits, never exceeding cash.
  5. If a target lot is unaffordable, skip it and retain cash; never buy odd lots.
  6. Rebalance only when target quantities differ from current quantities.
  7. Add tests for:
    • sale profit funds next purchase
    • dividend payment funds next purchase
    • insufficient cash leaves cash and skips order
    • unchanged recommendation produces zero trades
    • no negative cash under rounding/fees

Acceptance: Every trade is in 100-share multiples, and ending cash is always non-negative.

Task 5: Rewrite Backtest Engine as Event Processor

Objective: Replace _rebalance_dates() calendar loop with the event calendar and portfolio ledger.

Files:

  • Rewrite: backend/app/backtest.py
  • Modify: backend/app/pit_scorer.py
  • Test: backend/tests/test_backtest.py
  • Test: backend/tests/test_backtest_integration.py

Steps:

  1. RED test the complete scenario:
    • start recommendation buys A
    • release three months later changes target to B
    • sell A at profit
    • dividend entitlement/payment occurs
    • sale proceeds + paid dividend fund B purchase
    • final holdings/value and P&L reconcile
  2. Initialize from strict PIT readiness and initial signal.
  3. Process events in order:
    • release → freeze scores/target
    • execution → reconcile portfolio
    • ex-date → entitlement
    • payment → cash credit
    • end → mark-to-market
  4. Record every release, rebalance, trade, and dividend event.
  5. Ensure leakage_guard=true only if every score at every release is PIT; otherwise reject strict mode.
  6. Remove monthly/quarterly frequency as the primary trigger. Optionally retain a separate explicit scheduled research mode, clearly non-default/non-event-driven.
  7. Verify no double-counted dividends and no repeated historical credits.

Acceptance: Engine output exactly follows the user-described lifecycle and passes accounting reconciliation for every tested scenario.

Task 6: Expand Backtest Result Contract and Durable Run Storage

Objective: Return all requested end-state and P&L breakdowns with a durable audit trail.

Files:

  • Modify: backend/app/backtest.py
  • Create: backend/app/backtest_store.py
  • Modify: backend/app/__init__.py
  • Test: backend/tests/test_backtest_store.py
  • Modify: API tests in backend/tests/test_api.py

Response v2:

{
  "start": "...",
  "end": "...",
  "initial_capital": 1000000,
  "ending_cash": 12345,
  "ending_market_value": 1100000,
  "dividend_receivable": 500,
  "final_equity": 1112845,
  "net_change": 112845,
  "net_return": 0.112845,
  "realized_trading_pnl": 45000,
  "unrealized_trading_pnl": 50000,
  "price_pnl": 95000,
  "dividend_cash_received": 18000,
  "transaction_costs": 155,
  "fee_rate": 0.003,
  "holdings": [{"symbol":"...","qty":100,"average_cost":...,"last_price":...,"market_value":...,"unrealized_pnl":...}],
  "trades": [...],
  "dividends": [...],
  "rebalance_events": [...],
  "accounting_reconciled": true,
  "leakage_guard": true
}

Steps:

  1. Add result dataclasses and compatibility aliases (final_value, dividend_income).
  2. Add durable JSON run store (atomic writes) rather than process-local backtest_runs.
  3. Persist exact input coverage, event calendar, and scorer provenance with each run.
  4. Add reconciliation assertions before serializing; fail closed if imbalance exceeds tolerance.
  5. Test reload after app restart and API history output.

Task 7: Update Frontend Defaults, Controls, and Report

Objective: Make UI defaults/readiness and results reflect the new engine honestly.

Files:

  • Modify: frontend/src/App.vue
  • Test/build: frontend/package.json existing build

Steps:

  1. On load, call /api/v1/backtest/readiness.
  2. Set start to recommended_start and end to recommended_end; remove hardcoded 2024-06-01 / 2026-06-01.
  3. If readiness is blocked, disable Run and show exact missing coverage.
  4. Replace “รายเดือน/รายไตรมาส” primary control with “ตามข้อมูลใหม่ (event-driven)”. Keep scheduled frequency only under advanced/descriptive mode if retained.
  5. Display:
    • final holdings with qty, average cost, latest price, market value, unrealized P&L
    • final equity, net change, return
    • dividend cash received and dividend receivable
    • realized trading P&L, unrealized P&L, total price P&L, fees
    • event timeline / trade table
  6. Keep visible PIT/non-PIT and dated-ledger/proxy badges.
  7. Run production build and verify rendered result state.

Task 8: Verification and Independent Review Gate

Objective: Prove correctness before shipping.

Files:

  • Create: docs/test-evidence/YYYY-MM-DD-event-driven-backtest.md
  • Update: docs/engineering-log.md
  • Update: docs/HANDOFF.md

Required tests/gates:

  1. Targeted event-calendar suite.
  2. Portfolio-ledger suite.
  3. Rebalancer suite.
  4. Full backtest integration suite.
  5. Full backend suite.
  6. Frontend production build.
  7. Static security scan; no secrets/debug leftovers.
  8. git diff --check, compileall.
  9. Live API probe with a deterministic temporary PIT fixture.
  10. Browser verification of readiness-blocked and completed-result states.
  11. Fresh independent reviewer subagent; fail closed on any logic/security blocker.
  12. Auto-fix/re-review max two cycles.

Critical invariants:

  • cash never negative
  • all buy/sell quantities multiples of 100
  • no trade before signal is knowable
  • no dividend entitlement for shares bought on/after ex-date
  • unpaid dividends never fund purchases
  • paid dividends can fund subsequent purchases
  • realized P&L uses average cost
  • unchanged target creates no trade
  • final accounting identity reconciles within one satang/rounding tolerance
  • current local data returns readiness blocked rather than fake PIT performance

Files Likely to Change

  • backend/app/backtest.py
  • backend/app/backtest_readiness.py (new)
  • backend/app/backtest_events.py (new)
  • backend/app/portfolio_ledger.py (new)
  • backend/app/portfolio_rebalancer.py (new)
  • backend/app/backtest_store.py (new)
  • backend/app/dividend_ledger.py
  • backend/app/factor_vintages.py
  • backend/app/siamchart_vintages.py
  • backend/app/prices.py
  • backend/app/simulation.py
  • backend/app/__init__.py
  • backend/tests/test_backtest*.py
  • new focused backend test modules
  • frontend/src/App.vue
  • engineering/test-evidence docs

Explicit Non-Goals

  • Live order execution / MT5 dispatch
  • Guessing historical factor values that were never recorded
  • Claiming PIT performance before coverage readiness passes
  • Broker-specific fee assumptions without configuration
  • Odd-lot trading

Main Risks

  1. No current strict PIT-ready window: the local store has no factor vintages/Siamchart manifest. The correct initial UI state is blocked.
  2. Dividend payment timing is assumed: current Siamchart data supplies ex-date+DPS, not payment date. The user confirmed assumed_payment_date = ex_date + 30 calendar days; every result must disclose timing_method=ex_date_plus_30d.
  3. Price quality: current Yahoo revised history is explicitly non-PIT; strict validated backtests need an evidence-bearing price archive contract.
  4. Universe coverage: current price snapshot contains 9 symbols, not the full 49-symbol board.
  5. Response compatibility: current UI/API expect scalar holdings and price_pnl; v2 needs compatibility aliases during migration.

Open Questions

None for the core accounting/event model. The implementation decisions above are confirmed.

Reference in New Issue View Git Blame Copy Permalink