set50-system/.hermes/plans/2026-08-26_research-sources-backtest.md

Research + Data-Source + Simulation Backtest — plan (5-point)

For Hermes: execute after user approval. This covers:

1. Literature review → formula improvements
2. New data sources per theme
3. Frontend sources table = NOT mock, auto from backend FACTORS registry
4. Simulation: real backtest engine w/ custom start/end date + P&L
5. Sources table shows next-update column

Findings (facts, verified)

Sources table is hardcoded (point 3 true): dashboard._build_sources returns a hardcoded 5-row list (TradingEconomics, BOT NPL, Thai Oil, BOT Economy, Tourism). It does NOT come from the FACTORS registry, so adding a factor does NOT add a source row automatically. Must wire sources from FACTORS registry (single source of truth).

Simulation is NOT a real backtest (point 4 true): simulation.allocate_capital allocates once at latest price. Both "backtest" and "forward" modes just differ in whether an MT5 order is sent. There is no start/end date, no rebalancing over time, no performance/P&L, and results are not persisted (vanish on refresh). This matches the user's confusion exactly — the two modes don't do what the labels imply.

5 data sources today (real): TradingEconomics (auto sales) · BOT NPL · Thai Oil (TOP quarterly) · BOT Thai Economy (macro) · BOT Tourism — all real, but hardcoded in the sources table.

---

Work items

1. Literature / methodology research → formula improvements

• Research: factor-based Thai-equity scoring, alternative-data alpha, market-regime timing, cross-frequency factor combination (mixed cadence). Deliver: a short recommendation doc with concrete refinements to the current formula (60/40, quality_within_theme, regime gate), grounded in citations.
• Output: docs/methodology-research.md; proposals applied only after user OK.

2. Additional data sources per theme

• Research & (where cheap/feasible) wire one new honest Thai source per theme:
  • banks → BOT commercial-bank loan growth / NPL (real, report already seen)
  • retail → consumer-comfort/retail index
  • utilities → BOT electricity demand proxy
  • telecom → subscriber data (if public)
  • property → BOT housing / transfer volume
• Add as FACTORS registry entries (declarative — that's the whole point). Sources that need auth (BOT Portal API) are gated by user-provided key.

3. Sources table auto-derived (not mock, not hardcoded)

• dashboard._build_sources → derive from FACTORS registry: iterate FACTORS, group by source module, emit {แหล่ง, ขอบเขต, data_value, as_of, fetched_at}.
• Frontend sources table consumes /api/v1/dashboard.sources — so adding a FACTOR auto-appends its source row. No hardcoded list in backend or frontend.

4. Real backtest engine (replaces forward-only allocation)

• New backend/app/backtest.py: takes {start_date, end_date, capital, bucket_weights, rebalance} and RUNS the allocation across time using the Yahoo price series:
  • at each rebalance date, recompute combined scores (theme+firm+quarterlies) on point-in-time basis (no future leak), allocate 50/20/30, track holdings.
  • mark-to-market daily; accrue dividends; end → report: {capital, invested, final_value, price_pnl, dividend_income, net_return, win_rate, trades}
• Backtest section in UI: pick start/end date, run, see performance summary.
• Persist run results (keep history) so refresh doesn't lose them.

5. Next-update column

• Each source row gains next_update_after (from frequency → next scheduled refresh time). Show "อัปเดตอีกครั้ง ~X" in the sources table.

Files

• docs/methodology-research.md (new, point 1)
• backend/app/factors.py (more factors; sources derive here)
• backend/app/dashboard.py (sources FROM registry + next_update)
• backend/app/backtest.py (new engine, point 4)
• backend/app/__init__.py (backtest endpoint, sources wiring)
• frontend/src/App.vue (backtest section + next-update col + sources from API)
• tests

Acceptance

1. Adding a FACTOR to factors.py auto-appends a sources-table row (no manual edit).
2. /api/v1/dashboard/sources has 5+ rows, all real, each with next_update_after.
3. Backtest with {start,end} returns a full P&L report (price + dividend + net), and results persist (not lost on refresh).
4. Literature doc recommends ≥1 concrete formula refinement, cited.
5. npm build + full suite pass; browser shows backtest section + next-update column.

Open decisions for user

• D1: rebalancing frequency in backtest (monthly / quarterly / weekly)?
• D2: do BOT Portal API sources require your API key, or only auth-free ones for now?
• D3: backtest default range (last 12 months / last N quarters)?