diff --git a/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation-r2.md b/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation-r2.md new file mode 100644 index 0000000..3d23b82 --- /dev/null +++ b/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation-r2.md @@ -0,0 +1,150 @@ +# Multi-Theme + Simulation Implementation Plan — REVISION 2 + +> **For Hermes:** Execute task-by-task with `todo` tracking and per-task commits. TDD for all logic; visual verification for UI/CSS. + +**Goal:** Rebuild the single-theme tourism demo into a **multi-theme research platform** with (a) a combined dashboard (all themes + stock recommendation table), (b) per-theme detail pages, (c) a **capital allocation simulation** (same engine used for both backtest and forward test), and (d) an MT5 order interface (forward test sends dry-run orders; live gate kept off in this round). UI in **Thai** (technical terms stay English). Uses **real data + daily cache**, not mocks. + +**Themes (3 alternative factors):** +1. `tourism` — Tourism Pulse (signal exists, real BOT data path) +2. `auto_credit` — Auto Credit Cycle (registry slot; data pipeline needs a real source before it can score) +3. `refining_energy` — Refining / Energy Spread (registry slot; data pipeline TBD) + +**Siamchart** is treated as a **fundamental data provider**, not a theme. It supplies PE/EPS/Yield/P/BV/ROE per symbol and contributes to the combined score with a **lower weight**. + +--- + +## Confirmed requirements (from user, 2026-08-25) + +1. **3 themes = 3 alternative factors** (tourism, auto_credit, refining_energy). A symbol can appear in multiple themes. +2. **Score combining:** each symbol has a *theme score* (average across all themes that score it) and a *siamchart score*. Combined = **theme_score × 60% + siamchart_score × 40%**. +3. **8-symbol live data** is OK for testing logic; production must cover the full universe (collectors must be scalable, not hard-coded to 8). +4. **"ไม่จ่ายปันผล" (bucket 2)** = a high-score stock that does **not** pay a dividend (is_dividend = False / yield ≤ 0). +5. **Paper ledger is replaced** by a forward test that reuses the **same simulation engine as backtest** — the only difference is real MT5 order dispatch. We build the MT5 order interface; live dispatch remains gated off (forward/dry-run in this round). +6. **Data quality / provenance = real sources, cached daily** — not fixture mocks. Provenance shows where each alternative factor comes from. +7. **Signal board + Stock board = ONE table**, signal-focused. +8. **UI language = Thai** (technical terms English). +9. **Remove unused/unnecessary dashboard sections.** + +--- + +# PART A — Multi-Theme Backend + +## Task A1: Theme registry +**Files:** create `backend/app/themes.py`; create `backend/tests/test_themes.py` + +Registry with 3 theme slots + their factor provider. `list_themes()` returns them. Registry is pluggable so a theme can be `enabled=False` while its data pipeline is absent. + +## Task A2: Combined signal aggregation +**Files:** modify `backend/app/themes.py`, `backend/app/__init__.py`; tests + +`/api/v1/dashboard` returns: +- Per-symbol `theme_score` = mean of enabled themes that score it +- `siamchart_score` (normalized) +- `combined_score = 0.6 * theme_score + 0.4 * siamchart_score` +- `side` (LONG/SHORT/NEUTRAL) from combined +- per-theme contributions + +Normalization: each provider's raw score is z-scored (or min-max over the live universe) before combining, so weights are meaningful across heterogeneous scales. + +## Task A3: Per-theme detail endpoint +`/api/v1/themes/` — 404 unknown; returns that theme's observations/surprise/signals/lineage. Also exposes `/api/v1/themes` (registry incl. enabled status + data source). + +## Task A4: Daily cache layer (real data, no mocks) +**Files:** create `backend/app/daily_cache.py`; tests + +Fetch-once-then-cache daily: +- Keyed by `(source, as_of)`. +- TTL ~24h; stale-if-error fallback to last good cache. +- Ensures tourism + siamchart reads hit cache instead of re-fetching every dashboard load, and sets up the "update daily" cadence the user wants. + +--- + +# PART B — Simulation / Backtest / Forward-Test Engine + +## Task B1: Price-series loader +**Files:** create `backend/app/simulation.py`; tests + +Load Yahoo price snapshot `series[SYM].bars` → `{date: adjusted_close}` for all symbols. Covers 2024-01-03 → 2026-08-24. + +## Task B2: Capital allocation core (`allocate_capital`) +**Files:** `backend/app/simulation.py` + tests + +Given `capital`, per-symbol `combined_score`, `is_dividend`, `dividend_yield`, latest `price`: +- **Bucket 1 (50%)** — highest combined_score among **dividend-paying** names. +- **Bucket 2 (20%)** — highest combined_score among **non-dividend** names. +- **Bucket 3 (30%)** — highest **dividend yield** among names **not already bought in bucket 1** (ignores score). + +Per symbol: **minimum 100 shares**. Rank by combined_score desc. If a bucket's first pick can't afford 100 shares, try progressively cheaper eligible names (that still pass the bucket's criteria). If no eligible name fits, leave remainder as cash. +- Allocation math: `qty = max(100, floor(available_cash / price / 100) * 100)` within bucket cash. +- Overflow flows bucket → bucket → cash. + +## Task B3: Simulation endpoint + forward-test mode +**Files:** `backend/app/__init__.py`; tests + +- `POST /api/v1/simulation` `{capital}` → allocation plan (per-bucket orders, qty, notional, total, unallocated cash, data caveat: revised history not PIT). +- `POST /api/v1/simulation/forward` — same engine, but marks orders `execution=paper` and, if an MT5 bridge is configured, would dispatch. **Live dispatch gated off** (env `MT5_ENABLE_ORDER` must be set AND explicit approval). + +## Task B4: MT5 order interface (interface only, live gated) +**Files:** create `backend/app/mt5_bridge.py`; tests (mock `MetaTrader5` per `mt5-data-service` skill gotcha) + +Abstract `MT5Bridge` with: `connect`, `symbol_resolve`, `order_new` (dry-run default), and a structural kill switch — no real dispatch path in code unless `MT5_ENABLE_ORDER=1` AND explicit user approval. Follows the `mt5-data-service` / `mql5-ea-development` skill patterns (Windows-only `MetaTrader5` stub for tests). + +--- + +# PART C — Frontend Multi-Theme UI (Thai, signal-first) + +## Task C1: Route shell +**Files:** `frontend/src/App.vue`, `style.css` + +Hash-routing: `#/` (dashboard), `#/theme/:id`, `#/simulation`. Sidebar nav in Thai: แดชบอร์ด, ธีม, การจำลอง (Simulation). + +## Task C2: Combined Dashboard (Thai) +- Theme strip: each active theme's surprise/label. +- **ONE stock table** (signal-first: combined score, side, theme contributions, then fundamentals PE/EPS/Yield/P/BV/ROE). Dividend filter + sortable columns kept. This **replaces** the separate Signal board + Stock board. +- Remove unused sections (Paper Ledger card, live receiver text, etc.). + +## Task C3: Theme detail page +`#/theme/:id` — that theme's observations (surprise bars), provenance (source used), and its signal contribution table. + +## Task C4: Simulation tab +`#/simulation` — Thai UI: enter capital, choose backtest/forward, render 3 buckets with orders; show unallocated cash and the "revised history, paper-only" caveat. + +## Task C5: Provenance (real data) +Dashboard provenance section reads from live cache metadata: source id, retrieved_at (daily cache), raw hash. No fixture labels. + +--- + +# Data requirements / open items (NOT guessed) + +- **auto_credit data source**: needs a real public/paid source for vehicle sales, auto loan growth, auto NPL. Not yet identified. `auto_credit` theme stays in registry as `enabled=False` until a source + collector exist. +- **refining_energy data source**: crack/GRM spread or Thai refining/energy stats. Not yet identified. `enabled=False` until built. +- **Full SET50 universe prices**: Yahoo history currently covers 8 symbols; production simulation needs price history for the full universe. Requires extending the price collector to all symbols (scrape/SETSMART/MT5). + +The three-theme registry is the *interface*; this round builds tourism + siamchart fully (real, cached) and stubs auto/energy until their data exists. Simulation works on whatever real symbols have both a price series and a score. + +--- + +# Verification + +1. Full backend suite (`-W error unittest discover`) all pass (existing 140 + new). +2. `cd frontend && npm run build` success. +3. Browser (Thai): dashboard lists themes (enabled + disabled), combined signal-first table with dividend filter + sort, simulation returns allocation, per-theme detail works. +4. `git diff --check`, static scan clean, independent fail-closed code review. +5. Per-task commits. + +--- + +# Closed/open per user's 10 points + +| # | User requirement | Status in plan | +|---|---|---| +| 1 | 3 themes (tourism, auto, energy) | Registry + tourism live; auto/energy `enabled=False` pending data source | +| 2 | combined = 60% theme + 40% siamchart | A2 | +| 3 | 8-symbol OK for logic, prod full | Simulation scales to any symbol with price+score | +| 4 | bucket2 = high score, no dividend | B2 | +| 5 | paper→forward test = same engine; MT5 iface | B3, B4 | +| 6 | real data + daily cache, provenance real | A4, C5 | +| 7 | one signal-first table | C2 | +| 8 | Thai UI | C1–C4 | +| 9 | remove unused | C2 | +| 10 | (mt5 scope) interface built, live gated | B4 | diff --git a/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation.md b/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation.md new file mode 100644 index 0000000..d92cacf --- /dev/null +++ b/.hermes/plans/2026-08-25_092044-multi-theme-dashboard-and-simulation.md @@ -0,0 +1,203 @@ +# Multi-Theme Dashboard + Capital Simulation Engine Implementation Plan + +> **For Hermes:** Execute task-by-task. Use `todo` progress tracking and per-task commits. TDD for logic, visual verification for UI/CSS. + +**Goal:** Turn the current single-theme (tourism) demo into a **multi-theme dashboard** (combined overview page + per-theme detail pages) plus a **separate Simulation tab** that runs capital allocation (50/20/30) ranked by signal score, using the existing Siamchart fundamental snapshot and the existing Yahoo price history (2024-01-03 → 2026-08-24). + +**Architecture:** A theme registry that each alternative factor registers into (tourism, siamchart-fundamental). The dashboard combines all themes' signals into a recommendation; each theme has its own detail page. The simulation engine consumes the combined signal scores + 2.5yr price history + dividend yields to allocate a user-entered capital across three buckets (min 100 shares/share). + +**Tech Stack:** Existing Flask backend (`backend/app/`), Vite+Vue frontend (`frontend/src/`). New modules `backend/app/themes.py`, `backend/app/simulation.py`. New route `/api/v1/simulation`. Frontend route-based dashboard (`#/`, `#/theme/:id`, `#/simulation`). + +--- + +## Current-state findings (from exploration) + +- **Themes today:** `tourism` signal (`backend/app/tourism.py::compute_tourism_signal` → `{theme, theme_surprise, observations, signals, as_of}`) + `siamchart` factor view (`backend/app/siamchart_factors.py::build_factor_view` → per-symbol PE/EPS/Yield/P/BV/ROE/is_dividend). +- **`/api/v1/factors`** already merges Siamchart fundamentals with tourism signal per symbol (added in `commit 8a6991b`). +- **Price history EXISTS**: `backend/data/prices/snapshots/prices-yahoo-chart-2024-01-01-2026-08-24-a1cf5b3c2ecf.json` — `{benchmark_symbol: "SET50", series: {SYM: {bars: [{date, open, high, low, close, adjusted_close, volume}]}}}` covering 2024-01-03 → 2026-08-24 for 8 symbols (AOT, AWC, BEM, CPALL, CPN, CRC, MINT, PTT) + index_SET. ~1.3 MB per vintage. +- **Fundamental snapshot**: `backend/data/siamchart/set50_master.json` — per-symbol EPS/PE/Yield%/is_dividend (49 symbols). +- **Paper ledger** (`backend/app/paper.py`) is a manual record-a-fill tool — NOT the automated allocation we now build. + +--- + +# PART A — Multi-Theme Backend Architecture + +## Task A1: Create a theme registry + +**Objective:** A single source enumerating all available themes and their factor/signal providers. + +**Files:** +- Create: `backend/app/themes.py` + +**Step 1: Write test** → `backend/tests/test_themes.py` +```python +def test_registry_lists_themes(): + from app.themes import list_themes + themes = list_themes() + assert {"id": "tourism"} in themes + assert {"id": "siamchart"} in themes +``` +Run: `PYTHONPATH=backend .venv/bin/python -W error -m unittest tests.test_themes -v` +Expected: FAIL (module missing). + +**Step 2: Implement `themes.py`** +```python +THEMES = [ + {"id": "tourism", "label": "Tourism Pulse", "kind": "signal", + "source": "bot_tourism", "provider": "compute_tourism_signal"}, + {"id": "siamchart", "label": "Siamchart Fundamentals", "kind": "fundamental", + "source": "siamchart", "provider": "siamchart_factors"}, +] + +def list_themes(): + return THEMES +``` +**Step 3: rerun test** → PASS. + +**Step 4: Commit:** `git add backend/app/themes.py backend/tests/test_themes.py && git commit -m "feat(backend): theme registry"` + +--- + +## Task A2: Combined signal aggregation endpoint + +**Objective:** `/api/v1/dashboard` returns an aggregate across all themes: per-symbol combined score (sum across themes that have a signal), side, the themes that fired, plus carousel info per theme. + +**Files:** +- Modify: `backend/app/__init__.py` +- Create: `backend/tests/test_dashboard.py` + +**Step 1: Test** — combined aggregation with a mock theme. +**Step 2: Implement** an aggregate function in `themes.py` that consumes each theme's signals (tourism signals + siamchart scores) and combines per symbol. +**Step 3: PASS + commit** `feat(backend): aggregated multi-theme dashboard endpoint` + +--- + +## Task A3: Per-theme detail endpoint + +**Objective:** `/api/v1/themes/` returns that theme's observations/surprise/signals for the detail page. + +**Files:** +- Modify: `backend/app/__init__.py` +- Create: tests in `test_dashboard.py` + +**Step:** route dispatch to the theme's provider; validate `theme_id` against registry; 404 unknown. PASS + commit. + +--- + +# PART B — Simulation / Capital Allocation Engine + +## Task B1: Price-history loader + +**Objective:** Load the latest Yahoo price snapshot series for a symbol as date→adjusted_close. + +**Files:** +- Create: `backend/app/simulation.py` +- Create: `backend/tests/test_simulation.py` + +**Step 1: Test** +```python +def test_load_price_series(tmp_path): + snap = {"benchmark_symbol":"SET50","series":{"PTT":{"bars":[ + {"date":"2024-01-03","adjusted_close":50.0}, + {"date":"2024-01-04","adjusted_close":51.0}]}}} + # write temp snapshot, call loader + series = load_price_series(snap, "PTT") + assert series == {"2024-01-03":50.0,"2024-01-04":51.0} +``` +**Step 2: Implement** `load_price_series(snapshot, symbol)` → `dict[date, adjusted_close]`; reject unknown symbol with `SimulationError`. PASS + commit `feat(backend): price series loader`. + +--- + +## Task B2: Capital allocation core + +**Objective:** Given capital, per-symbol signal score + is_dividend + dividend yield, allocate across three buckets: +- **Bucket 1 (50%)**: highest signal score among **dividend-paying** names +- **Bucket 2 (20%)**: highest signal score among **non-dividend** names (combine with leftover/overflow from bucket 1 shortfall) +- **Bucket 3 (30%)**: highest dividend yield among names **not** already in bucket 1 (ignore profit) + +Each bucket buys from its ranked list, **minimum 100 shares per symbol**, at latest price; leftover cash (from price-bucket rounding) flows to next bucket / stays unallocated. + +**Files:** `backend/app/simulation.py`, `backend/tests/test_simulation.py` + +**Step 1: Test** `test_allocate_buckets_basic` — small capital, verify 50/20/30 money split, min-100-share floor, ranking by score. +**Step 2: Test edge:** insufficient capital for min 100 shares in a bucket → skip to next; no infinite loop. +**Step 3: Implement** `allocate_capital(capital, factors, prices)` returning per-bucket orders + unallocated cash. +**Step 4: PASS + commit** `feat(backend): capital allocation engine (50/20/30)` + +--- + +## Task B3: Simulation endpoint + +**Objective:** `POST /api/v1/simulation` takes `{capital}` and returns the allocation plan (per bucket, per-symbol qty/price/notional, total invested, unallocated). + +**Files:** `backend/app/__init__.py`, tests. + +**Step:** wire `allocate_capital` to the endpoint; validate capital>0 finite; 400 on bad input. PASS + commit `feat(backend): simulation endpoint`. + +--- + +# PART C — Frontend Multi-Theme UI + +## Task C1: Route-based shell (Dashboard + theme detail + simulation) + +**Objective:** `App.vue` becomes route-aware (hash routing): `#/` dashboard, `#/theme/:id`, `#/simulation`. Sidebar nav splits into these. + +**Files:** `frontend/src/App.vue`, `frontend/src/style.css` +- Keep `Stock board`/`Signal board` semantics but restructure nav: + - **Dashboard** (`#/`) — combined overview + stock recommendation + - **Simulation** (`#/simulation`) — separate tab + - **per-theme** (`#/theme/:id`) — theme detail +- TDD skipped (visual); verify via dev server + browser. + +**Verify:** `npm run build` passes; open dev server; nav switches views. + +## Task C2: Combined Dashboard page + +**Objective:** Overview aggregating all themes: theme surprise cards, combined stock recommendation table (with dividend filter + sortable columns — reuse existing logic), theme list with links. + +**Files:** `frontend/src/App.vue` +- Reuse `sortedFactorRows`/`dividendOnly`/`setSort` from current stock board. +- Add theme-surprise strip + "combined" context. + +## Task C3: Theme detail page + +**Objective:** `#/theme/:id` shows that theme's observations (surprise bars), source lineage (provenance), and its own signal table. + +**Files:** `frontend/src/App.vue` +- Fetch `/api/v1/themes/:id`. + +## Task C4: Simulation tab UI + +**Objective:** `#/simulation` — input capital, POST `/api/v1/simulation`, render 3 buckets (each with the buy list, qty, notional; min-100 clarity), unallocated cash, and a note on the data limits (revised history, not PIT). + +**Files:** `frontend/src/App.vue`, `style.css` +- Verify via browser after wiring. + +--- + +# Migration pitfalls this plan must respect (from `plan` skill) + +- **Multi-theme wiring**: do not hardcode "tourism" prose; pull labels from `/api/v1/themes`. +- **Simulation honesty**: clearly label output as a *paper/backtest* on **revised vendor history** (`point_in_time=false`) — never a validated PIT claim. +- **Bucket edge cases**: min-100-share floor can leave a bucket with zero capital allocated → must flow to next bucket gracefully (tested). +- **XSS**: all UI via Vue `{{ }}`, no `v-html`. +- **Per-theme 404**: unknown theme_id must 404, not silently render empty. + +--- + +# Verification (final) + +1. `PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests` → all OK (existing 140 + new). +2. `cd frontend && npm run build` → success. +3. Browser: dashboard lists themes; `#/simulation` with capital input returns an allocation table; dividend filter + sort still work. +4. `git diff --check`, static scan (no secrets/eval/exec/pickle). +5. Independent code review (fail-closed JSON) before commit + push. + +--- + +# Open questions / assumptions (call out if wrong) + +- **"Combined signal score"**: sum of normalized scores across themes that provide a signal for a symbol. If only one theme fires, its score is the combined score. (Assumed; the user said "signal score from factor combined".) +- **Dividend filter for bucket 1 vs bucket 3**: bucket 1 = dividend-paying AND high score; bucket 3 = high dividend yield regardless of score, **excluding bucket-1 names**. User: "จ่ายเงินปันผลมากที่สุด (ไม่ซ้ำกับข้อ 1 และไม่สนเรื่องโอกาสทำกำไร)". Implemented as exclusion, not a forced different-name set. +- **Prices**: only 8 symbols have Yahoo history (AOT, AWC, BEM, CPALL, CPN, CRC, MINT, PTT) — the simulation can only allocate among symbols with both a price series AND a factor score. Symbols without a price series are skipped with a warning. +- **Min 100 shares** — quantity may exceed cash in bucket for high-priced names; we buy floor(available_cash / price / 100) * 100 shares, leftover flows onward. diff --git a/.hermes/plans/2026-08-25_dashboard-polish-table-theme-symbol-detail.md b/.hermes/plans/2026-08-25_dashboard-polish-table-theme-symbol-detail.md new file mode 100644 index 0000000..c531be4 --- /dev/null +++ b/.hermes/plans/2026-08-25_dashboard-polish-table-theme-symbol-detail.md @@ -0,0 +1,68 @@ +# Dashboard Polish — Table UX, Per-Theme Narrative, Per-Symbol Analysis Page + +> **For Hermes:** This plan covers the 7-point UX + analysis-transparency review. No code until user approves. + +**Goal:** Make the stock table informative & sortable, explain each theme's analysis in narrative (not just a number), and give every symbol a dedicated detail view showing HOW the system analyzed it (weights, components, combined-score derivation) so the user can verify the analysis honours the agreed methodology. + +--- + +## The 7 complaints → fix + +| # | Complaint | Fix | Effort | +|---|-----------|-----|--------| +| 1 | Table header text เบียด/ไม่เต็มพื้นที่ | Full-width header, better line-height/whitespace | CSS | +| 2 | "รวม" column ไม่ชัดว่าคืออะไร + เรียงไม่ได้ | Rename → **"คะแนนรวม"**; make it sortable; add tooltip "60% ธีม + 40% พื้นฐาน" | FE + sort | +| 3 | ธีม กับ 05/บันทึกการวิเคราะห์ ซ้ำ | **Remove 05/บันทึกการวิเคราะห์** (theme panel already has thesis) | FE remove | +| 4 | section-kicker ต้อง format เดียวกัน ("07 /" ออก) | Remove numeric prefixes → plain Thai kickers | FE | +| 5 | Theme ต้องการคำอธิบาย/บรรยายผลวิเคราะห์ | Long-form narrative per theme (auto hon 3) | BE + FE | +| 6 | ตารางหุ้น ต้องมี column theme | Add "ธีม" column showing which theme(s) each symbol belongs to | FE | +| 7 | แต่ละหุ้นต้องมี หน้า detail: คลิก แล้วเห็นการวิเคราะห์ทุกอย่าง ต่อหุ้น (ข้อมูล→น้ำหนัก→ที่มาคะแนนรวม) | New per-symbol endpoint + detail view (modal or page) | BE + FE | + +--- + +## Backend tasks + +### Task B7: Per-symbol analysis breakdown endpoint +- `GET /api/v1/symbols/` returns for one symbol: + - `symbol`, `company_name` + - `themes`: which theme(s) this symbol belongs to (from THEME_SYMBOLS) + each theme's factor reading + surprise contribution + - `fundamentals`: PE, EPS, EPS YoY, dividend_yield, PBV, ROE (Siamchart) + - `score_breakdown`: + - `theme_score` = mean of the theme surprise scores covering this symbol (with per-theme lines) + - `siamchart_score` = growth + yield*2, z-scored + - `combined` = 0.6*theme_score + 0.4*siamchart_score (with the actual numbers, not just final) + - `price`: latest close + date +- Deterministic, reuses `themes.combine_score` logic per-symbol; NO new magic. + +### Task B5: Long-form per-theme narrative +- Add a `narrative` field to each theme read: 2-4 sentence Thai explanation of what the factor means, whether it's positive/negative, and what it implies for stocks in that group. Built deterministically from the factor reading + surprise sign (not hardcoded, not an LLM). + +## Frontend tasks + +### Task F1: Table header full-width (complaint 1) +- Give `.stock-panel`/`.factor-table` a max-width-safe full-width layout, better padding/line-height so the subtitle + controls breathe. + +### Task F2: Rename + sort "รวม" (complaint 2) +- Header "รวม" → **"คะแนนรวม"** with a small `60/40` hint; add `combined` to the sortable keys (already have `setSort('combined')` — wire it to actually sort by combined score desc default). + +### Task F3: Remove 05/บันทึกการวิเคราะห์ (complaint 3) +- Delete the bottom-grid thesis section (superseded by theme panel narratives). + +### Task F4: Uniform section-kicker (complaint 4) +- Strip "07 /", "05 /" etc. → plain Thai kickers ("การจำลองการลงทุน", "ธีม", "ที่มาของข้อมูล", "ตารางหุ้น"). + +### Task F6: Add Theme column (complaint 6) +- In stock table add a "ธีม" column: for each symbol show compact chips of the theme(s) it belongs to (e.g. "โรงกลั่น", "—" if none). Reuse `boardBySymbol` / THEME lookup. + +### Task F7: Per-symbol detail view (complaint 7) +- Clicking a symbol row opens a detail panel/modal showing the full `/api/v1/symbols/` breakdown: themes + fundamentals + score components + how combined was derived (60/40). This is the transparent-analysis view. + +--- + +## Verification +- Backend: full suite passes; `GET /api/v1/symbols/AOT` returns full breakdown with correct combined derivation. +- Frontend: `npm run build`; browser shows full-width table, "คะแนนรวม" sortable, theme column, no 05 section, uniform kickers, per-symbol modal works & shows derivation. +- Screenshot evidence; push after user OK. + +## Open questions +None blocking — methodology is deterministic and already implemented in `themes.combine_score`; this plan surfaces it transparently rather than changing it. diff --git a/.hermes/plans/2026-08-25_dashboard-rebuild-multitheme-real-data.md b/.hermes/plans/2026-08-25_dashboard-rebuild-multitheme-real-data.md new file mode 100644 index 0000000..723e55b --- /dev/null +++ b/.hermes/plans/2026-08-25_dashboard-rebuild-multitheme-real-data.md @@ -0,0 +1,99 @@ +# Dashboard Rebuild — Multi-theme + Real Thai Data + +> **For Hermes:** Execute task-by-task, commit per task, verify with backend tests + browser. Plan mode — no code until user approves. + +**Goal:** Rebuild the SET50 dashboard so it reflects REAL multi-theme Thai data (tourism + auto + energy + NPL), not the current tourism-only fixture, per the user's 10-point review. + +**Architecture:** Replace the single-vertical `tourism_result`-driven dashboard with a multi-theme real-data source; add per-symbol theme exposure (multi-source factors PER theme); update all UI sections to read real data, Thai-only, and remove the 5 obsolete KPI cards + paper ledger. + +--- + +## The 10 user complaints → fix + +| # | Complaint | Root cause | Fix | +|---|-----------|-----------|-----| +| 1 | Theme Surprise ไม่เข้าใจ, ลบได้ | KPI card from tourism-only | **Remove** card | +| 2 | สัญญาณที่ใช้งาน: ซื้อ 7 แต่โชว์ 8 | total=8 (incl neutral) mislabeled | Show long/short/neutral correctly, Thai | +| 3 | สมุดบันทึก paper มีทำไม — ไม่มี paper trade แล้ว | legacy section | **Remove** (simulation replaces it) | +| 4 | คุณภาพข้อมูล — ข้อมูลจริงรึยัง? | `TOURISM_SOURCE=fixture` default | **Switch to real BOT data**; show real source+as_of | +| 5 | ประตู backtest (blocked) เอาออก | legacy gate | **Remove** card (research is exploratory/forward) | +| 6 | แต่ละธีมควรดูหลายแหล่ง (ประชากร ฯลฯ) | 1 source/theme | **Add multi-source factors per theme** (see Data plan) | +| 7 | ตารางหุ้น + ตารางสัญญาณ ควรรวมกัน | split | **Merge** into one combined table | +| 8 | Theme surprise ไม่ครบ 3 ธีม | tourism-only | **3 theme surprise** (auto, energy, tourism) | +| 9 | ที่มาข้อมูล ควรเป็นตาราง (แหล่ง+เวลา) | prose | **Table** of all sources + last fetched | +| 10 | บันทึกการวิเคราะห์ มีแค่ 1 ธีม | tourism thesis | **Per-theme thesis** (3) + multi-source note | + +**Overall:** "ข้อมูลทั้งหมดตอนนี้ยังไม่ได้ใช้ข้อมูลจริงใช่ไหม?" → ใช่, dashboard ใช้ `tourism_result` (fixture default). Rebuild to real. + +--- + +## User-confirmed decisions (2026-08-25) +1. **Real data ONLY** — switch to BOT real (no fixture fallback); dashboard fails if no real data. +2. **Theme surprise = z-score** of each theme's headline factor (uniform across all themes). +3. **Multi-source per theme (A+B)**: wire existing multi-source (auto = volume + NPL + production + export); AND add broader macro (population/GDP/inflation) — **found BOT SDDS page (bot.or.th/en/statistics/sdds.html) is server-rendered and scrapable WITHOUT auth**: provides GDP (Q2/2026 4,793.5bn), Private Consumption +4.9%, Private Investment +18.1%, Manufacturing −3.1%, Population 70,472k, Headline Inflation 1.95%, Unemployment 0.93%. This is the macro backdrop layer (complaint #6). + +## Macro backdrop factor (new) — BOT SDDS collector +- Create `backend/app/macro_thai.py` — scrape BOT SDDS HTML for: private consumption index %YoY, private investment %YoY, mfg production %YoY, population (thousands), headline inflation, core inflation, unemployment. Real Thai macro backdrop. +- Wire into a "macro" theme/context read + into the sources table. + +## Backend task list + +### Task A: Real multi-theme dashboard source module +- Create `backend/app/dashboard.py` — assembles REAL data from all collectors: + - tourism (BOT real, via collector), auto_credit (Trading Econ + **auto_npl** BOT), energy (TOP), siamchart fundamentals, prices + - Returns `{themes: [{id, name, frequency, read_* , surprise}], board: [{symbol, combined_score, dividend, ...}], sources: [{from, source, as_of, fetched_at}], thesis_per_theme: {...}}` +- **Multi-source per theme (complaint #6):** registry mapping theme → list of factor sources (not 1). E.g. auto theme = {new_car_sales_yoy (TradingEcon), auto_npl_pct (BOT), vehicle_production, auto_exports}. Add aggregatable fields. +- **3 theme surprise (#8):** compute surprise per theme (z-score of the theme's headline factor), not just tourism. + +### Task B: Switch create_app to real data +- `backend/app/__init__.py`: default `TOURISM_SOURCE` → `bot` (real), or better: build `multi_dashboard` from collectors at startup, keep tourism as one theme. `_load_default_snapshot` no longer the dashboard root. +- `dashboard/summary`, `/signals`, `/factors` endpoints → read from `dashboard.py` real assembly, not `tourism_result`. + +### Task C: Sources table endpoint +- `dashboard.py` collects `sources` list: each {from (e.g. "TradingEconomics"), source, url, as_of, fetched_at, frequency}. Expose in `/api/v1/themes` + summary. + +### Task D: Per-theme thesis (#10) +- `dashboard.py` builds a short deterministic thesis per theme from that theme's factor readings (not a hardcoded tourism sentence). + +## Frontend task list + +### Task F: Remove 5 obsolete items (#1,#3,#5) +- Delete cards: Theme Surprise (#1), คุณภาพข้อมูล (#4 replaced by honest sources), ประตู backtest (#5), สมุดบันทึก paper (#3) → card 2 (สัญญาณ) only + 3-theme panel. +- Remove paper-ledger UI + backtest-gate UI blocks (already partially removed paper ledger; remove backtest KPI). + +### Task G: Correct signal count (#2) +- Show `long` / `short` / `neutral` as separate Thai chips; value = long only, with "+X ซื้อ −Y ขาย" clearly, no off-by-one. + +### Task H: Merge stock + signal tables (#7) +- Single combined table: [สีสัญญาณ, symbol, theme(s), combined_score, dividend, PE, EPS YoY, Yield, P/BV, ROE] — one table, sortable, dividend filter. + +### Task I: Sources table UI (#9) +- Render `sources` as a ``: ตัวจาก, แหล่ง, ข้อมูล, as_of, fetched_at, frequency. + +### Task J: 3-theme surprise + thesis (#8,#10) +- Theme panel shows 3 themes, each with its own surprise + read + thesis. + +### Task K: Thai-only consistency + visual verify +- Audit all remaining English; keep only technical terms. Browser verify at 1440/1024/768. Screenshot for user. + +--- + +## Multi-source factor registry (complaint #6) — where to add data +For "แต่ละธีมดูข้อมูลรอบด้านของประเทศ" — extend themes to multiple Thai sources: +- **Auto**: new_car_sales_yoy (TradingEcon) + auto_npl_pct (BOT) + vehicle_production + auto_exports (TradingEcon) [already in collector] +- **Energy**: TOP net_profit/ebitda + Thai retail fuel/EPPO + (optional) import/refining +- **Tourism**: BOT tourism surprise + arrivals + hotel occupancy +- **Macro backdrop (new)**: Thai population / GDP (BOT API — needs auth, mark deferred) per user's "จำนวนประชากร" + +**Note:** Full multi-source (population/GDP via BOT portal API) requires API key registration — mark as a separate later task; this plan wires the already-available multi-source readings per theme. + +## Verification +- Backend: full suite passes; live `/api/v1/themes` shows 3 themes with real Thai readings + sources table + per-theme thesis; no fixture. +- Frontend: `npm run build`; browser shows combined single table, 3-theme surprise, sources table, no removed cards; Thai-only. +- Screenshot evidence for user review before push (push gate). + +## Open questions / risks +1. **Real-vs-fixture default:** Should dashboard run on BOT real data by default (needs a fetch), or keep a cache so it works offline? → Recommend: real BOT via daily cache, fallback to last-good. +2. **Population/GDP macro**: needs BOT portal API key (user registers). Defer unless user provides key. +3. **Theme surprise definition**: need concrete per-theme surprise metric — z-score on each theme's headline factor (auto YoY, energy net profit trend, tourism surprise). Confirm with user if ambiguous. +4. **Push gate**: after visual verify, user approves before push (per repo convention). diff --git a/.hermes/plans/2026-08-25_full-app-consistency-refactor.md b/.hermes/plans/2026-08-25_full-app-consistency-refactor.md new file mode 100644 index 0000000..0d1ec58 --- /dev/null +++ b/.hermes/plans/2026-08-25_full-app-consistency-refactor.md @@ -0,0 +1,95 @@ +# Full-App Consistency Refactor — single source of truth + per-symbol selection + +> **For Hermes:** Execute after user approval. Fixes two debts at once: +> (a) theme mapping/data duplicated & diverged across backend/frontend; +> (b) theme selection is FLAT — every symbol in a theme gets the same surprise +> score, so there's no real "เลือกหุ้น" (which name benefits most within theme). + +**Root problems:** +1. Theme registry + labels live in 3 places that diverge (`themes.py` correct, + `dashboard.py` maco-proxy, `__init__.py` old `/api/v1/themes` still 3 themes, + `App.vue` hardcoded `THEME_BY_SYMBOL`/`themeLabelById`). +2. **Selection is not real:** `_build_board` assigns the SAME `surprise` to every + symbol in a theme (flat). A strong BBL vs weak TTB get identical theme score. + The user wants each theme to *analyze and pick* — i.e. differentiate per-symbol + quality/benefit within the theme. + +--- + +## What "theme analyzes & selects stocks" means (per-theme pipeline) +For EVERY theme, a symbol in that theme gets: + theme_score_for_symbol = + theme_surprise (macro/theme tailwind, common to all in theme) + × firm_quality (per-symbol relative strength within the theme) + +`firm_quality` is deterministic, derived from the symbol's own fundamentals +relative to its THEME cohort (not the whole market), e.g.: + - banks: ROE, NIM proxy (ROE/EPS), loan book health (EPS growth) + - energy: PTT/TOP margins via ROE + yield + - retail: margin/EPS growth + - generic: ROE vs theme median, EPS growth vs theme median, dividend floor +This way within a hot theme the *better-run* names rank higher → real stock picking. + +We generalize the existing 60/40 so theme_score is no longer flat: + combined = 0.6 * (theme_surprise × quality) + 0.4 * siamchart_score + +--- + +## Refactor plan + +### Backend +**Task B0 — per-symbol theme quality (selection):** +- Add `themes.quality_within_theme(symbol, theme_id, factor_view) -> float` that + computes a symbol's relative quality within its theme cohort (ROE/EPS-growth vs + theme median, dividend floor). deterministic, unit-testable. +- In `dashboard._build_board`, when applying a theme surprise to a symbol, multiply + by `quality_within_theme` so symbols in the same theme differentiate. +- Expose `quality` + `theme_score breakdown` in `/api/v1/symbols/` so it's transparent. + +**Task B1 — single source of truth:** +- `themes.py` add `theme_of(symbol)`, `label_of(theme)`, `quality_within_theme`. +- `dashboard.build()` returns `symbol_theme_map` + board rows carry per-symbol `themes` + `theme_score`. + +**Task B3 — remove old `/api/v1/themes` (3 themes, dup logic):** +- Delegate to `RealDashboard.build()` so `/api/v1/themes` == `/api/v1/dashboard` theme set (13, same labels/surprises). Remove dup auto/energy sign code. + +**Task B4 — single fetch path:** +- RealDashboard.build() is the one place that fetches auto/energy/macro; endpoints + scheduler consume it (no dupe fetch). + +### Frontend +**Task F1 — delete hardcoded `THEME_BY_SYMBOL`/`themeLabelById`:** +- theme column + modal come from `/api/v1/dashboard` (board[].themes, themes[].label_th) + `/api/v1/symbols/` breakdown. No local map. + +**Task F2 — modal shows per-symbol selection logic:** +- Add to symbol view: the `quality` within each theme + how theme surprise × quality = theme score, so stock-picking is explainable. + +### Tests +- `test_themes`: quality_within_theme differentiates (strong vs weak name), unit. +- `test_dashboard`/`test_api`: `/api/v1/themes` == `/api/v1/dashboard` theme set; board rows have per-symbol themes; two symbols in same theme get different theme_score when quality differs. + +### Files +- `backend/app/themes.py`, `backend/app/dashboard.py`, `backend/app/__init__.py` +- `frontend/src/App.vue` +- `backend/tests/test_themes.py`, `test_dashboard.py`, `test_api.py` + +## Acceptance (definition of done) +1. Two banks (BBL strong, TTB weak) get DIFFERENT theme score within banks theme — shows real stock selection, not flat. +2. Editing themes.py propagates to `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols`, frontend — no frontend change needed. +3. `/api/v1/themes` == `/api/v1/dashboard` theme set (consistency test). +4. Modal explains: theme surprise × firm quality = theme score per symbol. +5. npm run build + full backend suite pass; browser shows BBL>TTB theme score when quality differs. + +### Files +- `backend/app/themes.py` (add theme_of/label_of helpers) +- `backend/app/dashboard.py` (symbol_theme_map in build; board row.themes) +- `backend/app/__init__.py` (themes endpoint → delegate; de-dup; board.themes) +- `frontend/src/App.vue` (remove hardcode maps; derive from API) +- `backend/tests/test_dashboard.py`, `backend/tests/test_api.py` (consistency tests) + +## Acceptance (definition of done) +1. Changing a theme in `themes.py` propagates to `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols/`, and the frontend column/modal — with NO change to frontend code. +2. No duplicate theme mapping in frontend. +3. `npm run build` + full backend suite pass; browser shows BBL=ธนาคาร from API. + +## Open question / tradeoff +- One cache/fetch per data refresh vs per-day: keep single `RealDashboard.build()` as the one fetch path (scheduler + endpoints both call it) so data is consistent within a cycle. Accept minor latency first call. diff --git a/.hermes/plans/2026-08-26_factor-engine-architecture.md b/.hermes/plans/2026-08-26_factor-engine-architecture.md new file mode 100644 index 0000000..a49b2ad --- /dev/null +++ b/.hermes/plans/2026-08-26_factor-engine-architecture.md @@ -0,0 +1,148 @@ +# Architecture Plan — Declarative Factor/Theme Analysis Engine + +> Status: DRAFT — pending user approval (decision gate G0). +> Author: Macky. Date: 2026-08-26. + +## Goal +Rebuild the scoring system so that **adding/removing a data source or factor is a config change, not an edit to scoring functions**. Every theme gets a full analyze-and-select pipeline built from registered, declarative factors. This makes the platform easy to extend as the user keeps finding new Thai data sources. + +## Why this is needed (current debt) +Today the analysis is **hardcoded**: +- `dashboard._theme_surprises()` — per-theme surprise logic written inline (tourism special, auto NPL composite, energy TOP, then macro-proxy branches). +- `dashboard._theme_narrative()` — per-theme narrative strings hardcoded. +- `dashboard.build()` — proxy_reads dict hardcoded. +- `App.vue` — `THEME_BY_SYMBOL` + `themeLabelById` hardcoded (duplicate of backend). +- `__init__.py` — old `/api/v1/themes` still builds only 3 themes. + +Every new source = editing functions and strings in 3+ places → exactly the "จุดนึงเสร็จ จุดอื่นไม่ตาม" problem. + +## Target Architecture + +``` + ┌──────────────────────────────┐ + │ FACTOR REGISTRY (config) │ + ┌──────────────────▶ (declarative, data-driven) │ + │ └──────────────────────────────┘ + │ reads reads + ▼ ▼ +┌───────────────┐ fetch/parse ┌──────────────────────┐ +│ COLLECTORS │─────────────▶ │ FACTOR PIPELINE │ +│ (per source) │ raw observable │ surprise / z-score / │ +└───────────────┘ │ frequency alignment │ + ▲ └──────────────────────┘ + │ registry.fetcher │ per-factor value + └──┘ ▼ + ┌──────────────────────┐ + │ THEME SCORER │ + │ factors ✕ weights │ + │ ✕ firm_quality │ + └──────────────────────┘ + │ theme_score + ▼ + ┌──────────────────────┐ + │ COMBINE (60/40) + │ + │ SELECT / board │ + └──────────────────────┘ + │ + ▼ + /api/v1/themes, /api/v1/dashboard, + /api/v1/symbols/, frontend (read-only) +``` + +## Key Decisions +| Decision | Choice | Rationale | Alternatives | +|---|---|---|---| +| Factor = declarative unit | `FACTORS` registry: dict of `{key, source, fetch_fn, frequency, sign, weight, periods}` | add source = add one dict entry | hardcoded functions (rejected) | +| Theme = composition of factors | `THEMES` registry: `{id, label_th, symbol_map, factors:[{factor_key, weight}]}` | theme reuses factors; new theme = add entry | inline per-theme logic (rejected) | +| Source of truth | `themes.py`/`factors.py` registries (backend) | one place; frontend reads via API | App.vue hardcode (remove) | +| Per-symbol selection | `quality_within_theme(symbol, theme, factor_view)` | differentiates strong vs weak within a theme | flat surprise (rejected) | +| Frequency alignment | each factor carries `frequency`; scorer aligns by period (as-of semantics), never mixes timestamps | user's explicit requirement | naive mixing (rejected) | + +## Decision Gates (one per milestone → user approves before code) + +### G0 — Approve declarative registry design +**Acceptance:** user approves FACTORS + THEMES registry schema & that adding a source = one config entry. +**Exit:** proceed to B0. + +### G1 — Per-symbol selection works +**Acceptance:** two banks with different fundamentals get **different** theme_score within banks theme (BBL strong > TTB weak), unit-tested. +**Exit:** proceed to frontend. + +### G2 — Frontend no longer hardcodes themes +**Acceptance:** edit `themes.py` (add theme/factor) → `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols/`, and the board column/modal all pick it up with zero frontend change. +**Exit:** done. + +## Data Model +**FACTORS** (in `backend/app/factors.py`): +```python +FACTORS: dict[str, dict] = { + "tourism_arrivals_yoy": { + "name_th": "นักท่องเที่ยว", + "source": "BOT", "frequency": "monthly", + "fetch": "bot_tourism", # module name; returns dict with numeric value + "value_key": "arrivals_yoy", # key of the numeric value + "sign": 1, # +1 = higher is bullish for theme + "weight": 1.0, # (themes can override) + }, + "auto_sales_yoy": {"source":"TradingEconomics","frequency":"monthly","fetch":"auto_credit","value_key":"new_car_sales_yoy","sign":1}, + "auto_npl": {"source":"BOT","frequency":"quarterly","fetch":"auto_npl","value_key":"pct_of_npls","sign":-1}, + "energy_top_netmarg": {"source":"TOP","frequency":"quarterly","fetch":"energy_thai","value_key":"net_margin","sign":1}, + "macro_consumption": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"private_consumption_yoy","sign":1}, + "macro_investment": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"private_investment_yoy","sign":1}, + "macro_inflation": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"headline_inflation_yoy","sign":-1}, + "macro_mfg": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"manufacturing_yoy","sign":1}, +} +``` + +**THEMES** (in `backend/app/themes.py` — keeps symbol_map + label): +```python +THEMES: dict[str, dict] = { + "auto_credit": { + "label_th": "รถยนต์/สินเชื่อ", "frequency": "monthly", + "factors": [{"key":"auto_sales_yoy","weight":1.0},{"key":"auto_npl","weight":-0.5}], + "symbols": {...}, + }, + "banks": { + "label_th": "ธนาคาร", "frequency": "quarterly", + "factors": [{"key":"macro_investment","weight":1.0},{"key":"macro_inflation","weight":-0.3}], + "symbols": {...}, + }, + ... +} +``` + +**surprise per theme** = weighted, sign-aware blend of its factors (each factor z-scored / normalized per its frequency cohort → frequency-aligned, no naive mixing). + +**theme_score per symbol** = theme_surprise × `quality_within_theme(symbol, theme, factor_view)`. + +**combined** = 0.6 × theme_score + 0.4 × siamchart_score. + +## Module Boundaries +``` +factors.py — FACTORS registry + normalize/z-score helper (single source) +themes.py — THEMES registry (symbol_map, labels, factor refs), quality_within_theme, theme_of, label_of, combine_score +dashboard.py — orchestrator: fetch registered factors once, compute surprises, build board/macro/sources (no per-theme hardcode) +collectors/ — bot_tourism, auto_credit, auto_npl, energy_thai, macro_thai (return dict, registered in FACTORS.fetch) +__init__.py — endpoints only; /api/v1/themes delegates to dashboard.build(); /api/v1/symbols uses themes helpers +App.vue — PURE read-only; theme column/labels/board from /api/v1/dashboard; NO hardcoded theme maps +``` + +## Explicit Non-Goals (v1) +- No live trading / MT5 execution (paper/backtest only, unchanged). +- No new collectors yet — this refactor *enables* easy addition; sources are added as follow-up (user will name them). +- No LLM involvement in scoring (deterministic only). +- No changing the 60/40 or 50/20/30 business rules — only making them data-driven. + +## Open Questions (for G0) +1. Should factor weights live per-theme (override) or global? → proposal: per-theme override with global default (flexible). +2. Do we keep the current 3 real collectors as the first factors registered (yes), and new sources added later by the user? + +--- + +## Acceptance (definition of done — all gates pass) +1. Adding a new factor = 1 dict entry in `FACTORS` + (optionally) a theme factor line — no scoring-function edit. +2. `/api/v1/themes` == `/api/v1/dashboard` theme set (13, same labels/surprises) — consistency test. +3. BBL vs TTB get different banks theme_score by quality — selection proof test. +4. Frontend has NO hardcoded theme maps; column + modal label from API. +5. Modal shows `theme surprise × firm quality = theme score` per symbol. +6. `npm run build` + full backend suite pass; browser BBL > TTB when quality differs. diff --git a/.hermes/plans/2026-08-26_research-sources-backtest.md b/.hermes/plans/2026-08-26_research-sources-backtest.md new file mode 100644 index 0000000..2013a3f --- /dev/null +++ b/.hermes/plans/2026-08-26_research-sources-backtest.md @@ -0,0 +1,88 @@ +# Research + Data-Source + Simulation Backtest — plan (5-point) + +> For Hermes: execute after user approval. This covers: +> 1. Literature review → formula improvements +> 2. New data sources per theme +> 3. Frontend sources table = NOT mock, auto from backend FACTORS registry +> 4. Simulation: real backtest engine w/ custom start/end date + P&L +> 5. Sources table shows next-update column + +## Findings (facts, verified) + +**Sources table is hardcoded (point 3 true):** `dashboard._build_sources` returns a +hardcoded 5-row list (TradingEconomics, BOT NPL, Thai Oil, BOT Economy, Tourism). +It does NOT come from the FACTORS registry, so adding a factor does NOT add a source +row automatically. Must wire sources from FACTORS registry (single source of truth). + +**Simulation is NOT a real backtest (point 4 true):** `simulation.allocate_capital` +allocates once at latest price. Both "backtest" and "forward" modes just differ in +whether an MT5 order is sent. There is no start/end date, no rebalancing over time, +no performance/P&L, and results are not persisted (vanish on refresh). This matches +the user's confusion exactly — the two modes don't do what the labels imply. + +**5 data sources today** (real): TradingEconomics (auto sales) · BOT NPL · Thai Oil +(TOP quarterly) · BOT Thai Economy (macro) · BOT Tourism — all real, but hardcoded +in the sources table. + +--- + +## Work items + +### 1. Literature / methodology research → formula improvements +- Research: factor-based Thai-equity scoring, alternative-data alpha, market-regime + timing, cross-frequency factor combination (mixed cadence). Deliver: a short + recommendation doc with concrete refinements to the current formula + (60/40, quality_within_theme, regime gate), grounded in citations. +- Output: `docs/methodology-research.md`; proposals applied only after user OK. + +### 2. Additional data sources per theme +- Research & (where cheap/feasible) wire one new honest Thai source per theme: + - banks → BOT commercial-bank loan growth / NPL (real, report already seen) + - retail → consumer-comfort/retail index + - utilities → BOT electricity demand proxy + - telecom → subscriber data (if public) + - property → BOT housing / transfer volume +- Add as FACTORS registry entries (declarative — that's the whole point). Sources + that need auth (BOT Portal API) are gated by user-provided key. + +### 3. Sources table auto-derived (not mock, not hardcoded) +- `dashboard._build_sources` → derive from FACTORS registry: iterate FACTORS, + group by source module, emit {แหล่ง, ขอบเขต, data_value, as_of, fetched_at}. +- Frontend sources table consumes `/api/v1/dashboard.sources` — so adding a FACTOR + auto-appends its source row. No hardcoded list in backend or frontend. + +### 4. Real backtest engine (replaces forward-only allocation) +- New `backend/app/backtest.py`: takes {start_date, end_date, capital, bucket_weights, + rebalance} and RUNS the allocation across time using the Yahoo price series: + - at each rebalance date, recompute combined scores (theme+firm+quarterlies) on + *point-in-time* basis (no future leak), allocate 50/20/30, track holdings. + - mark-to-market daily; accrue dividends; end → report: + `{capital, invested, final_value, price_pnl, dividend_income, net_return, win_rate, trades}` +- Backtest section in UI: pick start/end date, run, see performance summary. +- Persist run results (keep history) so refresh doesn't lose them. + +### 5. Next-update column +- Each source row gains `next_update_after` (from frequency → next scheduled + refresh time). Show "อัปเดตอีกครั้ง ~X" in the sources table. + +## Files +- `docs/methodology-research.md` (new, point 1) +- `backend/app/factors.py` (more factors; sources derive here) +- `backend/app/dashboard.py` (sources FROM registry + next_update) +- `backend/app/backtest.py` (new engine, point 4) +- `backend/app/__init__.py` (backtest endpoint, sources wiring) +- `frontend/src/App.vue` (backtest section + next-update col + sources from API) +- tests + +## Acceptance +1. Adding a FACTOR to `factors.py` auto-appends a sources-table row (no manual edit). +2. `/api/v1/dashboard/sources` has 5+ rows, all real, each with next_update_after. +3. Backtest with {start,end} returns a full P&L report (price + dividend + net), and + results persist (not lost on refresh). +4. Literature doc recommends ≥1 concrete formula refinement, cited. +5. npm build + full suite pass; browser shows backtest section + next-update column. + +## Open decisions for user +- D1: rebalancing frequency in backtest (monthly / quarterly / weekly)? +- D2: do BOT Portal API sources require your API key, or only auth-free ones for now? +- D3: backtest default range (last 12 months / last N quarters)? diff --git a/.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md b/.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md new file mode 100644 index 0000000..ca9d39a --- /dev/null +++ b/.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md @@ -0,0 +1,396 @@ +# Event-Driven PIT Backtest Implementation Plan + +> **For Hermes:** Execute task-by-task. Use TDD for accounting/event logic, keep one task in progress, commit by phase, and run `requesting-code-review` before delivery. + +**Goal:** Replace the current calendar-rebalance/backtest proxy with a strict point-in-time, event-driven portfolio simulation that reinvests sale proceeds and paid dividends, respects 100-share lots, records realized/unrealized P&L, and reconciles final equity exactly. + +**Architecture:** Build a deterministic event calendar from data releases, execution dates, dividend entitlement/payment dates, and the end valuation date. Feed those events through a portfolio ledger that owns cash, holdings, average cost, dividend receivables, orders, realized P&L, and costs. The API derives readiness/default dates from actual PIT coverage and blocks rather than silently falling back to current data. + +**Tech Stack:** Python 3.11, Flask, filesystem PIT stores, Vue/Vite, unittest/pytest. + +--- + +## Confirmed Decisions + +| Decision | Chosen default | Rationale | +|---|---|---| +| Backtest readiness | Strict PIT coverage for every required scoring input; otherwise block and report missing coverage | Prevents mixed-PIT results from looking valid | +| Signal execution | Freeze signal on release date D; execute at next available trading-day close after D | Avoids same-day close look-ahead | +| Dividend timing | Entitlement at ex-date; cash becomes available exactly 30 calendar days after ex-date | User-approved deterministic timing assumption; expose it as `ex_date_plus_30d` rather than claiming an observed payment date | +| Cost basis | Average cost | Simple, auditable, common portfolio accounting | +| Fees/taxes | All-in fee 0.3% of notional on every buy and every sell; do not add VAT/tax again | Explicit user decision; deterministic and auditable | +| Same-day releases | Coalesce all releases known by the execution cutoff into one frozen signal and one rebalance | Avoids churn from multiple events on the same day | +| End date on holiday/weekend | Value at latest trading close on or before end date | Deterministic and realistic | + +These decisions were confirmed by the user and are implementation requirements. + +--- + +## Current-State Gap Matrix + +| Requirement | Current behavior | Gap | +|---|---|---| +| Default start = first date advice is genuinely available | API/UI hardcode `2024-06-01` | No readiness calculation; current disk has no factor vintages or Siamchart vintage manifest | +| Default end = yesterday | API/UI hardcode `2026-06-01` | On 2026-08-28 +07, expected default is 2026-08-27 | +| Recompute only when new information arrives | `_rebalance_dates()` emits monthly/quarterly dates (`backend/app/backtest.py:103-125`) | Calendar-driven, not release/event-driven | +| Strict PIT | Optional `score_fn`; current data fallback exists | Current local data has 0 factor-vintage files and 0 Siamchart vintage snapshots, so no strict PIT-ready start exists | +| Execute after signal is known | `_latest_close(..., d)` uses close on rebalance date | Can use the same close that occurred before/while data was released | +| Dividend entitlement and cash timing | `credit_dividends()` is called only at end, using final holdings (`backtest.py:253-268`) | Wrong holdings-at-ex-date, no mid-run cash credit, no reinvestment, historic dividends can be overcounted | +| Sale P&L | Sell proceeds update cash only (`backtest.py:232-241`) | No cost basis, realized P&L, fees, or per-trade ledger | +| Reinvest gains/dividends | Sale cash can fund buys in the same rebalance; dividends are added only after all rebalances | Dividends cannot fund later buys; profit/cash trace is absent | +| 100-share lots | `allocate_capital()` floors to 100 shares | Already present; must preserve during target reconciliation | +| Final report | final holdings, total value, combined price P&L, dividend total | Must split realized/unrealized P&L, cash, receivables, costs, trades, and prove reconciliation | +| Current data coverage | Price snapshot: 9 symbols, 2024-01-03 to common 2026-07-17; no factor vintages/Siamchart manifest on disk | Strict PIT backtest should currently block, not manufacture a default start | + +--- + +## Target Event Flow + +```text +Coverage/readiness + → earliest date where ALL PIT inputs + executable prices exist + → initial recommendation event at start + → freeze target signal + → next trading day execution (100-share lots) + → for each data release: + coalesce same-day changes + recompute recommendation from PIT state + if target changes: sell first → credit proceeds/realized P&L → buy using current cash + → for each dividend: + ex-date: record entitlement using shares held before ex-date + ex-date + 30 calendar days: credit cash; available to next rebalance + → end date: + mark holdings to latest close ≤ end + report cash + market value + dividend receivable (separate) + reconcile all P&L components +``` + +Accounting invariant: + +```text +ending_equity - initial_capital + = realized_trading_pnl + + unrealized_trading_pnl + + dividend_cash_received + + accrued_dividend_receivable + - transaction_costs +``` + +`price_pnl = realized_trading_pnl + unrealized_trading_pnl` for compatibility. + +--- + +### Task 1: Add Strict Backtest Coverage and Default-Date Readiness + +**Objective:** Derive valid start/end defaults from actual PIT stores and block when coverage is incomplete. + +**Files:** +- Create: `backend/app/backtest_readiness.py` +- Modify: `backend/app/factor_vintages.py` +- Modify: `backend/app/siamchart_vintages.py` +- Modify: `backend/app/prices.py` or add a read-only adapter for price coverage +- Modify: `backend/app/__init__.py` +- Test: `backend/tests/test_backtest_readiness.py` + +**Steps:** +1. Write failing tests for: + - no factor vintages → `ready=false`, missing factor list + - no Siamchart snapshot → blocked + - price coverage starting after PIT factors → recommended start equals latest of first-ready dates + - recommended end = yesterday in Bangkok, bounded by latest price date + - user start earlier than readiness → HTTP 400 with exact missing coverage +2. Add store coverage APIs: + - `FactorVintageStore.first_release(factor_key)` and coverage summary + - `SiamchartVintageStore.first_retrieved_at()` + - price coverage: earliest/latest executable trading dates by required universe +3. Implement `BacktestReadiness` response: + - `ready`, `recommended_start`, `recommended_end`, `missing`, `coverage`, `timezone` +4. Add `GET /api/v1/backtest/readiness`. +5. Change `POST /api/v1/backtest` defaults to readiness-derived dates; never hardcode dates. +6. Verify targeted tests and API smoke. + +**Acceptance:** Current local dataset returns `ready=false` because factor/Siamchart PIT history is absent; it does not start a purported PIT run. + +--- + +### Task 2: Build the Unified Backtest Event Calendar + +**Objective:** Rebalance on information changes, not arbitrary monthly/quarterly dates. + +**Files:** +- Create: `backend/app/backtest_events.py` +- Modify: `backend/app/factor_vintages.py` +- Modify: `backend/app/siamchart_vintages.py` +- Modify: `backend/app/dividend_ledger.py` +- Test: `backend/tests/test_backtest_events.py` + +**Steps:** +1. Define typed/dataclass events: + - `SignalReleaseEvent(released_at, sources, factor_keys)` + - `ExecutionEvent(signal_date, execution_date)` + - `DividendEntitlementEvent(ex_date, symbol, per_share)` + - `DividendPaymentEvent(assumed_payment_date=ex_date+30d, entitlement_id, timing_method="ex_date_plus_30d")` + - `EndValuationEvent(end_date)` +2. Add chronological iterators to factor and Siamchart vintage stores. +3. Build release calendar from all PIT changes within `[start, end]`. +4. Coalesce releases on the same date. +5. Map each signal date to the next available trading date strictly after the release date. +6. Include dividend entitlement/payment events and final valuation event. +7. Create an assumed payment event exactly 30 calendar days after ex-date. Keep the dividend as a receivable until that event, then credit spendable cash. Expose the timing assumption in every event/result. +8. Test ordering, holiday/weekend handling, coalescing, and no-look-ahead execution. + +**Acceptance:** A factor release three months after start generates one new frozen signal and, only if targets differ, one rebalance on the next trading day. + +--- + +### Task 3: Implement the Portfolio Accounting Ledger + +**Objective:** Own all cash, holdings, cost basis, trades, dividends, and P&L in one auditable component. + +**Files:** +- Create: `backend/app/portfolio_ledger.py` +- Test: `backend/tests/test_portfolio_ledger.py` + +**Data model:** + +```python +Position(symbol, qty, average_cost) +Trade(date, signal_date, symbol, side, qty, price, notional, + cost_basis_released, realized_pnl, fees, cash_after, reason) +DividendReceivable(symbol, ex_date, assumed_payment_date, + timing_method="ex_date_plus_30d", qty_entitled, + per_share, amount, status) +PortfolioState(cash, positions, receivables, realized_pnl, + dividend_cash, fees, trades) +``` + +**Steps:** +1. Write RED tests for average-cost buys, partial sells, full sells, realized P&L, and no negative cash. +2. Implement `buy()`: + - qty positive and multiple of 100 + - all-in fee = `notional * 0.003` + - cost + fee <= cash + - recompute weighted average cost +3. Implement `sell()`: + - qty positive/multiple of 100 and <= position + - all-in fee = `notional * 0.003` + - realized P&L = proceeds - released average cost - sell fee + - credit net proceeds (`notional - fee`) to cash before later buys +4. Implement dividend entitlement at ex-date using quantity held before ex-date. +5. Implement payment-date cash credit; paid dividend can fund later orders. +6. Keep unpaid dividends as receivables, not spendable cash. +7. Add mark-to-market and reconciliation methods. +8. Test cash conservation and accounting identity after mixed buys/sells/dividends. + +**Acceptance:** Profitable sales and paid dividends increase cash available to subsequent 100-share-lot purchases; unpaid receivables do not. + +--- + +### Task 4: Reconcile Target Portfolio with Lot and Cash Constraints + +**Objective:** Convert each frozen recommendation into executable sell/buy orders using current total equity and current cash. + +**Files:** +- Modify: `backend/app/simulation.py` +- Create or modify: `backend/app/portfolio_rebalancer.py` +- Test: `backend/tests/test_portfolio_rebalancer.py` + +**Steps:** +1. Preserve canonical 50/20/30 allocation rules and 100-share lots. +2. Compute target quantities from current equity (cash + market value + paid dividends; exclude receivables). +3. Generate sells first, execute them, then recompute available cash. +4. Generate buys from remaining target deficits, never exceeding cash. +5. If a target lot is unaffordable, skip it and retain cash; never buy odd lots. +6. Rebalance only when target quantities differ from current quantities. +7. Add tests for: + - sale profit funds next purchase + - dividend payment funds next purchase + - insufficient cash leaves cash and skips order + - unchanged recommendation produces zero trades + - no negative cash under rounding/fees + +**Acceptance:** Every trade is in 100-share multiples, and ending cash is always non-negative. + +--- + +### Task 5: Rewrite Backtest Engine as Event Processor + +**Objective:** Replace `_rebalance_dates()` calendar loop with the event calendar and portfolio ledger. + +**Files:** +- Rewrite: `backend/app/backtest.py` +- Modify: `backend/app/pit_scorer.py` +- Test: `backend/tests/test_backtest.py` +- Test: `backend/tests/test_backtest_integration.py` + +**Steps:** +1. RED test the complete scenario: + - start recommendation buys A + - release three months later changes target to B + - sell A at profit + - dividend entitlement/payment occurs + - sale proceeds + paid dividend fund B purchase + - final holdings/value and P&L reconcile +2. Initialize from strict PIT readiness and initial signal. +3. Process events in order: + - release → freeze scores/target + - execution → reconcile portfolio + - ex-date → entitlement + - payment → cash credit + - end → mark-to-market +4. Record every release, rebalance, trade, and dividend event. +5. Ensure `leakage_guard=true` only if every score at every release is PIT; otherwise reject strict mode. +6. Remove monthly/quarterly frequency as the primary trigger. Optionally retain a separate explicit `scheduled` research mode, clearly non-default/non-event-driven. +7. Verify no double-counted dividends and no repeated historical credits. + +**Acceptance:** Engine output exactly follows the user-described lifecycle and passes accounting reconciliation for every tested scenario. + +--- + +### Task 6: Expand Backtest Result Contract and Durable Run Storage + +**Objective:** Return all requested end-state and P&L breakdowns with a durable audit trail. + +**Files:** +- Modify: `backend/app/backtest.py` +- Create: `backend/app/backtest_store.py` +- Modify: `backend/app/__init__.py` +- Test: `backend/tests/test_backtest_store.py` +- Modify: API tests in `backend/tests/test_api.py` + +**Response v2:** + +```json +{ + "start": "...", + "end": "...", + "initial_capital": 1000000, + "ending_cash": 12345, + "ending_market_value": 1100000, + "dividend_receivable": 500, + "final_equity": 1112845, + "net_change": 112845, + "net_return": 0.112845, + "realized_trading_pnl": 45000, + "unrealized_trading_pnl": 50000, + "price_pnl": 95000, + "dividend_cash_received": 18000, + "transaction_costs": 155, + "fee_rate": 0.003, + "holdings": [{"symbol":"...","qty":100,"average_cost":...,"last_price":...,"market_value":...,"unrealized_pnl":...}], + "trades": [...], + "dividends": [...], + "rebalance_events": [...], + "accounting_reconciled": true, + "leakage_guard": true +} +``` + +**Steps:** +1. Add result dataclasses and compatibility aliases (`final_value`, `dividend_income`). +2. Add durable JSON run store (atomic writes) rather than process-local `backtest_runs`. +3. Persist exact input coverage, event calendar, and scorer provenance with each run. +4. Add reconciliation assertions before serializing; fail closed if imbalance exceeds tolerance. +5. Test reload after app restart and API history output. + +--- + +### Task 7: Update Frontend Defaults, Controls, and Report + +**Objective:** Make UI defaults/readiness and results reflect the new engine honestly. + +**Files:** +- Modify: `frontend/src/App.vue` +- Test/build: `frontend/package.json` existing build + +**Steps:** +1. On load, call `/api/v1/backtest/readiness`. +2. Set start to `recommended_start` and end to `recommended_end`; remove hardcoded `2024-06-01` / `2026-06-01`. +3. If readiness is blocked, disable Run and show exact missing coverage. +4. Replace “รายเดือน/รายไตรมาส” primary control with “ตามข้อมูลใหม่ (event-driven)”. Keep scheduled frequency only under advanced/descriptive mode if retained. +5. Display: + - final holdings with qty, average cost, latest price, market value, unrealized P&L + - final equity, net change, return + - dividend cash received and dividend receivable + - realized trading P&L, unrealized P&L, total price P&L, fees + - event timeline / trade table +6. Keep visible PIT/non-PIT and dated-ledger/proxy badges. +7. Run production build and verify rendered result state. + +--- + +### Task 8: Verification and Independent Review Gate + +**Objective:** Prove correctness before shipping. + +**Files:** +- Create: `docs/test-evidence/YYYY-MM-DD-event-driven-backtest.md` +- Update: `docs/engineering-log.md` +- Update: `docs/HANDOFF.md` + +**Required tests/gates:** +1. Targeted event-calendar suite. +2. Portfolio-ledger suite. +3. Rebalancer suite. +4. Full backtest integration suite. +5. Full backend suite. +6. Frontend production build. +7. Static security scan; no secrets/debug leftovers. +8. `git diff --check`, compileall. +9. Live API probe with a deterministic temporary PIT fixture. +10. Browser verification of readiness-blocked and completed-result states. +11. Fresh independent reviewer subagent; fail closed on any logic/security blocker. +12. Auto-fix/re-review max two cycles. + +**Critical invariants:** +- cash never negative +- all buy/sell quantities multiples of 100 +- no trade before signal is knowable +- no dividend entitlement for shares bought on/after ex-date +- unpaid dividends never fund purchases +- paid dividends can fund subsequent purchases +- realized P&L uses average cost +- unchanged target creates no trade +- final accounting identity reconciles within one satang/rounding tolerance +- current local data returns readiness blocked rather than fake PIT performance + +--- + +## Files Likely to Change + +- `backend/app/backtest.py` +- `backend/app/backtest_readiness.py` (new) +- `backend/app/backtest_events.py` (new) +- `backend/app/portfolio_ledger.py` (new) +- `backend/app/portfolio_rebalancer.py` (new) +- `backend/app/backtest_store.py` (new) +- `backend/app/dividend_ledger.py` +- `backend/app/factor_vintages.py` +- `backend/app/siamchart_vintages.py` +- `backend/app/prices.py` +- `backend/app/simulation.py` +- `backend/app/__init__.py` +- `backend/tests/test_backtest*.py` +- new focused backend test modules +- `frontend/src/App.vue` +- engineering/test-evidence docs + +## Explicit Non-Goals + +- Live order execution / MT5 dispatch +- Guessing historical factor values that were never recorded +- Claiming PIT performance before coverage readiness passes +- Broker-specific fee assumptions without configuration +- Odd-lot trading + +## Main Risks + +1. **No current strict PIT-ready window:** the local store has no factor vintages/Siamchart manifest. The correct initial UI state is blocked. +2. **Dividend payment timing is assumed:** current Siamchart data supplies ex-date+DPS, not payment date. The user confirmed `assumed_payment_date = ex_date + 30 calendar days`; every result must disclose `timing_method=ex_date_plus_30d`. +3. **Price quality:** current Yahoo revised history is explicitly non-PIT; strict validated backtests need an evidence-bearing price archive contract. +4. **Universe coverage:** current price snapshot contains 9 symbols, not the full 49-symbol board. +5. **Response compatibility:** current UI/API expect scalar holdings and `price_pnl`; v2 needs compatibility aliases during migration. + +## Open Questions + +None for the core accounting/event model. The implementation decisions above are confirmed. diff --git a/.hermes/plans/2026-08-29_additional-data-sources-per-theme.md b/.hermes/plans/2026-08-29_additional-data-sources-per-theme.md new file mode 100644 index 0000000..1a459a1 --- /dev/null +++ b/.hermes/plans/2026-08-29_additional-data-sources-per-theme.md @@ -0,0 +1,101 @@ +# Architecture Plan — Additional Data Sources per Theme + Use-in-Analysis Enforcement + +> Status: DRAFT — pending user approval (decision gate G0). +> Author: Macky. Date: 2026-08-29. + +## Goal +Close the "few sources per theme" gap AND lock the user's rule — every data point fetched must feed analysis, not just display. This is the follow-up the declarative factor engine (2026-08-26 plan) explicitly deferred: "no new collectors yet; sources are added as follow-up." + +Two deliverables: +1. **Wiring convention that guarantees use-in-analysis** — a build/test pattern so a new source cannot be added without being consumed by `compute_theme_surprises`. +2. **Two new high-feasibility Thai sources**, wired end-to-end, verified against live pages: **BOT Balance of Payments** (report 60) and **BOT Commercial-bank loans** (reportID confirmed at build). A **medium-feasibility** source (**REIC house-price/transfer**) is scoped but flagged for a feasibility spike before committing. + +## Why this is needed (evidence — verified against live sources this session) +- 6 of 13 themes (`telecom_it`, `property`, `healthcare`, `consumer_staples`, `petrochem_materials`, `retail`) rest on **one BOT macro page**. Source count is genuinely thin. +- Found + fixed (this session, prior commit pending): `core_inflation_yoy` and `unemployment_pct` were **fetched but never used**; worse, every `sign:-1` factor (NPL/inflation/unemployment) used a **negative** theme weight → double-negative → **higher NPL raised the theme score**. Both fixed in `factors.py`/`themes.py` with regression tests. This plan prevents recurrence and adds genuinely *new providers*, not just new fields on the same page. +- **Feasibility verified live**: BOT `ReportPage.aspx?reportID=60` (Balance of Payments) responds to the exact `dgExcel` form-POST flow already implemented in `bot_tourism.py` (200, parseable — exports/imports/current-account series extracted this session). REIC homepage is a portal — data lives on subpages/endpoints (medium feasibility). + +## Why the current architecture makes this cheap +Adding a factor already needs **no scoring-function change** (`factors.py` registry + a theme factor line). The new work is only: +- a collector module exposing `.to_dict()`, +- registry + theme wiring, +- a scheduler `_REFRESH_JOBS` entry (so it collects + appears in source-health log), +- a `_source_label` provenance entry, +- tests. + +## Decision Gates (approve before code) +- **G0** — Approve the 2 source choices + the mandatory-use test contract (below). Exit → build B0. +- **G1** — Collector parses a frozen fixture; factor value extracted; **used-in-analysis test passes**. Exit → wire. +- **G2** — `/api/v1/dashboard` shows the new source row + `source_summary` factor_keys increases; board surprises change with it. Exit → done. + +## New collectors + +### B0 — Thailand External Sector / Trade (`thai_trade.py`) [high feasibility — VERIFIED current] +- **PIVOT from BOT BOP report 60:** probing live this session showed the BOT statistics portal's `reportID=60` only carries data **through 2011** (its `drpToYear` options cap at `2011xxxx`) — stale, not fit for a 2026 platform. Per the plan's gate ("do not ship a factor reading a wrong/stale series") we source the external sector from **TradingEconomics Thailand trade**, which is **live through June 2026** and reuses the exact `auto_credit.py` scraping pattern (same site/infobox): + - `https://tradingeconomics.com/thailand/current-account` → current-account balance (USD mn, June 2026 verified) + - `https://tradingeconomics.com/thailand/exports` → exports (USD mn, 2026 verified) + - `https://tradingeconomics.com/thailand/imports` → imports (USD mn, 2026 verified) +- Factors (declarative additions): + - `external_current_account` — current-account balance → sign +1 (strong external position supportive). Feeds **tourism**, **exploration**, **petrochem_materials**, **retail**. + - `external_exports` — exports USD mn → sign +1. Feeds **petrochem_materials**, **exploration**, **telecom_it**. + - `external_imports` — imports USD mn (domestic demand) → sign +1. Feeds **retail**, **consumer_staples**, **property**. +- Frequency: monthly. Scheduler cadence: monthly. +- Test: frozen fixture (one value sentence) → parse → assert current account / exports / imports; used-in-analysis test asserts a change in the fed themes. + +### B1 — BOT Commercial-bank loans (`bot_bank_loans.py`) [high feasibility — flow VERIFIED, reportID TBD] +- Same `dgExcel` adapter, new reportID. The exact BOT report for commercial-bank loans/deposits is **confirmed accessible in principle** (report 60 proved the flow); its numeric reportID is discovered at build time from BOT's statistics index / report search (do NOT hardcode an unverified ID). +- Fields expected: total loans outstanding, deposits, NPL (commercial banks), **loan growth YoY** as derived factor. +- Factors: + - `bank_loan_growth` — commercial-bank loan growth YoY → sign +1. Feeds **banks** (primary), **nonbank_finance**, **property** (mortgage-linked). + - `bank_deposit_growth` — deposit growth → sign +1 (funding base). Feeds **banks**. +- Frequency: monthly (BOT reports these monthly). Scheduler cadence: monthly. +- **Gate**: if the found reportID/data is not cleanly loan-related, fall back to BOT "credit by sector" report and re-scope the factor — do not ship a factor reading a wrong series. + +### B2 — REIC house-price / transfer index (`reic_housing.py`) [MEDIUM feasibility — SPIKE FIRST] +- Homepage is a portal (verified). The house-price index / residential transfer data live on a subpage or a data endpoint (likely JSON/PDF). **Spike task before G0 sign-off on this one**: locate the real data URL, confirm it is scrapeable without an API key. +- If feasible → factor `reic_housing_price_index` / `reic_transfer_value_yoy` → sign +1 → feeds **property**. +- If the only clean access is a PDF chart with no numeric series → **defer** (do not ship a display-only scrape). This is the exact "แค่ดึงข้อมูลมาเฉย ๆ" the user forbids. + +## Wiring checklist per source (all must land) +1. `backend/app/.py` — dataclass + `to_dict()` + `fetch_()` (+ frozen-fixture parse fn). +2. `factors.py` — FACTORS entries (source/frequency/fetch/value_key/sign/center/span). +3. `themes.py` — factor lines with **positive** weights on the sign-correct themes (the sign lives in the factor). +4. `scheduler.py` — `_REFRESH_JOBS` entry (module/fn/fetch_module/frequency) so it collects + shows in the health log. +5. `dashboard.py` — `_build_sources` `_source_label` map entry (provenance table). +6. `__init__.py` — no endpoint change needed (dashboard drives the UI); verify `/api/v1/dashboard` picks it up (G2). + +## Use-in-analysis enforcement (the user's rule, made a test) +Two contracts, added as tests so "fetched but not used" can never silently return: +- **Contract A (value_key resolves):** every `FACTORS.value_key` must be a real field the registered fetch module emits. Already added (`test_every_factor_value_key_resolves_to_a_fetched_field`). Extend it as new modules land. +- **Contract B (new factor moves scores):** for each new factor, a test feeds a neutral vs a hot value and asserts the target theme surprise changes **in the right direction** (bearish factors lower it). Pattern already added (`test_new_macro_factors_actually_move_scores`, `test_bearish_factors_move_score_the_right_way`). One such test per new factor. + +This dual contract is the build guarantee that fulfills "มีแหล่งข้อมูลใหม่ → ต้องใช้ในการวิเคราะห์จริง". + +## Data Model sketch +```python +# factors.py +"macro_current_account": {"source":"BOT BOP","frequency":"monthly", + "fetch":"bot_bop","value_key":"current_account_balance_mb", + "sign":1, "center":0.0, "span":80000.0}, +"bank_loan_growth": {"source":"BOT CB loans","frequency":"monthly", + "fetch":"bot_bank_loans","value_key":"loan_growth_yoy", + "sign":1, "center":5.0, "span":10.0}, + +# themes.py — positive weights; direction is in the factor +"banks": { "factors": [ {"key":"macro_investment","weight":1.0}, + {"key":"bank_loan_growth","weight":0.8}, + ... ] }, +``` + +## Non-Goals (v1) +- No new factors on the existing single BOT macro page beyond the two already wired (core inflation, unemployment) — avoid the "one page, many pseudo-independent factors" trap. +- No live trading / MT5 changes. No LLM in scoring. No API-key sources (free/scrapeable only). +- Do not ship a REIC scrape unless a clean numeric series exists. + +## Acceptance (definition of done) +1. Thailand external-sector (TradingEconomics trade) collector + 3 factors land, verified against a frozen fixture and a live fetch (current through 2026 — not the stale BOT-60 series). +2. BOT commercial-bank-loan collector + factors land (reportID confirmed; else scoped fallback). +3. REIC either lands with a real series **or** is explicitly deferred (no display-only scrape). +4. Every new factor has Contract A + Contract B tests green. +5. Full backend suite (352 → target ≥ 366) + `npm run build` pass. +6. `/api/v1/dashboard` `sources` + `source_summary.factor_keys` increase; board surprises visibly move with a hot value. diff --git a/.hermes/plans/2026-08-29_data-source-expansion-phase-2.md b/.hermes/plans/2026-08-29_data-source-expansion-phase-2.md new file mode 100644 index 0000000..d72e9c3 --- /dev/null +++ b/.hermes/plans/2026-08-29_data-source-expansion-phase-2.md @@ -0,0 +1,138 @@ +# Data Source Expansion Plan — Phase 2 (more Thai sources per theme) + +> **For Hermes:** Execute task-by-task. One task in progress at a time; commit by phase; run `requesting-code-review` before delivery. Every new source MUST end wired into `FACTORS` (one dict entry) **and** referenced by at least one `THEMES[].factors` row — so it feeds `compute_theme_surprises()`, never just the source table. Follow the existing collector template (`backend/app/thai_trade.py`, `auto_credit.py`). Run the full backend suite + `npm run build` after each phase. + +**Goal:** Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table. + +**Architecture (unchanged, declarative):** +``` +new source → collector module (fetch_() → Snapshot.to_dict()) + → FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span} + → THEMES[].factors [{key, weight}] + → compute_theme_surprises() picks it up with zero scoring-fn change +``` +Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by `thai_trade`. + +**Current state (verified, baseline green: 352/352 tests):** +- 7 live sources: `bot_tourism`, `auto_credit`, `auto_npl`, `bank_npl`, `energy_thai`, `macro_thai`, `thai_trade`. +- 17 FACTORS registered. All `value_key` resolve to a real fetched field (locked by `test_every_factor_value_key_resolves_to_a_fetched_field`). +- Sign convention FIXED this session: `sign` lives only in the factor; theme weights are positive magnitude. Regression-locked by `test_bearish_factors_move_score_the_right_way` (higher NPL ⇒ lower score, etc.). +- `thai_trade.py` (TradingEconomics current-account) is the reference collector: server-rendered HTML, `_fetch`/`parse`/`fetch_`/`to_dict`, registered in scheduler `_REFRESH_JOBS` + dashboard `_fetch_with_cache` + FACTORS. + +--- + +## Confirmed Decisions (from user: "วางแผนได้เลย") +| Decision | Chosen | Rationale | +|---|---|---| +| Scope | Phase A (macro-deepening) first, Phase B (sector) next | Fastest correctness win, lowest scrap fragility | +| New-source gate | Feasibility spike per source BEFORE building collector | Don't ship a wrong/stale/unscrapable series (same gate `thai_trade.py` used) | +| Integration | Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache | User rule: new data must feed analysis, not just be fetched | +| Test | One value-key-resolution + one direction test per new source | Locks "used in analysis" invariant | + +## Pending user decision (gate G0 — before Phase A code) +1. **Which sources to prioritize** — recommendation in priority order below. User may reorder/substitute. +2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page). + +--- + +## Current-State Gap Matrix (per theme, what feeds it today) +| Theme | Sources today | Gap / weakest link | +|---|---|---| +| banks | BOT macro (invest/inflation/NPL) — 3 fields, all BOT | No rate/loan-setting input; NPL only "financial sector" proxy | +| retail | BOT macro (consumption/inflation/unemployment) + TE imports | No direct retail-sales / consumer-mood series | +| consumer_staples | BOT macro + TE imports | Same as retail | +| telecom_it | BOT macro (consumption/investment) + TE exports | No telecom-specific (subs/data) series | +| property | BOT macro (invest/consumption/inflation) + TE imports | No real-estate-specific (transfer/mortgage) series | +| nonbank_finance | BOT macro + auto NPL + unemployment | No household-credit / consumer-loan series | +| healthcare | BOT macro (consumption/unemployment) | No healthcare/tourism-medical series | +| utilities | BOT macro (mfg) + TOP margin | No electricity-demand / generation series | +| tourism | BOT tourism + macro consumption + TE current acct | Covered well; optional: hotel occupancy | +| auto_credit | TE vehicle + BOT auto NPL | Covered well | +| refining_energy / exploration / petrochem | TOP + macro mfg/inflation + TE exports | Only TOP company; no commodity/oil price | + +--- + +## Phase A — deepen macro-proxy themes (priority order) +Each is a collector + FACTORS entry ×N + THEME wiring + tests. + +### A1. BOT policy/loan-rate + credit — banks, nonbank_finance +- Target: BOT monetary-policy / rate page (e.g. `bot.or.th` policy rate) + BOT credit/loan-growth report. +- Factors: `bank_policy_rate` (sign −1 for banks? higher rate squeezes demand), `bank_loan_growth_yoy` (sign +1). +- Feasibility: spike must confirm a scrapeable numeric series on a BOT page. + +### A2. Household / consumer credit — nonbank_finance (+ banks) +- Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report. +- Factor: `consumer_credit_yoy` (sign +1), `consumer_credit_npl` (sign −1). +- Feasibility: spike on BOT statistics page. + +### A3. Retail sales index — retail, consumer_staples +- Target: BOT or TradingEconomics "retail sales" Thailand YoY. +- Factor: `retail_sales_yoy` (sign +1). +- Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit). + +### A4. Consumer confidence — retail, consumer_staples, nonbank_finance +- Target: UTCC / Kasikorn Research consumer-confidence index (free HTML). +- Factor: `consumer_confidence` (sign +1). +- Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence". + +## Phase B — sector-specific (IMPLEMENTED 2026-08-29 via TradingEconomics single-page snapshots) +### B1. Property: `te_property_prices` (residential property prices % YoY) — DONE, feeds property theme +### B2. Utilities: EPPO electricity — DEFERRED (no clean single-page TE snapshot; needs EPPO scraper) +### B3. Energy breadth: PTT/PTTEP/BCP quarterly — DEFERRED (needs company-IR scrapers, heavier) +### B4. Telecom/backdrop: `te_business_confidence` — DONE, feeds telecom_it + property + healthcare +### B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence) +Both added to the existing `te_thailand.py` module (2 extra TE pages) — same reviewed pattern, +2 factors, +4 tests. + +## Remaining backlog (needs dedicated scrapers, not single-page snapshots) +- REIC property transfer/housing supply (TH-specific, richer than a TE index) +- EPPO electricity demand/generation for utilities +- PTT/PTTEP/BCP/IRPC quarterly financials (beyond TOP) for energy breadth +- NBTC subscriber/data for a true telecom-specific series + +--- + +## Task Breakdown +### Task A0 — Feasibility spikes (gate, not build) +For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under `backend/scripts/spike_.py`; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value. +- Exit: a table of "source → scrapable? → value → period". + +### Task A1 — BOT rate/loan + credit factors +- Add module `backend/app/bot_rates.py` (or extend existing) with Snapshot + `fetch_()` + `.to_dict()`. +- Register FACTORS (policy_rate, loan_growth…) + wire into `banks`/`nonbank_finance` THEMES. +- Add `_REFRESH_JOBS` row + dashboard cache line. +- Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand). +- Verify: full backend suite green; `/api/v1/dashboard` themes show the new factors in `sources`. + +### Task A2 — Household/consumer credit (nonbank_finance) +- Same pattern as A1; factor `consumer_credit_yoy` (+1), `consumer_credit_npl` (−1) into `nonbank_finance` (+ maybe banks). +- Tests + suite green. + +### Task A3 — Retail sales (retail, consumer_staples) +- Collector on TE Thailand retail-sales page; factor `retail_sales_yoy` (+1) into retail + consumer_staples. +- Tests + suite green. + +### Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance) +- Collector; factor `consumer_confidence` (+1). Use TE fallback if UTCC is paywalled. +- Tests + suite green. + +### Task A5 — Frontend macro chips / theme cards +- Add the new read values to `App.vue` theme cards / macro chips (single-colour, no new library, follow existing `theme-read-value` pattern). +- `npm run build` green; visual check at 320×568 and 500×768 (mobile single-column). + +### Task B1..B5 — Phase B (only after Phase A accepted) +- Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests. + +--- + +## Acceptance (definition of done) +1. Every new FACTOR appears in ≥1 `THEMES[].factors` and is referenced by dashboard `_build_sources` (auto from registry) — **no source shows in the table without feeding a surprise**. +2. `test_every_factor_value_key_resolves_to_a_fetched_field` still passes for every new factor (value_key is a real fetched field). +3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score). +4. Full backend suite green (was 352) + `npm run build` green. +5. Scheduler `_REFRESH_JOBS` + dashboard cache include every new source. +6. `/api/v1/dashboard` sources table length tracks FACTORS count; new theme reads appear on the dashboard. + +## Open questions for G0 +- Priority/order of A1–A4 (recommended order above; user may reorder). +- Source substitution if a spike fails: fallback list provided per task. +- Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed? diff --git a/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md b/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md new file mode 100644 index 0000000..c68a46b --- /dev/null +++ b/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md @@ -0,0 +1,50 @@ +# 2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug + +## Scope +Two-part request from the owner: +1. **More + genuinely-used data sources per theme** — the owner felt the source + count was too few and insisted any newly-fetched data must feed the analysis + (theme surprise), not just the provenance table. +2. **Frontend padding/margin issues** — several panels had misaligned spacing. + +## What changed (all verified) +| Area | Change | Evidence | +|---|---|---| +| `factors.py` | Wired 2 **fetched-but-unused** BOT fields into the registry: `macro_core_inflation` (core inflation, sign −1) and `macro_unemployment` (sign −1) → they now actually move themes | Before, `macro_thai.to_dict()` emitted 7 fields but only 5 were registered factors — 2 were dead display data | +| `themes.py` | Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights) | direction tests | +| **Sign bug fix** | Found + fixed a **double-negative**: several factors carry `sign: -1` (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in `sign`. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it. | `test_bearish_factors_move_score_the_right_way` | +| **NEW `te_thailand.py`** | New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence | live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8 | +| `themes.py` | Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence) | direction tests | +| `scheduler.py` / `dashboard.py` | Registered `te_thailand` in `_REFRESH_JOBS` + dashboard cached-fetch + `_build_sources` provenance | dashboard live build: 8 sources | +| `frontend/App.vue` | Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP) | build clean | +| `frontend/style.css` | **padding/margin + undefined-variable fix** — added missing design tokens (`--card/--foreground/--accent/--font/...`) that the theme/modal components reference but `:root` never declared (they rendered transparent/wrong-color); zeroed section-panel padding so `.signal-header` is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels) | build clean | + +## Key decisions +- **One new module** (`te_thailand.py`) covers 4 plan items (A1 rate/credit, A2 + consumer credit, A3 retail sales, A4 confidence) because all four series live + on just two TradingEconomics pages — avoids 4 fragile scrapers. +- **Every source must feed a surprise**: tightened `test_every_factor_value_key_resolves_to_a_fetched_field` so a factor whose `value_key` the collector never emits fails the suite — no dead factors. +- Sign convention locked: `sign` = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly. + +## Verified (evidence) +- Full backend suite: **360 tests OK** (up from 352 — 8 new `test_te_thailand`). +- Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise −0.165 + (correctly negative from retail-sales −14.5% YoY) vs old BOT-only (positive). +- Frontend `npm run build` clean. +- Independent code review: see delegation verdict (fail-closed). + +## Phase B (same day, follow-up "ทำต่อได้เลย") +Extended the same `te_thailand` module with 2 more TradingEconomics pages: +- `te_property_prices` (residential property prices +1.26% YoY) → **property** theme +- `te_business_confidence` (46.7) → **telecom_it + property + healthcare** ++2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now +**362 tests OK**; frontend build clean; live fetch confirmed (1.26 / 46.7). +Cleared a stale daily-cache `te_thailand` entry so the new fields show immediately. + +## Backlog (needs dedicated scrapers, not single-page snapshots) +REIC property supply, EPPO electricity (utilities), PTT/PTTEP/BCP quarterly breadth +(energy), NBTC subscriber/data (telecom). Marked deferred in the plan. + +- Note: a **concurrent process** also landed `thai_trade.py` (external-sector + exports/imports/current-account) and external_* factors mid-session; its 3 + initially-broken tests were fixed to reach the 360-green baseline here.