docs: data-source expansion plan + engineering log (te_thailand, sign fix, ui fix)
This commit is contained in:
Kunthawat Greethong
@@ -0,0 +1,150 @@
|
||||
# Multi-Theme + Simulation Implementation Plan — REVISION 2
|
||||
|
||||
> **For Hermes:** Execute task-by-task with `todo` tracking and per-task commits. TDD for all logic; visual verification for UI/CSS.
|
||||
|
||||
**Goal:** Rebuild the single-theme tourism demo into a **multi-theme research platform** with (a) a combined dashboard (all themes + stock recommendation table), (b) per-theme detail pages, (c) a **capital allocation simulation** (same engine used for both backtest and forward test), and (d) an MT5 order interface (forward test sends dry-run orders; live gate kept off in this round). UI in **Thai** (technical terms stay English). Uses **real data + daily cache**, not mocks.
|
||||
|
||||
**Themes (3 alternative factors):**
|
||||
1. `tourism` — Tourism Pulse (signal exists, real BOT data path)
|
||||
2. `auto_credit` — Auto Credit Cycle (registry slot; data pipeline needs a real source before it can score)
|
||||
3. `refining_energy` — Refining / Energy Spread (registry slot; data pipeline TBD)
|
||||
|
||||
**Siamchart** is treated as a **fundamental data provider**, not a theme. It supplies PE/EPS/Yield/P/BV/ROE per symbol and contributes to the combined score with a **lower weight**.
|
||||
|
||||
---
|
||||
|
||||
## Confirmed requirements (from user, 2026-08-25)
|
||||
|
||||
1. **3 themes = 3 alternative factors** (tourism, auto_credit, refining_energy). A symbol can appear in multiple themes.
|
||||
2. **Score combining:** each symbol has a *theme score* (average across all themes that score it) and a *siamchart score*. Combined = **theme_score × 60% + siamchart_score × 40%**.
|
||||
3. **8-symbol live data** is OK for testing logic; production must cover the full universe (collectors must be scalable, not hard-coded to 8).
|
||||
4. **"ไม่จ่ายปันผล" (bucket 2)** = a high-score stock that does **not** pay a dividend (is_dividend = False / yield ≤ 0).
|
||||
5. **Paper ledger is replaced** by a forward test that reuses the **same simulation engine as backtest** — the only difference is real MT5 order dispatch. We build the MT5 order interface; live dispatch remains gated off (forward/dry-run in this round).
|
||||
6. **Data quality / provenance = real sources, cached daily** — not fixture mocks. Provenance shows where each alternative factor comes from.
|
||||
7. **Signal board + Stock board = ONE table**, signal-focused.
|
||||
8. **UI language = Thai** (technical terms English).
|
||||
9. **Remove unused/unnecessary dashboard sections.**
|
||||
|
||||
---
|
||||
|
||||
# PART A — Multi-Theme Backend
|
||||
|
||||
## Task A1: Theme registry
|
||||
**Files:** create `backend/app/themes.py`; create `backend/tests/test_themes.py`
|
||||
|
||||
Registry with 3 theme slots + their factor provider. `list_themes()` returns them. Registry is pluggable so a theme can be `enabled=False` while its data pipeline is absent.
|
||||
|
||||
## Task A2: Combined signal aggregation
|
||||
**Files:** modify `backend/app/themes.py`, `backend/app/__init__.py`; tests
|
||||
|
||||
`/api/v1/dashboard` returns:
|
||||
- Per-symbol `theme_score` = mean of enabled themes that score it
|
||||
- `siamchart_score` (normalized)
|
||||
- `combined_score = 0.6 * theme_score + 0.4 * siamchart_score`
|
||||
- `side` (LONG/SHORT/NEUTRAL) from combined
|
||||
- per-theme contributions
|
||||
|
||||
Normalization: each provider's raw score is z-scored (or min-max over the live universe) before combining, so weights are meaningful across heterogeneous scales.
|
||||
|
||||
## Task A3: Per-theme detail endpoint
|
||||
`/api/v1/themes/<id>` — 404 unknown; returns that theme's observations/surprise/signals/lineage. Also exposes `/api/v1/themes` (registry incl. enabled status + data source).
|
||||
|
||||
## Task A4: Daily cache layer (real data, no mocks)
|
||||
**Files:** create `backend/app/daily_cache.py`; tests
|
||||
|
||||
Fetch-once-then-cache daily:
|
||||
- Keyed by `(source, as_of)`.
|
||||
- TTL ~24h; stale-if-error fallback to last good cache.
|
||||
- Ensures tourism + siamchart reads hit cache instead of re-fetching every dashboard load, and sets up the "update daily" cadence the user wants.
|
||||
|
||||
---
|
||||
|
||||
# PART B — Simulation / Backtest / Forward-Test Engine
|
||||
|
||||
## Task B1: Price-series loader
|
||||
**Files:** create `backend/app/simulation.py`; tests
|
||||
|
||||
Load Yahoo price snapshot `series[SYM].bars` → `{date: adjusted_close}` for all symbols. Covers 2024-01-03 → 2026-08-24.
|
||||
|
||||
## Task B2: Capital allocation core (`allocate_capital`)
|
||||
**Files:** `backend/app/simulation.py` + tests
|
||||
|
||||
Given `capital`, per-symbol `combined_score`, `is_dividend`, `dividend_yield`, latest `price`:
|
||||
- **Bucket 1 (50%)** — highest combined_score among **dividend-paying** names.
|
||||
- **Bucket 2 (20%)** — highest combined_score among **non-dividend** names.
|
||||
- **Bucket 3 (30%)** — highest **dividend yield** among names **not already bought in bucket 1** (ignores score).
|
||||
|
||||
Per symbol: **minimum 100 shares**. Rank by combined_score desc. If a bucket's first pick can't afford 100 shares, try progressively cheaper eligible names (that still pass the bucket's criteria). If no eligible name fits, leave remainder as cash.
|
||||
- Allocation math: `qty = max(100, floor(available_cash / price / 100) * 100)` within bucket cash.
|
||||
- Overflow flows bucket → bucket → cash.
|
||||
|
||||
## Task B3: Simulation endpoint + forward-test mode
|
||||
**Files:** `backend/app/__init__.py`; tests
|
||||
|
||||
- `POST /api/v1/simulation` `{capital}` → allocation plan (per-bucket orders, qty, notional, total, unallocated cash, data caveat: revised history not PIT).
|
||||
- `POST /api/v1/simulation/forward` — same engine, but marks orders `execution=paper` and, if an MT5 bridge is configured, would dispatch. **Live dispatch gated off** (env `MT5_ENABLE_ORDER` must be set AND explicit approval).
|
||||
|
||||
## Task B4: MT5 order interface (interface only, live gated)
|
||||
**Files:** create `backend/app/mt5_bridge.py`; tests (mock `MetaTrader5` per `mt5-data-service` skill gotcha)
|
||||
|
||||
Abstract `MT5Bridge` with: `connect`, `symbol_resolve`, `order_new` (dry-run default), and a structural kill switch — no real dispatch path in code unless `MT5_ENABLE_ORDER=1` AND explicit user approval. Follows the `mt5-data-service` / `mql5-ea-development` skill patterns (Windows-only `MetaTrader5` stub for tests).
|
||||
|
||||
---
|
||||
|
||||
# PART C — Frontend Multi-Theme UI (Thai, signal-first)
|
||||
|
||||
## Task C1: Route shell
|
||||
**Files:** `frontend/src/App.vue`, `style.css`
|
||||
|
||||
Hash-routing: `#/` (dashboard), `#/theme/:id`, `#/simulation`. Sidebar nav in Thai: แดชบอร์ด, ธีม, การจำลอง (Simulation).
|
||||
|
||||
## Task C2: Combined Dashboard (Thai)
|
||||
- Theme strip: each active theme's surprise/label.
|
||||
- **ONE stock table** (signal-first: combined score, side, theme contributions, then fundamentals PE/EPS/Yield/P/BV/ROE). Dividend filter + sortable columns kept. This **replaces** the separate Signal board + Stock board.
|
||||
- Remove unused sections (Paper Ledger card, live receiver text, etc.).
|
||||
|
||||
## Task C3: Theme detail page
|
||||
`#/theme/:id` — that theme's observations (surprise bars), provenance (source used), and its signal contribution table.
|
||||
|
||||
## Task C4: Simulation tab
|
||||
`#/simulation` — Thai UI: enter capital, choose backtest/forward, render 3 buckets with orders; show unallocated cash and the "revised history, paper-only" caveat.
|
||||
|
||||
## Task C5: Provenance (real data)
|
||||
Dashboard provenance section reads from live cache metadata: source id, retrieved_at (daily cache), raw hash. No fixture labels.
|
||||
|
||||
---
|
||||
|
||||
# Data requirements / open items (NOT guessed)
|
||||
|
||||
- **auto_credit data source**: needs a real public/paid source for vehicle sales, auto loan growth, auto NPL. Not yet identified. `auto_credit` theme stays in registry as `enabled=False` until a source + collector exist.
|
||||
- **refining_energy data source**: crack/GRM spread or Thai refining/energy stats. Not yet identified. `enabled=False` until built.
|
||||
- **Full SET50 universe prices**: Yahoo history currently covers 8 symbols; production simulation needs price history for the full universe. Requires extending the price collector to all symbols (scrape/SETSMART/MT5).
|
||||
|
||||
The three-theme registry is the *interface*; this round builds tourism + siamchart fully (real, cached) and stubs auto/energy until their data exists. Simulation works on whatever real symbols have both a price series and a score.
|
||||
|
||||
---
|
||||
|
||||
# Verification
|
||||
|
||||
1. Full backend suite (`-W error unittest discover`) all pass (existing 140 + new).
|
||||
2. `cd frontend && npm run build` success.
|
||||
3. Browser (Thai): dashboard lists themes (enabled + disabled), combined signal-first table with dividend filter + sort, simulation returns allocation, per-theme detail works.
|
||||
4. `git diff --check`, static scan clean, independent fail-closed code review.
|
||||
5. Per-task commits.
|
||||
|
||||
---
|
||||
|
||||
# Closed/open per user's 10 points
|
||||
|
||||
| # | User requirement | Status in plan |
|
||||
|---|---|---|
|
||||
| 1 | 3 themes (tourism, auto, energy) | Registry + tourism live; auto/energy `enabled=False` pending data source |
|
||||
| 2 | combined = 60% theme + 40% siamchart | A2 |
|
||||
| 3 | 8-symbol OK for logic, prod full | Simulation scales to any symbol with price+score |
|
||||
| 4 | bucket2 = high score, no dividend | B2 |
|
||||
| 5 | paper→forward test = same engine; MT5 iface | B3, B4 |
|
||||
| 6 | real data + daily cache, provenance real | A4, C5 |
|
||||
| 7 | one signal-first table | C2 |
|
||||
| 8 | Thai UI | C1–C4 |
|
||||
| 9 | remove unused | C2 |
|
||||
| 10 | (mt5 scope) interface built, live gated | B4 |
|
||||
@@ -0,0 +1,203 @@
|
||||
# Multi-Theme Dashboard + Capital Simulation Engine Implementation Plan
|
||||
|
||||
> **For Hermes:** Execute task-by-task. Use `todo` progress tracking and per-task commits. TDD for logic, visual verification for UI/CSS.
|
||||
|
||||
**Goal:** Turn the current single-theme (tourism) demo into a **multi-theme dashboard** (combined overview page + per-theme detail pages) plus a **separate Simulation tab** that runs capital allocation (50/20/30) ranked by signal score, using the existing Siamchart fundamental snapshot and the existing Yahoo price history (2024-01-03 → 2026-08-24).
|
||||
|
||||
**Architecture:** A theme registry that each alternative factor registers into (tourism, siamchart-fundamental). The dashboard combines all themes' signals into a recommendation; each theme has its own detail page. The simulation engine consumes the combined signal scores + 2.5yr price history + dividend yields to allocate a user-entered capital across three buckets (min 100 shares/share).
|
||||
|
||||
**Tech Stack:** Existing Flask backend (`backend/app/`), Vite+Vue frontend (`frontend/src/`). New modules `backend/app/themes.py`, `backend/app/simulation.py`. New route `/api/v1/simulation`. Frontend route-based dashboard (`#/`, `#/theme/:id`, `#/simulation`).
|
||||
|
||||
---
|
||||
|
||||
## Current-state findings (from exploration)
|
||||
|
||||
- **Themes today:** `tourism` signal (`backend/app/tourism.py::compute_tourism_signal` → `{theme, theme_surprise, observations, signals, as_of}`) + `siamchart` factor view (`backend/app/siamchart_factors.py::build_factor_view` → per-symbol PE/EPS/Yield/P/BV/ROE/is_dividend).
|
||||
- **`/api/v1/factors`** already merges Siamchart fundamentals with tourism signal per symbol (added in `commit 8a6991b`).
|
||||
- **Price history EXISTS**: `backend/data/prices/snapshots/prices-yahoo-chart-2024-01-01-2026-08-24-a1cf5b3c2ecf.json` — `{benchmark_symbol: "SET50", series: {SYM: {bars: [{date, open, high, low, close, adjusted_close, volume}]}}}` covering 2024-01-03 → 2026-08-24 for 8 symbols (AOT, AWC, BEM, CPALL, CPN, CRC, MINT, PTT) + index_SET. ~1.3 MB per vintage.
|
||||
- **Fundamental snapshot**: `backend/data/siamchart/set50_master.json` — per-symbol EPS/PE/Yield%/is_dividend (49 symbols).
|
||||
- **Paper ledger** (`backend/app/paper.py`) is a manual record-a-fill tool — NOT the automated allocation we now build.
|
||||
|
||||
---
|
||||
|
||||
# PART A — Multi-Theme Backend Architecture
|
||||
|
||||
## Task A1: Create a theme registry
|
||||
|
||||
**Objective:** A single source enumerating all available themes and their factor/signal providers.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/themes.py`
|
||||
|
||||
**Step 1: Write test** → `backend/tests/test_themes.py`
|
||||
```python
|
||||
def test_registry_lists_themes():
|
||||
from app.themes import list_themes
|
||||
themes = list_themes()
|
||||
assert {"id": "tourism"} in themes
|
||||
assert {"id": "siamchart"} in themes
|
||||
```
|
||||
Run: `PYTHONPATH=backend .venv/bin/python -W error -m unittest tests.test_themes -v`
|
||||
Expected: FAIL (module missing).
|
||||
|
||||
**Step 2: Implement `themes.py`**
|
||||
```python
|
||||
THEMES = [
|
||||
{"id": "tourism", "label": "Tourism Pulse", "kind": "signal",
|
||||
"source": "bot_tourism", "provider": "compute_tourism_signal"},
|
||||
{"id": "siamchart", "label": "Siamchart Fundamentals", "kind": "fundamental",
|
||||
"source": "siamchart", "provider": "siamchart_factors"},
|
||||
]
|
||||
|
||||
def list_themes():
|
||||
return THEMES
|
||||
```
|
||||
**Step 3: rerun test** → PASS.
|
||||
|
||||
**Step 4: Commit:** `git add backend/app/themes.py backend/tests/test_themes.py && git commit -m "feat(backend): theme registry"`
|
||||
|
||||
---
|
||||
|
||||
## Task A2: Combined signal aggregation endpoint
|
||||
|
||||
**Objective:** `/api/v1/dashboard` returns an aggregate across all themes: per-symbol combined score (sum across themes that have a signal), side, the themes that fired, plus carousel info per theme.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/__init__.py`
|
||||
- Create: `backend/tests/test_dashboard.py`
|
||||
|
||||
**Step 1: Test** — combined aggregation with a mock theme.
|
||||
**Step 2: Implement** an aggregate function in `themes.py` that consumes each theme's signals (tourism signals + siamchart scores) and combines per symbol.
|
||||
**Step 3: PASS + commit** `feat(backend): aggregated multi-theme dashboard endpoint`
|
||||
|
||||
---
|
||||
|
||||
## Task A3: Per-theme detail endpoint
|
||||
|
||||
**Objective:** `/api/v1/themes/<theme_id>` returns that theme's observations/surprise/signals for the detail page.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/__init__.py`
|
||||
- Create: tests in `test_dashboard.py`
|
||||
|
||||
**Step:** route dispatch to the theme's provider; validate `theme_id` against registry; 404 unknown. PASS + commit.
|
||||
|
||||
---
|
||||
|
||||
# PART B — Simulation / Capital Allocation Engine
|
||||
|
||||
## Task B1: Price-history loader
|
||||
|
||||
**Objective:** Load the latest Yahoo price snapshot series for a symbol as date→adjusted_close.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/simulation.py`
|
||||
- Create: `backend/tests/test_simulation.py`
|
||||
|
||||
**Step 1: Test**
|
||||
```python
|
||||
def test_load_price_series(tmp_path):
|
||||
snap = {"benchmark_symbol":"SET50","series":{"PTT":{"bars":[
|
||||
{"date":"2024-01-03","adjusted_close":50.0},
|
||||
{"date":"2024-01-04","adjusted_close":51.0}]}}}
|
||||
# write temp snapshot, call loader
|
||||
series = load_price_series(snap, "PTT")
|
||||
assert series == {"2024-01-03":50.0,"2024-01-04":51.0}
|
||||
```
|
||||
**Step 2: Implement** `load_price_series(snapshot, symbol)` → `dict[date, adjusted_close]`; reject unknown symbol with `SimulationError`. PASS + commit `feat(backend): price series loader`.
|
||||
|
||||
---
|
||||
|
||||
## Task B2: Capital allocation core
|
||||
|
||||
**Objective:** Given capital, per-symbol signal score + is_dividend + dividend yield, allocate across three buckets:
|
||||
- **Bucket 1 (50%)**: highest signal score among **dividend-paying** names
|
||||
- **Bucket 2 (20%)**: highest signal score among **non-dividend** names (combine with leftover/overflow from bucket 1 shortfall)
|
||||
- **Bucket 3 (30%)**: highest dividend yield among names **not** already in bucket 1 (ignore profit)
|
||||
|
||||
Each bucket buys from its ranked list, **minimum 100 shares per symbol**, at latest price; leftover cash (from price-bucket rounding) flows to next bucket / stays unallocated.
|
||||
|
||||
**Files:** `backend/app/simulation.py`, `backend/tests/test_simulation.py`
|
||||
|
||||
**Step 1: Test** `test_allocate_buckets_basic` — small capital, verify 50/20/30 money split, min-100-share floor, ranking by score.
|
||||
**Step 2: Test edge:** insufficient capital for min 100 shares in a bucket → skip to next; no infinite loop.
|
||||
**Step 3: Implement** `allocate_capital(capital, factors, prices)` returning per-bucket orders + unallocated cash.
|
||||
**Step 4: PASS + commit** `feat(backend): capital allocation engine (50/20/30)`
|
||||
|
||||
---
|
||||
|
||||
## Task B3: Simulation endpoint
|
||||
|
||||
**Objective:** `POST /api/v1/simulation` takes `{capital}` and returns the allocation plan (per bucket, per-symbol qty/price/notional, total invested, unallocated).
|
||||
|
||||
**Files:** `backend/app/__init__.py`, tests.
|
||||
|
||||
**Step:** wire `allocate_capital` to the endpoint; validate capital>0 finite; 400 on bad input. PASS + commit `feat(backend): simulation endpoint`.
|
||||
|
||||
---
|
||||
|
||||
# PART C — Frontend Multi-Theme UI
|
||||
|
||||
## Task C1: Route-based shell (Dashboard + theme detail + simulation)
|
||||
|
||||
**Objective:** `App.vue` becomes route-aware (hash routing): `#/` dashboard, `#/theme/:id`, `#/simulation`. Sidebar nav splits into these.
|
||||
|
||||
**Files:** `frontend/src/App.vue`, `frontend/src/style.css`
|
||||
- Keep `Stock board`/`Signal board` semantics but restructure nav:
|
||||
- **Dashboard** (`#/`) — combined overview + stock recommendation
|
||||
- **Simulation** (`#/simulation`) — separate tab
|
||||
- **per-theme** (`#/theme/:id`) — theme detail
|
||||
- TDD skipped (visual); verify via dev server + browser.
|
||||
|
||||
**Verify:** `npm run build` passes; open dev server; nav switches views.
|
||||
|
||||
## Task C2: Combined Dashboard page
|
||||
|
||||
**Objective:** Overview aggregating all themes: theme surprise cards, combined stock recommendation table (with dividend filter + sortable columns — reuse existing logic), theme list with links.
|
||||
|
||||
**Files:** `frontend/src/App.vue`
|
||||
- Reuse `sortedFactorRows`/`dividendOnly`/`setSort` from current stock board.
|
||||
- Add theme-surprise strip + "combined" context.
|
||||
|
||||
## Task C3: Theme detail page
|
||||
|
||||
**Objective:** `#/theme/:id` shows that theme's observations (surprise bars), source lineage (provenance), and its own signal table.
|
||||
|
||||
**Files:** `frontend/src/App.vue`
|
||||
- Fetch `/api/v1/themes/:id`.
|
||||
|
||||
## Task C4: Simulation tab UI
|
||||
|
||||
**Objective:** `#/simulation` — input capital, POST `/api/v1/simulation`, render 3 buckets (each with the buy list, qty, notional; min-100 clarity), unallocated cash, and a note on the data limits (revised history, not PIT).
|
||||
|
||||
**Files:** `frontend/src/App.vue`, `style.css`
|
||||
- Verify via browser after wiring.
|
||||
|
||||
---
|
||||
|
||||
# Migration pitfalls this plan must respect (from `plan` skill)
|
||||
|
||||
- **Multi-theme wiring**: do not hardcode "tourism" prose; pull labels from `/api/v1/themes`.
|
||||
- **Simulation honesty**: clearly label output as a *paper/backtest* on **revised vendor history** (`point_in_time=false`) — never a validated PIT claim.
|
||||
- **Bucket edge cases**: min-100-share floor can leave a bucket with zero capital allocated → must flow to next bucket gracefully (tested).
|
||||
- **XSS**: all UI via Vue `{{ }}`, no `v-html`.
|
||||
- **Per-theme 404**: unknown theme_id must 404, not silently render empty.
|
||||
|
||||
---
|
||||
|
||||
# Verification (final)
|
||||
|
||||
1. `PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests` → all OK (existing 140 + new).
|
||||
2. `cd frontend && npm run build` → success.
|
||||
3. Browser: dashboard lists themes; `#/simulation` with capital input returns an allocation table; dividend filter + sort still work.
|
||||
4. `git diff --check`, static scan (no secrets/eval/exec/pickle).
|
||||
5. Independent code review (fail-closed JSON) before commit + push.
|
||||
|
||||
---
|
||||
|
||||
# Open questions / assumptions (call out if wrong)
|
||||
|
||||
- **"Combined signal score"**: sum of normalized scores across themes that provide a signal for a symbol. If only one theme fires, its score is the combined score. (Assumed; the user said "signal score from factor combined".)
|
||||
- **Dividend filter for bucket 1 vs bucket 3**: bucket 1 = dividend-paying AND high score; bucket 3 = high dividend yield regardless of score, **excluding bucket-1 names**. User: "จ่ายเงินปันผลมากที่สุด (ไม่ซ้ำกับข้อ 1 และไม่สนเรื่องโอกาสทำกำไร)". Implemented as exclusion, not a forced different-name set.
|
||||
- **Prices**: only 8 symbols have Yahoo history (AOT, AWC, BEM, CPALL, CPN, CRC, MINT, PTT) — the simulation can only allocate among symbols with both a price series AND a factor score. Symbols without a price series are skipped with a warning.
|
||||
- **Min 100 shares** — quantity may exceed cash in bucket for high-priced names; we buy floor(available_cash / price / 100) * 100 shares, leftover flows onward.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Dashboard Polish — Table UX, Per-Theme Narrative, Per-Symbol Analysis Page
|
||||
|
||||
> **For Hermes:** This plan covers the 7-point UX + analysis-transparency review. No code until user approves.
|
||||
|
||||
**Goal:** Make the stock table informative & sortable, explain each theme's analysis in narrative (not just a number), and give every symbol a dedicated detail view showing HOW the system analyzed it (weights, components, combined-score derivation) so the user can verify the analysis honours the agreed methodology.
|
||||
|
||||
---
|
||||
|
||||
## The 7 complaints → fix
|
||||
|
||||
| # | Complaint | Fix | Effort |
|
||||
|---|-----------|-----|--------|
|
||||
| 1 | Table header text เบียด/ไม่เต็มพื้นที่ | Full-width header, better line-height/whitespace | CSS |
|
||||
| 2 | "รวม" column ไม่ชัดว่าคืออะไร + เรียงไม่ได้ | Rename → **"คะแนนรวม"**; make it sortable; add tooltip "60% ธีม + 40% พื้นฐาน" | FE + sort |
|
||||
| 3 | ธีม กับ 05/บันทึกการวิเคราะห์ ซ้ำ | **Remove 05/บันทึกการวิเคราะห์** (theme panel already has thesis) | FE remove |
|
||||
| 4 | section-kicker ต้อง format เดียวกัน ("07 /" ออก) | Remove numeric prefixes → plain Thai kickers | FE |
|
||||
| 5 | Theme ต้องการคำอธิบาย/บรรยายผลวิเคราะห์ | Long-form narrative per theme (auto <th> hon 3) | BE + FE |
|
||||
| 6 | ตารางหุ้น ต้องมี column theme | Add "ธีม" column showing which theme(s) each symbol belongs to | FE |
|
||||
| 7 | แต่ละหุ้นต้องมี หน้า detail: คลิก แล้วเห็นการวิเคราะห์ทุกอย่าง ต่อหุ้น (ข้อมูล→น้ำหนัก→ที่มาคะแนนรวม) | New per-symbol endpoint + detail view (modal or page) | BE + FE |
|
||||
|
||||
---
|
||||
|
||||
## Backend tasks
|
||||
|
||||
### Task B7: Per-symbol analysis breakdown endpoint
|
||||
- `GET /api/v1/symbols/<symbol>` returns for one symbol:
|
||||
- `symbol`, `company_name`
|
||||
- `themes`: which theme(s) this symbol belongs to (from THEME_SYMBOLS) + each theme's factor reading + surprise contribution
|
||||
- `fundamentals`: PE, EPS, EPS YoY, dividend_yield, PBV, ROE (Siamchart)
|
||||
- `score_breakdown`:
|
||||
- `theme_score` = mean of the theme surprise scores covering this symbol (with per-theme lines)
|
||||
- `siamchart_score` = growth + yield*2, z-scored
|
||||
- `combined` = 0.6*theme_score + 0.4*siamchart_score (with the actual numbers, not just final)
|
||||
- `price`: latest close + date
|
||||
- Deterministic, reuses `themes.combine_score` logic per-symbol; NO new magic.
|
||||
|
||||
### Task B5: Long-form per-theme narrative
|
||||
- Add a `narrative` field to each theme read: 2-4 sentence Thai explanation of what the factor means, whether it's positive/negative, and what it implies for stocks in that group. Built deterministically from the factor reading + surprise sign (not hardcoded, not an LLM).
|
||||
|
||||
## Frontend tasks
|
||||
|
||||
### Task F1: Table header full-width (complaint 1)
|
||||
- Give `.stock-panel`/`.factor-table` a max-width-safe full-width layout, better padding/line-height so the subtitle + controls breathe.
|
||||
|
||||
### Task F2: Rename + sort "รวม" (complaint 2)
|
||||
- Header "รวม" → **"คะแนนรวม"** with a small `60/40` hint; add `combined` to the sortable keys (already have `setSort('combined')` — wire it to actually sort by combined score desc default).
|
||||
|
||||
### Task F3: Remove 05/บันทึกการวิเคราะห์ (complaint 3)
|
||||
- Delete the bottom-grid thesis section (superseded by theme panel narratives).
|
||||
|
||||
### Task F4: Uniform section-kicker (complaint 4)
|
||||
- Strip "07 /", "05 /" etc. → plain Thai kickers ("การจำลองการลงทุน", "ธีม", "ที่มาของข้อมูล", "ตารางหุ้น").
|
||||
|
||||
### Task F6: Add Theme column (complaint 6)
|
||||
- In stock table add a "ธีม" column: for each symbol show compact chips of the theme(s) it belongs to (e.g. "โรงกลั่น", "—" if none). Reuse `boardBySymbol` / THEME lookup.
|
||||
|
||||
### Task F7: Per-symbol detail view (complaint 7)
|
||||
- Clicking a symbol row opens a detail panel/modal showing the full `/api/v1/symbols/<symbol>` breakdown: themes + fundamentals + score components + how combined was derived (60/40). This is the transparent-analysis view.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
- Backend: full suite passes; `GET /api/v1/symbols/AOT` returns full breakdown with correct combined derivation.
|
||||
- Frontend: `npm run build`; browser shows full-width table, "คะแนนรวม" sortable, theme column, no 05 section, uniform kickers, per-symbol modal works & shows derivation.
|
||||
- Screenshot evidence; push after user OK.
|
||||
|
||||
## Open questions
|
||||
None blocking — methodology is deterministic and already implemented in `themes.combine_score`; this plan surfaces it transparently rather than changing it.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Dashboard Rebuild — Multi-theme + Real Thai Data
|
||||
|
||||
> **For Hermes:** Execute task-by-task, commit per task, verify with backend tests + browser. Plan mode — no code until user approves.
|
||||
|
||||
**Goal:** Rebuild the SET50 dashboard so it reflects REAL multi-theme Thai data (tourism + auto + energy + NPL), not the current tourism-only fixture, per the user's 10-point review.
|
||||
|
||||
**Architecture:** Replace the single-vertical `tourism_result`-driven dashboard with a multi-theme real-data source; add per-symbol theme exposure (multi-source factors PER theme); update all UI sections to read real data, Thai-only, and remove the 5 obsolete KPI cards + paper ledger.
|
||||
|
||||
---
|
||||
|
||||
## The 10 user complaints → fix
|
||||
|
||||
| # | Complaint | Root cause | Fix |
|
||||
|---|-----------|-----------|-----|
|
||||
| 1 | Theme Surprise ไม่เข้าใจ, ลบได้ | KPI card from tourism-only | **Remove** card |
|
||||
| 2 | สัญญาณที่ใช้งาน: ซื้อ 7 แต่โชว์ 8 | total=8 (incl neutral) mislabeled | Show long/short/neutral correctly, Thai |
|
||||
| 3 | สมุดบันทึก paper มีทำไม — ไม่มี paper trade แล้ว | legacy section | **Remove** (simulation replaces it) |
|
||||
| 4 | คุณภาพข้อมูล — ข้อมูลจริงรึยัง? | `TOURISM_SOURCE=fixture` default | **Switch to real BOT data**; show real source+as_of |
|
||||
| 5 | ประตู backtest (blocked) เอาออก | legacy gate | **Remove** card (research is exploratory/forward) |
|
||||
| 6 | แต่ละธีมควรดูหลายแหล่ง (ประชากร ฯลฯ) | 1 source/theme | **Add multi-source factors per theme** (see Data plan) |
|
||||
| 7 | ตารางหุ้น + ตารางสัญญาณ ควรรวมกัน | split | **Merge** into one combined table |
|
||||
| 8 | Theme surprise ไม่ครบ 3 ธีม | tourism-only | **3 theme surprise** (auto, energy, tourism) |
|
||||
| 9 | ที่มาข้อมูล ควรเป็นตาราง (แหล่ง+เวลา) | prose | **Table** of all sources + last fetched |
|
||||
| 10 | บันทึกการวิเคราะห์ มีแค่ 1 ธีม | tourism thesis | **Per-theme thesis** (3) + multi-source note |
|
||||
|
||||
**Overall:** "ข้อมูลทั้งหมดตอนนี้ยังไม่ได้ใช้ข้อมูลจริงใช่ไหม?" → ใช่, dashboard ใช้ `tourism_result` (fixture default). Rebuild to real.
|
||||
|
||||
---
|
||||
|
||||
## User-confirmed decisions (2026-08-25)
|
||||
1. **Real data ONLY** — switch to BOT real (no fixture fallback); dashboard fails if no real data.
|
||||
2. **Theme surprise = z-score** of each theme's headline factor (uniform across all themes).
|
||||
3. **Multi-source per theme (A+B)**: wire existing multi-source (auto = volume + NPL + production + export); AND add broader macro (population/GDP/inflation) — **found BOT SDDS page (bot.or.th/en/statistics/sdds.html) is server-rendered and scrapable WITHOUT auth**: provides GDP (Q2/2026 4,793.5bn), Private Consumption +4.9%, Private Investment +18.1%, Manufacturing −3.1%, Population 70,472k, Headline Inflation 1.95%, Unemployment 0.93%. This is the macro backdrop layer (complaint #6).
|
||||
|
||||
## Macro backdrop factor (new) — BOT SDDS collector
|
||||
- Create `backend/app/macro_thai.py` — scrape BOT SDDS HTML for: private consumption index %YoY, private investment %YoY, mfg production %YoY, population (thousands), headline inflation, core inflation, unemployment. Real Thai macro backdrop.
|
||||
- Wire into a "macro" theme/context read + into the sources table.
|
||||
|
||||
## Backend task list
|
||||
|
||||
### Task A: Real multi-theme dashboard source module
|
||||
- Create `backend/app/dashboard.py` — assembles REAL data from all collectors:
|
||||
- tourism (BOT real, via collector), auto_credit (Trading Econ + **auto_npl** BOT), energy (TOP), siamchart fundamentals, prices
|
||||
- Returns `{themes: [{id, name, frequency, read_* , surprise}], board: [{symbol, combined_score, dividend, ...}], sources: [{from, source, as_of, fetched_at}], thesis_per_theme: {...}}`
|
||||
- **Multi-source per theme (complaint #6):** registry mapping theme → list of factor sources (not 1). E.g. auto theme = {new_car_sales_yoy (TradingEcon), auto_npl_pct (BOT), vehicle_production, auto_exports}. Add aggregatable fields.
|
||||
- **3 theme surprise (#8):** compute surprise per theme (z-score of the theme's headline factor), not just tourism.
|
||||
|
||||
### Task B: Switch create_app to real data
|
||||
- `backend/app/__init__.py`: default `TOURISM_SOURCE` → `bot` (real), or better: build `multi_dashboard` from collectors at startup, keep tourism as one theme. `_load_default_snapshot` no longer the dashboard root.
|
||||
- `dashboard/summary`, `/signals`, `/factors` endpoints → read from `dashboard.py` real assembly, not `tourism_result`.
|
||||
|
||||
### Task C: Sources table endpoint
|
||||
- `dashboard.py` collects `sources` list: each {from (e.g. "TradingEconomics"), source, url, as_of, fetched_at, frequency}. Expose in `/api/v1/themes` + summary.
|
||||
|
||||
### Task D: Per-theme thesis (#10)
|
||||
- `dashboard.py` builds a short deterministic thesis per theme from that theme's factor readings (not a hardcoded tourism sentence).
|
||||
|
||||
## Frontend task list
|
||||
|
||||
### Task F: Remove 5 obsolete items (#1,#3,#5)
|
||||
- Delete cards: Theme Surprise (#1), คุณภาพข้อมูล (#4 replaced by honest sources), ประตู backtest (#5), สมุดบันทึก paper (#3) → card 2 (สัญญาณ) only + 3-theme panel.
|
||||
- Remove paper-ledger UI + backtest-gate UI blocks (already partially removed paper ledger; remove backtest KPI).
|
||||
|
||||
### Task G: Correct signal count (#2)
|
||||
- Show `long` / `short` / `neutral` as separate Thai chips; value = long only, with "+X ซื้อ −Y ขาย" clearly, no off-by-one.
|
||||
|
||||
### Task H: Merge stock + signal tables (#7)
|
||||
- Single combined table: [สีสัญญาณ, symbol, theme(s), combined_score, dividend, PE, EPS YoY, Yield, P/BV, ROE] — one table, sortable, dividend filter.
|
||||
|
||||
### Task I: Sources table UI (#9)
|
||||
- Render `sources` as a `<table>`: ตัวจาก, แหล่ง, ข้อมูล, as_of, fetched_at, frequency.
|
||||
|
||||
### Task J: 3-theme surprise + thesis (#8,#10)
|
||||
- Theme panel shows 3 themes, each with its own surprise + read + thesis.
|
||||
|
||||
### Task K: Thai-only consistency + visual verify
|
||||
- Audit all remaining English; keep only technical terms. Browser verify at 1440/1024/768. Screenshot for user.
|
||||
|
||||
---
|
||||
|
||||
## Multi-source factor registry (complaint #6) — where to add data
|
||||
For "แต่ละธีมดูข้อมูลรอบด้านของประเทศ" — extend themes to multiple Thai sources:
|
||||
- **Auto**: new_car_sales_yoy (TradingEcon) + auto_npl_pct (BOT) + vehicle_production + auto_exports (TradingEcon) [already in collector]
|
||||
- **Energy**: TOP net_profit/ebitda + Thai retail fuel/EPPO + (optional) import/refining
|
||||
- **Tourism**: BOT tourism surprise + arrivals + hotel occupancy
|
||||
- **Macro backdrop (new)**: Thai population / GDP (BOT API — needs auth, mark deferred) per user's "จำนวนประชากร"
|
||||
|
||||
**Note:** Full multi-source (population/GDP via BOT portal API) requires API key registration — mark as a separate later task; this plan wires the already-available multi-source readings per theme.
|
||||
|
||||
## Verification
|
||||
- Backend: full suite passes; live `/api/v1/themes` shows 3 themes with real Thai readings + sources table + per-theme thesis; no fixture.
|
||||
- Frontend: `npm run build`; browser shows combined single table, 3-theme surprise, sources table, no removed cards; Thai-only.
|
||||
- Screenshot evidence for user review before push (push gate).
|
||||
|
||||
## Open questions / risks
|
||||
1. **Real-vs-fixture default:** Should dashboard run on BOT real data by default (needs a fetch), or keep a cache so it works offline? → Recommend: real BOT via daily cache, fallback to last-good.
|
||||
2. **Population/GDP macro**: needs BOT portal API key (user registers). Defer unless user provides key.
|
||||
3. **Theme surprise definition**: need concrete per-theme surprise metric — z-score on each theme's headline factor (auto YoY, energy net profit trend, tourism surprise). Confirm with user if ambiguous.
|
||||
4. **Push gate**: after visual verify, user approves before push (per repo convention).
|
||||
95
.hermes/plans/2026-08-25_full-app-consistency-refactor.md
Normal file
95
.hermes/plans/2026-08-25_full-app-consistency-refactor.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# Full-App Consistency Refactor — single source of truth + per-symbol selection
|
||||
|
||||
> **For Hermes:** Execute after user approval. Fixes two debts at once:
|
||||
> (a) theme mapping/data duplicated & diverged across backend/frontend;
|
||||
> (b) theme selection is FLAT — every symbol in a theme gets the same surprise
|
||||
> score, so there's no real "เลือกหุ้น" (which name benefits most within theme).
|
||||
|
||||
**Root problems:**
|
||||
1. Theme registry + labels live in 3 places that diverge (`themes.py` correct,
|
||||
`dashboard.py` maco-proxy, `__init__.py` old `/api/v1/themes` still 3 themes,
|
||||
`App.vue` hardcoded `THEME_BY_SYMBOL`/`themeLabelById`).
|
||||
2. **Selection is not real:** `_build_board` assigns the SAME `surprise` to every
|
||||
symbol in a theme (flat). A strong BBL vs weak TTB get identical theme score.
|
||||
The user wants each theme to *analyze and pick* — i.e. differentiate per-symbol
|
||||
quality/benefit within the theme.
|
||||
|
||||
---
|
||||
|
||||
## What "theme analyzes & selects stocks" means (per-theme pipeline)
|
||||
For EVERY theme, a symbol in that theme gets:
|
||||
theme_score_for_symbol =
|
||||
theme_surprise (macro/theme tailwind, common to all in theme)
|
||||
× firm_quality (per-symbol relative strength within the theme)
|
||||
|
||||
`firm_quality` is deterministic, derived from the symbol's own fundamentals
|
||||
relative to its THEME cohort (not the whole market), e.g.:
|
||||
- banks: ROE, NIM proxy (ROE/EPS), loan book health (EPS growth)
|
||||
- energy: PTT/TOP margins via ROE + yield
|
||||
- retail: margin/EPS growth
|
||||
- generic: ROE vs theme median, EPS growth vs theme median, dividend floor
|
||||
This way within a hot theme the *better-run* names rank higher → real stock picking.
|
||||
|
||||
We generalize the existing 60/40 so theme_score is no longer flat:
|
||||
combined = 0.6 * (theme_surprise × quality) + 0.4 * siamchart_score
|
||||
|
||||
---
|
||||
|
||||
## Refactor plan
|
||||
|
||||
### Backend
|
||||
**Task B0 — per-symbol theme quality (selection):**
|
||||
- Add `themes.quality_within_theme(symbol, theme_id, factor_view) -> float` that
|
||||
computes a symbol's relative quality within its theme cohort (ROE/EPS-growth vs
|
||||
theme median, dividend floor). deterministic, unit-testable.
|
||||
- In `dashboard._build_board`, when applying a theme surprise to a symbol, multiply
|
||||
by `quality_within_theme` so symbols in the same theme differentiate.
|
||||
- Expose `quality` + `theme_score breakdown` in `/api/v1/symbols/<x>` so it's transparent.
|
||||
|
||||
**Task B1 — single source of truth:**
|
||||
- `themes.py` add `theme_of(symbol)`, `label_of(theme)`, `quality_within_theme`.
|
||||
- `dashboard.build()` returns `symbol_theme_map` + board rows carry per-symbol `themes` + `theme_score`.
|
||||
|
||||
**Task B3 — remove old `/api/v1/themes` (3 themes, dup logic):**
|
||||
- Delegate to `RealDashboard.build()` so `/api/v1/themes` == `/api/v1/dashboard` theme set (13, same labels/surprises). Remove dup auto/energy sign code.
|
||||
|
||||
**Task B4 — single fetch path:**
|
||||
- RealDashboard.build() is the one place that fetches auto/energy/macro; endpoints + scheduler consume it (no dupe fetch).
|
||||
|
||||
### Frontend
|
||||
**Task F1 — delete hardcoded `THEME_BY_SYMBOL`/`themeLabelById`:**
|
||||
- theme column + modal come from `/api/v1/dashboard` (board[].themes, themes[].label_th) + `/api/v1/symbols/<x>` breakdown. No local map.
|
||||
|
||||
**Task F2 — modal shows per-symbol selection logic:**
|
||||
- Add to symbol view: the `quality` within each theme + how theme surprise × quality = theme score, so stock-picking is explainable.
|
||||
|
||||
### Tests
|
||||
- `test_themes`: quality_within_theme differentiates (strong vs weak name), unit.
|
||||
- `test_dashboard`/`test_api`: `/api/v1/themes` == `/api/v1/dashboard` theme set; board rows have per-symbol themes; two symbols in same theme get different theme_score when quality differs.
|
||||
|
||||
### Files
|
||||
- `backend/app/themes.py`, `backend/app/dashboard.py`, `backend/app/__init__.py`
|
||||
- `frontend/src/App.vue`
|
||||
- `backend/tests/test_themes.py`, `test_dashboard.py`, `test_api.py`
|
||||
|
||||
## Acceptance (definition of done)
|
||||
1. Two banks (BBL strong, TTB weak) get DIFFERENT theme score within banks theme — shows real stock selection, not flat.
|
||||
2. Editing themes.py propagates to `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols`, frontend — no frontend change needed.
|
||||
3. `/api/v1/themes` == `/api/v1/dashboard` theme set (consistency test).
|
||||
4. Modal explains: theme surprise × firm quality = theme score per symbol.
|
||||
5. npm run build + full backend suite pass; browser shows BBL>TTB theme score when quality differs.
|
||||
|
||||
### Files
|
||||
- `backend/app/themes.py` (add theme_of/label_of helpers)
|
||||
- `backend/app/dashboard.py` (symbol_theme_map in build; board row.themes)
|
||||
- `backend/app/__init__.py` (themes endpoint → delegate; de-dup; board.themes)
|
||||
- `frontend/src/App.vue` (remove hardcode maps; derive from API)
|
||||
- `backend/tests/test_dashboard.py`, `backend/tests/test_api.py` (consistency tests)
|
||||
|
||||
## Acceptance (definition of done)
|
||||
1. Changing a theme in `themes.py` propagates to `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols/<x>`, and the frontend column/modal — with NO change to frontend code.
|
||||
2. No duplicate theme mapping in frontend.
|
||||
3. `npm run build` + full backend suite pass; browser shows BBL=ธนาคาร from API.
|
||||
|
||||
## Open question / tradeoff
|
||||
- One cache/fetch per data refresh vs per-day: keep single `RealDashboard.build()` as the one fetch path (scheduler + endpoints both call it) so data is consistent within a cycle. Accept minor latency first call.
|
||||
148
.hermes/plans/2026-08-26_factor-engine-architecture.md
Normal file
148
.hermes/plans/2026-08-26_factor-engine-architecture.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Architecture Plan — Declarative Factor/Theme Analysis Engine
|
||||
|
||||
> Status: DRAFT — pending user approval (decision gate G0).
|
||||
> Author: Macky. Date: 2026-08-26.
|
||||
|
||||
## Goal
|
||||
Rebuild the scoring system so that **adding/removing a data source or factor is a config change, not an edit to scoring functions**. Every theme gets a full analyze-and-select pipeline built from registered, declarative factors. This makes the platform easy to extend as the user keeps finding new Thai data sources.
|
||||
|
||||
## Why this is needed (current debt)
|
||||
Today the analysis is **hardcoded**:
|
||||
- `dashboard._theme_surprises()` — per-theme surprise logic written inline (tourism special, auto NPL composite, energy TOP, then macro-proxy branches).
|
||||
- `dashboard._theme_narrative()` — per-theme narrative strings hardcoded.
|
||||
- `dashboard.build()` — proxy_reads dict hardcoded.
|
||||
- `App.vue` — `THEME_BY_SYMBOL` + `themeLabelById` hardcoded (duplicate of backend).
|
||||
- `__init__.py` — old `/api/v1/themes` still builds only 3 themes.
|
||||
|
||||
Every new source = editing functions and strings in 3+ places → exactly the "จุดนึงเสร็จ จุดอื่นไม่ตาม" problem.
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐
|
||||
│ FACTOR REGISTRY (config) │
|
||||
┌──────────────────▶ (declarative, data-driven) │
|
||||
│ └──────────────────────────────┘
|
||||
│ reads reads
|
||||
▼ ▼
|
||||
┌───────────────┐ fetch/parse ┌──────────────────────┐
|
||||
│ COLLECTORS │─────────────▶ │ FACTOR PIPELINE │
|
||||
│ (per source) │ raw observable │ surprise / z-score / │
|
||||
└───────────────┘ │ frequency alignment │
|
||||
▲ └──────────────────────┘
|
||||
│ registry.fetcher │ per-factor value
|
||||
└──┘ ▼
|
||||
┌──────────────────────┐
|
||||
│ THEME SCORER │
|
||||
│ factors ✕ weights │
|
||||
│ ✕ firm_quality │
|
||||
└──────────────────────┘
|
||||
│ theme_score
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ COMBINE (60/40) + │
|
||||
│ SELECT / board │
|
||||
└──────────────────────┘
|
||||
│
|
||||
▼
|
||||
/api/v1/themes, /api/v1/dashboard,
|
||||
/api/v1/symbols/<x>, frontend (read-only)
|
||||
```
|
||||
|
||||
## Key Decisions
|
||||
| Decision | Choice | Rationale | Alternatives |
|
||||
|---|---|---|---|
|
||||
| Factor = declarative unit | `FACTORS` registry: dict of `{key, source, fetch_fn, frequency, sign, weight, periods}` | add source = add one dict entry | hardcoded functions (rejected) |
|
||||
| Theme = composition of factors | `THEMES` registry: `{id, label_th, symbol_map, factors:[{factor_key, weight}]}` | theme reuses factors; new theme = add entry | inline per-theme logic (rejected) |
|
||||
| Source of truth | `themes.py`/`factors.py` registries (backend) | one place; frontend reads via API | App.vue hardcode (remove) |
|
||||
| Per-symbol selection | `quality_within_theme(symbol, theme, factor_view)` | differentiates strong vs weak within a theme | flat surprise (rejected) |
|
||||
| Frequency alignment | each factor carries `frequency`; scorer aligns by period (as-of semantics), never mixes timestamps | user's explicit requirement | naive mixing (rejected) |
|
||||
|
||||
## Decision Gates (one per milestone → user approves before code)
|
||||
|
||||
### G0 — Approve declarative registry design
|
||||
**Acceptance:** user approves FACTORS + THEMES registry schema & that adding a source = one config entry.
|
||||
**Exit:** proceed to B0.
|
||||
|
||||
### G1 — Per-symbol selection works
|
||||
**Acceptance:** two banks with different fundamentals get **different** theme_score within banks theme (BBL strong > TTB weak), unit-tested.
|
||||
**Exit:** proceed to frontend.
|
||||
|
||||
### G2 — Frontend no longer hardcodes themes
|
||||
**Acceptance:** edit `themes.py` (add theme/factor) → `/api/v1/themes`, `/api/v1/dashboard`, `/api/v1/symbols/<x>`, and the board column/modal all pick it up with zero frontend change.
|
||||
**Exit:** done.
|
||||
|
||||
## Data Model
|
||||
**FACTORS** (in `backend/app/factors.py`):
|
||||
```python
|
||||
FACTORS: dict[str, dict] = {
|
||||
"tourism_arrivals_yoy": {
|
||||
"name_th": "นักท่องเที่ยว",
|
||||
"source": "BOT", "frequency": "monthly",
|
||||
"fetch": "bot_tourism", # module name; returns dict with numeric value
|
||||
"value_key": "arrivals_yoy", # key of the numeric value
|
||||
"sign": 1, # +1 = higher is bullish for theme
|
||||
"weight": 1.0, # (themes can override)
|
||||
},
|
||||
"auto_sales_yoy": {"source":"TradingEconomics","frequency":"monthly","fetch":"auto_credit","value_key":"new_car_sales_yoy","sign":1},
|
||||
"auto_npl": {"source":"BOT","frequency":"quarterly","fetch":"auto_npl","value_key":"pct_of_npls","sign":-1},
|
||||
"energy_top_netmarg": {"source":"TOP","frequency":"quarterly","fetch":"energy_thai","value_key":"net_margin","sign":1},
|
||||
"macro_consumption": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"private_consumption_yoy","sign":1},
|
||||
"macro_investment": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"private_investment_yoy","sign":1},
|
||||
"macro_inflation": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"headline_inflation_yoy","sign":-1},
|
||||
"macro_mfg": {"source":"BOT","frequency":"monthly","fetch":"macro_thai","value_key":"manufacturing_yoy","sign":1},
|
||||
}
|
||||
```
|
||||
|
||||
**THEMES** (in `backend/app/themes.py` — keeps symbol_map + label):
|
||||
```python
|
||||
THEMES: dict[str, dict] = {
|
||||
"auto_credit": {
|
||||
"label_th": "รถยนต์/สินเชื่อ", "frequency": "monthly",
|
||||
"factors": [{"key":"auto_sales_yoy","weight":1.0},{"key":"auto_npl","weight":-0.5}],
|
||||
"symbols": {...},
|
||||
},
|
||||
"banks": {
|
||||
"label_th": "ธนาคาร", "frequency": "quarterly",
|
||||
"factors": [{"key":"macro_investment","weight":1.0},{"key":"macro_inflation","weight":-0.3}],
|
||||
"symbols": {...},
|
||||
},
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
**surprise per theme** = weighted, sign-aware blend of its factors (each factor z-scored / normalized per its frequency cohort → frequency-aligned, no naive mixing).
|
||||
|
||||
**theme_score per symbol** = theme_surprise × `quality_within_theme(symbol, theme, factor_view)`.
|
||||
|
||||
**combined** = 0.6 × theme_score + 0.4 × siamchart_score.
|
||||
|
||||
## Module Boundaries
|
||||
```
|
||||
factors.py — FACTORS registry + normalize/z-score helper (single source)
|
||||
themes.py — THEMES registry (symbol_map, labels, factor refs), quality_within_theme, theme_of, label_of, combine_score
|
||||
dashboard.py — orchestrator: fetch registered factors once, compute surprises, build board/macro/sources (no per-theme hardcode)
|
||||
collectors/ — bot_tourism, auto_credit, auto_npl, energy_thai, macro_thai (return dict, registered in FACTORS.fetch)
|
||||
__init__.py — endpoints only; /api/v1/themes delegates to dashboard.build(); /api/v1/symbols uses themes helpers
|
||||
App.vue — PURE read-only; theme column/labels/board from /api/v1/dashboard; NO hardcoded theme maps
|
||||
```
|
||||
|
||||
## Explicit Non-Goals (v1)
|
||||
- No live trading / MT5 execution (paper/backtest only, unchanged).
|
||||
- No new collectors yet — this refactor *enables* easy addition; sources are added as follow-up (user will name them).
|
||||
- No LLM involvement in scoring (deterministic only).
|
||||
- No changing the 60/40 or 50/20/30 business rules — only making them data-driven.
|
||||
|
||||
## Open Questions (for G0)
|
||||
1. Should factor weights live per-theme (override) or global? → proposal: per-theme override with global default (flexible).
|
||||
2. Do we keep the current 3 real collectors as the first factors registered (yes), and new sources added later by the user?
|
||||
|
||||
---
|
||||
|
||||
## Acceptance (definition of done — all gates pass)
|
||||
1. Adding a new factor = 1 dict entry in `FACTORS` + (optionally) a theme factor line — no scoring-function edit.
|
||||
2. `/api/v1/themes` == `/api/v1/dashboard` theme set (13, same labels/surprises) — consistency test.
|
||||
3. BBL vs TTB get different banks theme_score by quality — selection proof test.
|
||||
4. Frontend has NO hardcoded theme maps; column + modal label from API.
|
||||
5. Modal shows `theme surprise × firm quality = theme score` per symbol.
|
||||
6. `npm run build` + full backend suite pass; browser BBL > TTB when quality differs.
|
||||
88
.hermes/plans/2026-08-26_research-sources-backtest.md
Normal file
88
.hermes/plans/2026-08-26_research-sources-backtest.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Research + Data-Source + Simulation Backtest — plan (5-point)
|
||||
|
||||
> For Hermes: execute after user approval. This covers:
|
||||
> 1. Literature review → formula improvements
|
||||
> 2. New data sources per theme
|
||||
> 3. Frontend sources table = NOT mock, auto from backend FACTORS registry
|
||||
> 4. Simulation: real backtest engine w/ custom start/end date + P&L
|
||||
> 5. Sources table shows next-update column
|
||||
|
||||
## Findings (facts, verified)
|
||||
|
||||
**Sources table is hardcoded (point 3 true):** `dashboard._build_sources` returns a
|
||||
hardcoded 5-row list (TradingEconomics, BOT NPL, Thai Oil, BOT Economy, Tourism).
|
||||
It does NOT come from the FACTORS registry, so adding a factor does NOT add a source
|
||||
row automatically. Must wire sources from FACTORS registry (single source of truth).
|
||||
|
||||
**Simulation is NOT a real backtest (point 4 true):** `simulation.allocate_capital`
|
||||
allocates once at latest price. Both "backtest" and "forward" modes just differ in
|
||||
whether an MT5 order is sent. There is no start/end date, no rebalancing over time,
|
||||
no performance/P&L, and results are not persisted (vanish on refresh). This matches
|
||||
the user's confusion exactly — the two modes don't do what the labels imply.
|
||||
|
||||
**5 data sources today** (real): TradingEconomics (auto sales) · BOT NPL · Thai Oil
|
||||
(TOP quarterly) · BOT Thai Economy (macro) · BOT Tourism — all real, but hardcoded
|
||||
in the sources table.
|
||||
|
||||
---
|
||||
|
||||
## Work items
|
||||
|
||||
### 1. Literature / methodology research → formula improvements
|
||||
- Research: factor-based Thai-equity scoring, alternative-data alpha, market-regime
|
||||
timing, cross-frequency factor combination (mixed cadence). Deliver: a short
|
||||
recommendation doc with concrete refinements to the current formula
|
||||
(60/40, quality_within_theme, regime gate), grounded in citations.
|
||||
- Output: `docs/methodology-research.md`; proposals applied only after user OK.
|
||||
|
||||
### 2. Additional data sources per theme
|
||||
- Research & (where cheap/feasible) wire one new honest Thai source per theme:
|
||||
- banks → BOT commercial-bank loan growth / NPL (real, report already seen)
|
||||
- retail → consumer-comfort/retail index
|
||||
- utilities → BOT electricity demand proxy
|
||||
- telecom → subscriber data (if public)
|
||||
- property → BOT housing / transfer volume
|
||||
- Add as FACTORS registry entries (declarative — that's the whole point). Sources
|
||||
that need auth (BOT Portal API) are gated by user-provided key.
|
||||
|
||||
### 3. Sources table auto-derived (not mock, not hardcoded)
|
||||
- `dashboard._build_sources` → derive from FACTORS registry: iterate FACTORS,
|
||||
group by source module, emit {แหล่ง, ขอบเขต, data_value, as_of, fetched_at}.
|
||||
- Frontend sources table consumes `/api/v1/dashboard.sources` — so adding a FACTOR
|
||||
auto-appends its source row. No hardcoded list in backend or frontend.
|
||||
|
||||
### 4. Real backtest engine (replaces forward-only allocation)
|
||||
- New `backend/app/backtest.py`: takes {start_date, end_date, capital, bucket_weights,
|
||||
rebalance} and RUNS the allocation across time using the Yahoo price series:
|
||||
- at each rebalance date, recompute combined scores (theme+firm+quarterlies) on
|
||||
*point-in-time* basis (no future leak), allocate 50/20/30, track holdings.
|
||||
- mark-to-market daily; accrue dividends; end → report:
|
||||
`{capital, invested, final_value, price_pnl, dividend_income, net_return, win_rate, trades}`
|
||||
- Backtest section in UI: pick start/end date, run, see performance summary.
|
||||
- Persist run results (keep history) so refresh doesn't lose them.
|
||||
|
||||
### 5. Next-update column
|
||||
- Each source row gains `next_update_after` (from frequency → next scheduled
|
||||
refresh time). Show "อัปเดตอีกครั้ง ~X" in the sources table.
|
||||
|
||||
## Files
|
||||
- `docs/methodology-research.md` (new, point 1)
|
||||
- `backend/app/factors.py` (more factors; sources derive here)
|
||||
- `backend/app/dashboard.py` (sources FROM registry + next_update)
|
||||
- `backend/app/backtest.py` (new engine, point 4)
|
||||
- `backend/app/__init__.py` (backtest endpoint, sources wiring)
|
||||
- `frontend/src/App.vue` (backtest section + next-update col + sources from API)
|
||||
- tests
|
||||
|
||||
## Acceptance
|
||||
1. Adding a FACTOR to `factors.py` auto-appends a sources-table row (no manual edit).
|
||||
2. `/api/v1/dashboard/sources` has 5+ rows, all real, each with next_update_after.
|
||||
3. Backtest with {start,end} returns a full P&L report (price + dividend + net), and
|
||||
results persist (not lost on refresh).
|
||||
4. Literature doc recommends ≥1 concrete formula refinement, cited.
|
||||
5. npm build + full suite pass; browser shows backtest section + next-update column.
|
||||
|
||||
## Open decisions for user
|
||||
- D1: rebalancing frequency in backtest (monthly / quarterly / weekly)?
|
||||
- D2: do BOT Portal API sources require your API key, or only auth-free ones for now?
|
||||
- D3: backtest default range (last 12 months / last N quarters)?
|
||||
396
.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md
Normal file
396
.hermes/plans/2026-08-28_091000-event-driven-pit-backtest.md
Normal file
@@ -0,0 +1,396 @@
|
||||
# Event-Driven PIT Backtest Implementation Plan
|
||||
|
||||
> **For Hermes:** Execute task-by-task. Use TDD for accounting/event logic, keep one task in progress, commit by phase, and run `requesting-code-review` before delivery.
|
||||
|
||||
**Goal:** Replace the current calendar-rebalance/backtest proxy with a strict point-in-time, event-driven portfolio simulation that reinvests sale proceeds and paid dividends, respects 100-share lots, records realized/unrealized P&L, and reconciles final equity exactly.
|
||||
|
||||
**Architecture:** Build a deterministic event calendar from data releases, execution dates, dividend entitlement/payment dates, and the end valuation date. Feed those events through a portfolio ledger that owns cash, holdings, average cost, dividend receivables, orders, realized P&L, and costs. The API derives readiness/default dates from actual PIT coverage and blocks rather than silently falling back to current data.
|
||||
|
||||
**Tech Stack:** Python 3.11, Flask, filesystem PIT stores, Vue/Vite, unittest/pytest.
|
||||
|
||||
---
|
||||
|
||||
## Confirmed Decisions
|
||||
|
||||
| Decision | Chosen default | Rationale |
|
||||
|---|---|---|
|
||||
| Backtest readiness | Strict PIT coverage for every required scoring input; otherwise block and report missing coverage | Prevents mixed-PIT results from looking valid |
|
||||
| Signal execution | Freeze signal on release date D; execute at next available trading-day close after D | Avoids same-day close look-ahead |
|
||||
| Dividend timing | Entitlement at ex-date; cash becomes available exactly 30 calendar days after ex-date | User-approved deterministic timing assumption; expose it as `ex_date_plus_30d` rather than claiming an observed payment date |
|
||||
| Cost basis | Average cost | Simple, auditable, common portfolio accounting |
|
||||
| Fees/taxes | All-in fee 0.3% of notional on every buy and every sell; do not add VAT/tax again | Explicit user decision; deterministic and auditable |
|
||||
| Same-day releases | Coalesce all releases known by the execution cutoff into one frozen signal and one rebalance | Avoids churn from multiple events on the same day |
|
||||
| End date on holiday/weekend | Value at latest trading close on or before end date | Deterministic and realistic |
|
||||
|
||||
These decisions were confirmed by the user and are implementation requirements.
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Matrix
|
||||
|
||||
| Requirement | Current behavior | Gap |
|
||||
|---|---|---|
|
||||
| Default start = first date advice is genuinely available | API/UI hardcode `2024-06-01` | No readiness calculation; current disk has no factor vintages or Siamchart vintage manifest |
|
||||
| Default end = yesterday | API/UI hardcode `2026-06-01` | On 2026-08-28 +07, expected default is 2026-08-27 |
|
||||
| Recompute only when new information arrives | `_rebalance_dates()` emits monthly/quarterly dates (`backend/app/backtest.py:103-125`) | Calendar-driven, not release/event-driven |
|
||||
| Strict PIT | Optional `score_fn`; current data fallback exists | Current local data has 0 factor-vintage files and 0 Siamchart vintage snapshots, so no strict PIT-ready start exists |
|
||||
| Execute after signal is known | `_latest_close(..., d)` uses close on rebalance date | Can use the same close that occurred before/while data was released |
|
||||
| Dividend entitlement and cash timing | `credit_dividends()` is called only at end, using final holdings (`backtest.py:253-268`) | Wrong holdings-at-ex-date, no mid-run cash credit, no reinvestment, historic dividends can be overcounted |
|
||||
| Sale P&L | Sell proceeds update cash only (`backtest.py:232-241`) | No cost basis, realized P&L, fees, or per-trade ledger |
|
||||
| Reinvest gains/dividends | Sale cash can fund buys in the same rebalance; dividends are added only after all rebalances | Dividends cannot fund later buys; profit/cash trace is absent |
|
||||
| 100-share lots | `allocate_capital()` floors to 100 shares | Already present; must preserve during target reconciliation |
|
||||
| Final report | final holdings, total value, combined price P&L, dividend total | Must split realized/unrealized P&L, cash, receivables, costs, trades, and prove reconciliation |
|
||||
| Current data coverage | Price snapshot: 9 symbols, 2024-01-03 to common 2026-07-17; no factor vintages/Siamchart manifest on disk | Strict PIT backtest should currently block, not manufacture a default start |
|
||||
|
||||
---
|
||||
|
||||
## Target Event Flow
|
||||
|
||||
```text
|
||||
Coverage/readiness
|
||||
→ earliest date where ALL PIT inputs + executable prices exist
|
||||
→ initial recommendation event at start
|
||||
→ freeze target signal
|
||||
→ next trading day execution (100-share lots)
|
||||
→ for each data release:
|
||||
coalesce same-day changes
|
||||
recompute recommendation from PIT state
|
||||
if target changes: sell first → credit proceeds/realized P&L → buy using current cash
|
||||
→ for each dividend:
|
||||
ex-date: record entitlement using shares held before ex-date
|
||||
ex-date + 30 calendar days: credit cash; available to next rebalance
|
||||
→ end date:
|
||||
mark holdings to latest close ≤ end
|
||||
report cash + market value + dividend receivable (separate)
|
||||
reconcile all P&L components
|
||||
```
|
||||
|
||||
Accounting invariant:
|
||||
|
||||
```text
|
||||
ending_equity - initial_capital
|
||||
= realized_trading_pnl
|
||||
+ unrealized_trading_pnl
|
||||
+ dividend_cash_received
|
||||
+ accrued_dividend_receivable
|
||||
- transaction_costs
|
||||
```
|
||||
|
||||
`price_pnl = realized_trading_pnl + unrealized_trading_pnl` for compatibility.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add Strict Backtest Coverage and Default-Date Readiness
|
||||
|
||||
**Objective:** Derive valid start/end defaults from actual PIT stores and block when coverage is incomplete.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/backtest_readiness.py`
|
||||
- Modify: `backend/app/factor_vintages.py`
|
||||
- Modify: `backend/app/siamchart_vintages.py`
|
||||
- Modify: `backend/app/prices.py` or add a read-only adapter for price coverage
|
||||
- Modify: `backend/app/__init__.py`
|
||||
- Test: `backend/tests/test_backtest_readiness.py`
|
||||
|
||||
**Steps:**
|
||||
1. Write failing tests for:
|
||||
- no factor vintages → `ready=false`, missing factor list
|
||||
- no Siamchart snapshot → blocked
|
||||
- price coverage starting after PIT factors → recommended start equals latest of first-ready dates
|
||||
- recommended end = yesterday in Bangkok, bounded by latest price date
|
||||
- user start earlier than readiness → HTTP 400 with exact missing coverage
|
||||
2. Add store coverage APIs:
|
||||
- `FactorVintageStore.first_release(factor_key)` and coverage summary
|
||||
- `SiamchartVintageStore.first_retrieved_at()`
|
||||
- price coverage: earliest/latest executable trading dates by required universe
|
||||
3. Implement `BacktestReadiness` response:
|
||||
- `ready`, `recommended_start`, `recommended_end`, `missing`, `coverage`, `timezone`
|
||||
4. Add `GET /api/v1/backtest/readiness`.
|
||||
5. Change `POST /api/v1/backtest` defaults to readiness-derived dates; never hardcode dates.
|
||||
6. Verify targeted tests and API smoke.
|
||||
|
||||
**Acceptance:** Current local dataset returns `ready=false` because factor/Siamchart PIT history is absent; it does not start a purported PIT run.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Build the Unified Backtest Event Calendar
|
||||
|
||||
**Objective:** Rebalance on information changes, not arbitrary monthly/quarterly dates.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/backtest_events.py`
|
||||
- Modify: `backend/app/factor_vintages.py`
|
||||
- Modify: `backend/app/siamchart_vintages.py`
|
||||
- Modify: `backend/app/dividend_ledger.py`
|
||||
- Test: `backend/tests/test_backtest_events.py`
|
||||
|
||||
**Steps:**
|
||||
1. Define typed/dataclass events:
|
||||
- `SignalReleaseEvent(released_at, sources, factor_keys)`
|
||||
- `ExecutionEvent(signal_date, execution_date)`
|
||||
- `DividendEntitlementEvent(ex_date, symbol, per_share)`
|
||||
- `DividendPaymentEvent(assumed_payment_date=ex_date+30d, entitlement_id, timing_method="ex_date_plus_30d")`
|
||||
- `EndValuationEvent(end_date)`
|
||||
2. Add chronological iterators to factor and Siamchart vintage stores.
|
||||
3. Build release calendar from all PIT changes within `[start, end]`.
|
||||
4. Coalesce releases on the same date.
|
||||
5. Map each signal date to the next available trading date strictly after the release date.
|
||||
6. Include dividend entitlement/payment events and final valuation event.
|
||||
7. Create an assumed payment event exactly 30 calendar days after ex-date. Keep the dividend as a receivable until that event, then credit spendable cash. Expose the timing assumption in every event/result.
|
||||
8. Test ordering, holiday/weekend handling, coalescing, and no-look-ahead execution.
|
||||
|
||||
**Acceptance:** A factor release three months after start generates one new frozen signal and, only if targets differ, one rebalance on the next trading day.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Implement the Portfolio Accounting Ledger
|
||||
|
||||
**Objective:** Own all cash, holdings, cost basis, trades, dividends, and P&L in one auditable component.
|
||||
|
||||
**Files:**
|
||||
- Create: `backend/app/portfolio_ledger.py`
|
||||
- Test: `backend/tests/test_portfolio_ledger.py`
|
||||
|
||||
**Data model:**
|
||||
|
||||
```python
|
||||
Position(symbol, qty, average_cost)
|
||||
Trade(date, signal_date, symbol, side, qty, price, notional,
|
||||
cost_basis_released, realized_pnl, fees, cash_after, reason)
|
||||
DividendReceivable(symbol, ex_date, assumed_payment_date,
|
||||
timing_method="ex_date_plus_30d", qty_entitled,
|
||||
per_share, amount, status)
|
||||
PortfolioState(cash, positions, receivables, realized_pnl,
|
||||
dividend_cash, fees, trades)
|
||||
```
|
||||
|
||||
**Steps:**
|
||||
1. Write RED tests for average-cost buys, partial sells, full sells, realized P&L, and no negative cash.
|
||||
2. Implement `buy()`:
|
||||
- qty positive and multiple of 100
|
||||
- all-in fee = `notional * 0.003`
|
||||
- cost + fee <= cash
|
||||
- recompute weighted average cost
|
||||
3. Implement `sell()`:
|
||||
- qty positive/multiple of 100 and <= position
|
||||
- all-in fee = `notional * 0.003`
|
||||
- realized P&L = proceeds - released average cost - sell fee
|
||||
- credit net proceeds (`notional - fee`) to cash before later buys
|
||||
4. Implement dividend entitlement at ex-date using quantity held before ex-date.
|
||||
5. Implement payment-date cash credit; paid dividend can fund later orders.
|
||||
6. Keep unpaid dividends as receivables, not spendable cash.
|
||||
7. Add mark-to-market and reconciliation methods.
|
||||
8. Test cash conservation and accounting identity after mixed buys/sells/dividends.
|
||||
|
||||
**Acceptance:** Profitable sales and paid dividends increase cash available to subsequent 100-share-lot purchases; unpaid receivables do not.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Reconcile Target Portfolio with Lot and Cash Constraints
|
||||
|
||||
**Objective:** Convert each frozen recommendation into executable sell/buy orders using current total equity and current cash.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/simulation.py`
|
||||
- Create or modify: `backend/app/portfolio_rebalancer.py`
|
||||
- Test: `backend/tests/test_portfolio_rebalancer.py`
|
||||
|
||||
**Steps:**
|
||||
1. Preserve canonical 50/20/30 allocation rules and 100-share lots.
|
||||
2. Compute target quantities from current equity (cash + market value + paid dividends; exclude receivables).
|
||||
3. Generate sells first, execute them, then recompute available cash.
|
||||
4. Generate buys from remaining target deficits, never exceeding cash.
|
||||
5. If a target lot is unaffordable, skip it and retain cash; never buy odd lots.
|
||||
6. Rebalance only when target quantities differ from current quantities.
|
||||
7. Add tests for:
|
||||
- sale profit funds next purchase
|
||||
- dividend payment funds next purchase
|
||||
- insufficient cash leaves cash and skips order
|
||||
- unchanged recommendation produces zero trades
|
||||
- no negative cash under rounding/fees
|
||||
|
||||
**Acceptance:** Every trade is in 100-share multiples, and ending cash is always non-negative.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Rewrite Backtest Engine as Event Processor
|
||||
|
||||
**Objective:** Replace `_rebalance_dates()` calendar loop with the event calendar and portfolio ledger.
|
||||
|
||||
**Files:**
|
||||
- Rewrite: `backend/app/backtest.py`
|
||||
- Modify: `backend/app/pit_scorer.py`
|
||||
- Test: `backend/tests/test_backtest.py`
|
||||
- Test: `backend/tests/test_backtest_integration.py`
|
||||
|
||||
**Steps:**
|
||||
1. RED test the complete scenario:
|
||||
- start recommendation buys A
|
||||
- release three months later changes target to B
|
||||
- sell A at profit
|
||||
- dividend entitlement/payment occurs
|
||||
- sale proceeds + paid dividend fund B purchase
|
||||
- final holdings/value and P&L reconcile
|
||||
2. Initialize from strict PIT readiness and initial signal.
|
||||
3. Process events in order:
|
||||
- release → freeze scores/target
|
||||
- execution → reconcile portfolio
|
||||
- ex-date → entitlement
|
||||
- payment → cash credit
|
||||
- end → mark-to-market
|
||||
4. Record every release, rebalance, trade, and dividend event.
|
||||
5. Ensure `leakage_guard=true` only if every score at every release is PIT; otherwise reject strict mode.
|
||||
6. Remove monthly/quarterly frequency as the primary trigger. Optionally retain a separate explicit `scheduled` research mode, clearly non-default/non-event-driven.
|
||||
7. Verify no double-counted dividends and no repeated historical credits.
|
||||
|
||||
**Acceptance:** Engine output exactly follows the user-described lifecycle and passes accounting reconciliation for every tested scenario.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Expand Backtest Result Contract and Durable Run Storage
|
||||
|
||||
**Objective:** Return all requested end-state and P&L breakdowns with a durable audit trail.
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/app/backtest.py`
|
||||
- Create: `backend/app/backtest_store.py`
|
||||
- Modify: `backend/app/__init__.py`
|
||||
- Test: `backend/tests/test_backtest_store.py`
|
||||
- Modify: API tests in `backend/tests/test_api.py`
|
||||
|
||||
**Response v2:**
|
||||
|
||||
```json
|
||||
{
|
||||
"start": "...",
|
||||
"end": "...",
|
||||
"initial_capital": 1000000,
|
||||
"ending_cash": 12345,
|
||||
"ending_market_value": 1100000,
|
||||
"dividend_receivable": 500,
|
||||
"final_equity": 1112845,
|
||||
"net_change": 112845,
|
||||
"net_return": 0.112845,
|
||||
"realized_trading_pnl": 45000,
|
||||
"unrealized_trading_pnl": 50000,
|
||||
"price_pnl": 95000,
|
||||
"dividend_cash_received": 18000,
|
||||
"transaction_costs": 155,
|
||||
"fee_rate": 0.003,
|
||||
"holdings": [{"symbol":"...","qty":100,"average_cost":...,"last_price":...,"market_value":...,"unrealized_pnl":...}],
|
||||
"trades": [...],
|
||||
"dividends": [...],
|
||||
"rebalance_events": [...],
|
||||
"accounting_reconciled": true,
|
||||
"leakage_guard": true
|
||||
}
|
||||
```
|
||||
|
||||
**Steps:**
|
||||
1. Add result dataclasses and compatibility aliases (`final_value`, `dividend_income`).
|
||||
2. Add durable JSON run store (atomic writes) rather than process-local `backtest_runs`.
|
||||
3. Persist exact input coverage, event calendar, and scorer provenance with each run.
|
||||
4. Add reconciliation assertions before serializing; fail closed if imbalance exceeds tolerance.
|
||||
5. Test reload after app restart and API history output.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Update Frontend Defaults, Controls, and Report
|
||||
|
||||
**Objective:** Make UI defaults/readiness and results reflect the new engine honestly.
|
||||
|
||||
**Files:**
|
||||
- Modify: `frontend/src/App.vue`
|
||||
- Test/build: `frontend/package.json` existing build
|
||||
|
||||
**Steps:**
|
||||
1. On load, call `/api/v1/backtest/readiness`.
|
||||
2. Set start to `recommended_start` and end to `recommended_end`; remove hardcoded `2024-06-01` / `2026-06-01`.
|
||||
3. If readiness is blocked, disable Run and show exact missing coverage.
|
||||
4. Replace “รายเดือน/รายไตรมาส” primary control with “ตามข้อมูลใหม่ (event-driven)”. Keep scheduled frequency only under advanced/descriptive mode if retained.
|
||||
5. Display:
|
||||
- final holdings with qty, average cost, latest price, market value, unrealized P&L
|
||||
- final equity, net change, return
|
||||
- dividend cash received and dividend receivable
|
||||
- realized trading P&L, unrealized P&L, total price P&L, fees
|
||||
- event timeline / trade table
|
||||
6. Keep visible PIT/non-PIT and dated-ledger/proxy badges.
|
||||
7. Run production build and verify rendered result state.
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Verification and Independent Review Gate
|
||||
|
||||
**Objective:** Prove correctness before shipping.
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/test-evidence/YYYY-MM-DD-event-driven-backtest.md`
|
||||
- Update: `docs/engineering-log.md`
|
||||
- Update: `docs/HANDOFF.md`
|
||||
|
||||
**Required tests/gates:**
|
||||
1. Targeted event-calendar suite.
|
||||
2. Portfolio-ledger suite.
|
||||
3. Rebalancer suite.
|
||||
4. Full backtest integration suite.
|
||||
5. Full backend suite.
|
||||
6. Frontend production build.
|
||||
7. Static security scan; no secrets/debug leftovers.
|
||||
8. `git diff --check`, compileall.
|
||||
9. Live API probe with a deterministic temporary PIT fixture.
|
||||
10. Browser verification of readiness-blocked and completed-result states.
|
||||
11. Fresh independent reviewer subagent; fail closed on any logic/security blocker.
|
||||
12. Auto-fix/re-review max two cycles.
|
||||
|
||||
**Critical invariants:**
|
||||
- cash never negative
|
||||
- all buy/sell quantities multiples of 100
|
||||
- no trade before signal is knowable
|
||||
- no dividend entitlement for shares bought on/after ex-date
|
||||
- unpaid dividends never fund purchases
|
||||
- paid dividends can fund subsequent purchases
|
||||
- realized P&L uses average cost
|
||||
- unchanged target creates no trade
|
||||
- final accounting identity reconciles within one satang/rounding tolerance
|
||||
- current local data returns readiness blocked rather than fake PIT performance
|
||||
|
||||
---
|
||||
|
||||
## Files Likely to Change
|
||||
|
||||
- `backend/app/backtest.py`
|
||||
- `backend/app/backtest_readiness.py` (new)
|
||||
- `backend/app/backtest_events.py` (new)
|
||||
- `backend/app/portfolio_ledger.py` (new)
|
||||
- `backend/app/portfolio_rebalancer.py` (new)
|
||||
- `backend/app/backtest_store.py` (new)
|
||||
- `backend/app/dividend_ledger.py`
|
||||
- `backend/app/factor_vintages.py`
|
||||
- `backend/app/siamchart_vintages.py`
|
||||
- `backend/app/prices.py`
|
||||
- `backend/app/simulation.py`
|
||||
- `backend/app/__init__.py`
|
||||
- `backend/tests/test_backtest*.py`
|
||||
- new focused backend test modules
|
||||
- `frontend/src/App.vue`
|
||||
- engineering/test-evidence docs
|
||||
|
||||
## Explicit Non-Goals
|
||||
|
||||
- Live order execution / MT5 dispatch
|
||||
- Guessing historical factor values that were never recorded
|
||||
- Claiming PIT performance before coverage readiness passes
|
||||
- Broker-specific fee assumptions without configuration
|
||||
- Odd-lot trading
|
||||
|
||||
## Main Risks
|
||||
|
||||
1. **No current strict PIT-ready window:** the local store has no factor vintages/Siamchart manifest. The correct initial UI state is blocked.
|
||||
2. **Dividend payment timing is assumed:** current Siamchart data supplies ex-date+DPS, not payment date. The user confirmed `assumed_payment_date = ex_date + 30 calendar days`; every result must disclose `timing_method=ex_date_plus_30d`.
|
||||
3. **Price quality:** current Yahoo revised history is explicitly non-PIT; strict validated backtests need an evidence-bearing price archive contract.
|
||||
4. **Universe coverage:** current price snapshot contains 9 symbols, not the full 49-symbol board.
|
||||
5. **Response compatibility:** current UI/API expect scalar holdings and `price_pnl`; v2 needs compatibility aliases during migration.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None for the core accounting/event model. The implementation decisions above are confirmed.
|
||||
101
.hermes/plans/2026-08-29_additional-data-sources-per-theme.md
Normal file
101
.hermes/plans/2026-08-29_additional-data-sources-per-theme.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Architecture Plan — Additional Data Sources per Theme + Use-in-Analysis Enforcement
|
||||
|
||||
> Status: DRAFT — pending user approval (decision gate G0).
|
||||
> Author: Macky. Date: 2026-08-29.
|
||||
|
||||
## Goal
|
||||
Close the "few sources per theme" gap AND lock the user's rule — every data point fetched must feed analysis, not just display. This is the follow-up the declarative factor engine (2026-08-26 plan) explicitly deferred: "no new collectors yet; sources are added as follow-up."
|
||||
|
||||
Two deliverables:
|
||||
1. **Wiring convention that guarantees use-in-analysis** — a build/test pattern so a new source cannot be added without being consumed by `compute_theme_surprises`.
|
||||
2. **Two new high-feasibility Thai sources**, wired end-to-end, verified against live pages: **BOT Balance of Payments** (report 60) and **BOT Commercial-bank loans** (reportID confirmed at build). A **medium-feasibility** source (**REIC house-price/transfer**) is scoped but flagged for a feasibility spike before committing.
|
||||
|
||||
## Why this is needed (evidence — verified against live sources this session)
|
||||
- 6 of 13 themes (`telecom_it`, `property`, `healthcare`, `consumer_staples`, `petrochem_materials`, `retail`) rest on **one BOT macro page**. Source count is genuinely thin.
|
||||
- Found + fixed (this session, prior commit pending): `core_inflation_yoy` and `unemployment_pct` were **fetched but never used**; worse, every `sign:-1` factor (NPL/inflation/unemployment) used a **negative** theme weight → double-negative → **higher NPL raised the theme score**. Both fixed in `factors.py`/`themes.py` with regression tests. This plan prevents recurrence and adds genuinely *new providers*, not just new fields on the same page.
|
||||
- **Feasibility verified live**: BOT `ReportPage.aspx?reportID=60` (Balance of Payments) responds to the exact `dgExcel` form-POST flow already implemented in `bot_tourism.py` (200, parseable — exports/imports/current-account series extracted this session). REIC homepage is a portal — data lives on subpages/endpoints (medium feasibility).
|
||||
|
||||
## Why the current architecture makes this cheap
|
||||
Adding a factor already needs **no scoring-function change** (`factors.py` registry + a theme factor line). The new work is only:
|
||||
- a collector module exposing `.to_dict()`,
|
||||
- registry + theme wiring,
|
||||
- a scheduler `_REFRESH_JOBS` entry (so it collects + appears in source-health log),
|
||||
- a `_source_label` provenance entry,
|
||||
- tests.
|
||||
|
||||
## Decision Gates (approve before code)
|
||||
- **G0** — Approve the 2 source choices + the mandatory-use test contract (below). Exit → build B0.
|
||||
- **G1** — Collector parses a frozen fixture; factor value extracted; **used-in-analysis test passes**. Exit → wire.
|
||||
- **G2** — `/api/v1/dashboard` shows the new source row + `source_summary` factor_keys increases; board surprises change with it. Exit → done.
|
||||
|
||||
## New collectors
|
||||
|
||||
### B0 — Thailand External Sector / Trade (`thai_trade.py`) [high feasibility — VERIFIED current]
|
||||
- **PIVOT from BOT BOP report 60:** probing live this session showed the BOT statistics portal's `reportID=60` only carries data **through 2011** (its `drpToYear` options cap at `2011xxxx`) — stale, not fit for a 2026 platform. Per the plan's gate ("do not ship a factor reading a wrong/stale series") we source the external sector from **TradingEconomics Thailand trade**, which is **live through June 2026** and reuses the exact `auto_credit.py` scraping pattern (same site/infobox):
|
||||
- `https://tradingeconomics.com/thailand/current-account` → current-account balance (USD mn, June 2026 verified)
|
||||
- `https://tradingeconomics.com/thailand/exports` → exports (USD mn, 2026 verified)
|
||||
- `https://tradingeconomics.com/thailand/imports` → imports (USD mn, 2026 verified)
|
||||
- Factors (declarative additions):
|
||||
- `external_current_account` — current-account balance → sign +1 (strong external position supportive). Feeds **tourism**, **exploration**, **petrochem_materials**, **retail**.
|
||||
- `external_exports` — exports USD mn → sign +1. Feeds **petrochem_materials**, **exploration**, **telecom_it**.
|
||||
- `external_imports` — imports USD mn (domestic demand) → sign +1. Feeds **retail**, **consumer_staples**, **property**.
|
||||
- Frequency: monthly. Scheduler cadence: monthly.
|
||||
- Test: frozen fixture (one value sentence) → parse → assert current account / exports / imports; used-in-analysis test asserts a change in the fed themes.
|
||||
|
||||
### B1 — BOT Commercial-bank loans (`bot_bank_loans.py`) [high feasibility — flow VERIFIED, reportID TBD]
|
||||
- Same `dgExcel` adapter, new reportID. The exact BOT report for commercial-bank loans/deposits is **confirmed accessible in principle** (report 60 proved the flow); its numeric reportID is discovered at build time from BOT's statistics index / report search (do NOT hardcode an unverified ID).
|
||||
- Fields expected: total loans outstanding, deposits, NPL (commercial banks), **loan growth YoY** as derived factor.
|
||||
- Factors:
|
||||
- `bank_loan_growth` — commercial-bank loan growth YoY → sign +1. Feeds **banks** (primary), **nonbank_finance**, **property** (mortgage-linked).
|
||||
- `bank_deposit_growth` — deposit growth → sign +1 (funding base). Feeds **banks**.
|
||||
- Frequency: monthly (BOT reports these monthly). Scheduler cadence: monthly.
|
||||
- **Gate**: if the found reportID/data is not cleanly loan-related, fall back to BOT "credit by sector" report and re-scope the factor — do not ship a factor reading a wrong series.
|
||||
|
||||
### B2 — REIC house-price / transfer index (`reic_housing.py`) [MEDIUM feasibility — SPIKE FIRST]
|
||||
- Homepage is a portal (verified). The house-price index / residential transfer data live on a subpage or a data endpoint (likely JSON/PDF). **Spike task before G0 sign-off on this one**: locate the real data URL, confirm it is scrapeable without an API key.
|
||||
- If feasible → factor `reic_housing_price_index` / `reic_transfer_value_yoy` → sign +1 → feeds **property**.
|
||||
- If the only clean access is a PDF chart with no numeric series → **defer** (do not ship a display-only scrape). This is the exact "แค่ดึงข้อมูลมาเฉย ๆ" the user forbids.
|
||||
|
||||
## Wiring checklist per source (all must land)
|
||||
1. `backend/app/<module>.py` — dataclass + `to_dict()` + `fetch_<module>()` (+ frozen-fixture parse fn).
|
||||
2. `factors.py` — FACTORS entries (source/frequency/fetch/value_key/sign/center/span).
|
||||
3. `themes.py` — factor lines with **positive** weights on the sign-correct themes (the sign lives in the factor).
|
||||
4. `scheduler.py` — `_REFRESH_JOBS` entry (module/fn/fetch_module/frequency) so it collects + shows in the health log.
|
||||
5. `dashboard.py` — `_build_sources` `_source_label` map entry (provenance table).
|
||||
6. `__init__.py` — no endpoint change needed (dashboard drives the UI); verify `/api/v1/dashboard` picks it up (G2).
|
||||
|
||||
## Use-in-analysis enforcement (the user's rule, made a test)
|
||||
Two contracts, added as tests so "fetched but not used" can never silently return:
|
||||
- **Contract A (value_key resolves):** every `FACTORS.value_key` must be a real field the registered fetch module emits. Already added (`test_every_factor_value_key_resolves_to_a_fetched_field`). Extend it as new modules land.
|
||||
- **Contract B (new factor moves scores):** for each new factor, a test feeds a neutral vs a hot value and asserts the target theme surprise changes **in the right direction** (bearish factors lower it). Pattern already added (`test_new_macro_factors_actually_move_scores`, `test_bearish_factors_move_score_the_right_way`). One such test per new factor.
|
||||
|
||||
This dual contract is the build guarantee that fulfills "มีแหล่งข้อมูลใหม่ → ต้องใช้ในการวิเคราะห์จริง".
|
||||
|
||||
## Data Model sketch
|
||||
```python
|
||||
# factors.py
|
||||
"macro_current_account": {"source":"BOT BOP","frequency":"monthly",
|
||||
"fetch":"bot_bop","value_key":"current_account_balance_mb",
|
||||
"sign":1, "center":0.0, "span":80000.0},
|
||||
"bank_loan_growth": {"source":"BOT CB loans","frequency":"monthly",
|
||||
"fetch":"bot_bank_loans","value_key":"loan_growth_yoy",
|
||||
"sign":1, "center":5.0, "span":10.0},
|
||||
|
||||
# themes.py — positive weights; direction is in the factor
|
||||
"banks": { "factors": [ {"key":"macro_investment","weight":1.0},
|
||||
{"key":"bank_loan_growth","weight":0.8},
|
||||
... ] },
|
||||
```
|
||||
|
||||
## Non-Goals (v1)
|
||||
- No new factors on the existing single BOT macro page beyond the two already wired (core inflation, unemployment) — avoid the "one page, many pseudo-independent factors" trap.
|
||||
- No live trading / MT5 changes. No LLM in scoring. No API-key sources (free/scrapeable only).
|
||||
- Do not ship a REIC scrape unless a clean numeric series exists.
|
||||
|
||||
## Acceptance (definition of done)
|
||||
1. Thailand external-sector (TradingEconomics trade) collector + 3 factors land, verified against a frozen fixture and a live fetch (current through 2026 — not the stale BOT-60 series).
|
||||
2. BOT commercial-bank-loan collector + factors land (reportID confirmed; else scoped fallback).
|
||||
3. REIC either lands with a real series **or** is explicitly deferred (no display-only scrape).
|
||||
4. Every new factor has Contract A + Contract B tests green.
|
||||
5. Full backend suite (352 → target ≥ 366) + `npm run build` pass.
|
||||
6. `/api/v1/dashboard` `sources` + `source_summary.factor_keys` increase; board surprises visibly move with a hot value.
|
||||
138
.hermes/plans/2026-08-29_data-source-expansion-phase-2.md
Normal file
138
.hermes/plans/2026-08-29_data-source-expansion-phase-2.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# Data Source Expansion Plan — Phase 2 (more Thai sources per theme)
|
||||
|
||||
> **For Hermes:** Execute task-by-task. One task in progress at a time; commit by phase; run `requesting-code-review` before delivery. Every new source MUST end wired into `FACTORS` (one dict entry) **and** referenced by at least one `THEMES[].factors` row — so it feeds `compute_theme_surprises()`, never just the source table. Follow the existing collector template (`backend/app/thai_trade.py`, `auto_credit.py`). Run the full backend suite + `npm run build` after each phase.
|
||||
|
||||
**Goal:** Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table.
|
||||
|
||||
**Architecture (unchanged, declarative):**
|
||||
```
|
||||
new source → collector module (fetch_<x>() → Snapshot.to_dict())
|
||||
→ FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span}
|
||||
→ THEMES[<theme>].factors [{key, weight}]
|
||||
→ compute_theme_surprises() picks it up with zero scoring-fn change
|
||||
```
|
||||
Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by `thai_trade`.
|
||||
|
||||
**Current state (verified, baseline green: 352/352 tests):**
|
||||
- 7 live sources: `bot_tourism`, `auto_credit`, `auto_npl`, `bank_npl`, `energy_thai`, `macro_thai`, `thai_trade`.
|
||||
- 17 FACTORS registered. All `value_key` resolve to a real fetched field (locked by `test_every_factor_value_key_resolves_to_a_fetched_field`).
|
||||
- Sign convention FIXED this session: `sign` lives only in the factor; theme weights are positive magnitude. Regression-locked by `test_bearish_factors_move_score_the_right_way` (higher NPL ⇒ lower score, etc.).
|
||||
- `thai_trade.py` (TradingEconomics current-account) is the reference collector: server-rendered HTML, `_fetch`/`parse`/`fetch_<x>`/`to_dict`, registered in scheduler `_REFRESH_JOBS` + dashboard `_fetch_with_cache` + FACTORS.
|
||||
|
||||
---
|
||||
|
||||
## Confirmed Decisions (from user: "วางแผนได้เลย")
|
||||
| Decision | Chosen | Rationale |
|
||||
|---|---|---|
|
||||
| Scope | Phase A (macro-deepening) first, Phase B (sector) next | Fastest correctness win, lowest scrap fragility |
|
||||
| New-source gate | Feasibility spike per source BEFORE building collector | Don't ship a wrong/stale/unscrapable series (same gate `thai_trade.py` used) |
|
||||
| Integration | Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache | User rule: new data must feed analysis, not just be fetched |
|
||||
| Test | One value-key-resolution + one direction test per new source | Locks "used in analysis" invariant |
|
||||
|
||||
## Pending user decision (gate G0 — before Phase A code)
|
||||
1. **Which sources to prioritize** — recommendation in priority order below. User may reorder/substitute.
|
||||
2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page).
|
||||
|
||||
---
|
||||
|
||||
## Current-State Gap Matrix (per theme, what feeds it today)
|
||||
| Theme | Sources today | Gap / weakest link |
|
||||
|---|---|---|
|
||||
| banks | BOT macro (invest/inflation/NPL) — 3 fields, all BOT | No rate/loan-setting input; NPL only "financial sector" proxy |
|
||||
| retail | BOT macro (consumption/inflation/unemployment) + TE imports | No direct retail-sales / consumer-mood series |
|
||||
| consumer_staples | BOT macro + TE imports | Same as retail |
|
||||
| telecom_it | BOT macro (consumption/investment) + TE exports | No telecom-specific (subs/data) series |
|
||||
| property | BOT macro (invest/consumption/inflation) + TE imports | No real-estate-specific (transfer/mortgage) series |
|
||||
| nonbank_finance | BOT macro + auto NPL + unemployment | No household-credit / consumer-loan series |
|
||||
| healthcare | BOT macro (consumption/unemployment) | No healthcare/tourism-medical series |
|
||||
| utilities | BOT macro (mfg) + TOP margin | No electricity-demand / generation series |
|
||||
| tourism | BOT tourism + macro consumption + TE current acct | Covered well; optional: hotel occupancy |
|
||||
| auto_credit | TE vehicle + BOT auto NPL | Covered well |
|
||||
| refining_energy / exploration / petrochem | TOP + macro mfg/inflation + TE exports | Only TOP company; no commodity/oil price |
|
||||
|
||||
---
|
||||
|
||||
## Phase A — deepen macro-proxy themes (priority order)
|
||||
Each is a collector + FACTORS entry ×N + THEME wiring + tests.
|
||||
|
||||
### A1. BOT policy/loan-rate + credit — banks, nonbank_finance
|
||||
- Target: BOT monetary-policy / rate page (e.g. `bot.or.th` policy rate) + BOT credit/loan-growth report.
|
||||
- Factors: `bank_policy_rate` (sign −1 for banks? higher rate squeezes demand), `bank_loan_growth_yoy` (sign +1).
|
||||
- Feasibility: spike must confirm a scrapeable numeric series on a BOT page.
|
||||
|
||||
### A2. Household / consumer credit — nonbank_finance (+ banks)
|
||||
- Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report.
|
||||
- Factor: `consumer_credit_yoy` (sign +1), `consumer_credit_npl` (sign −1).
|
||||
- Feasibility: spike on BOT statistics page.
|
||||
|
||||
### A3. Retail sales index — retail, consumer_staples
|
||||
- Target: BOT or TradingEconomics "retail sales" Thailand YoY.
|
||||
- Factor: `retail_sales_yoy` (sign +1).
|
||||
- Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit).
|
||||
|
||||
### A4. Consumer confidence — retail, consumer_staples, nonbank_finance
|
||||
- Target: UTCC / Kasikorn Research consumer-confidence index (free HTML).
|
||||
- Factor: `consumer_confidence` (sign +1).
|
||||
- Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence".
|
||||
|
||||
## Phase B — sector-specific (IMPLEMENTED 2026-08-29 via TradingEconomics single-page snapshots)
|
||||
### B1. Property: `te_property_prices` (residential property prices % YoY) — DONE, feeds property theme
|
||||
### B2. Utilities: EPPO electricity — DEFERRED (no clean single-page TE snapshot; needs EPPO scraper)
|
||||
### B3. Energy breadth: PTT/PTTEP/BCP quarterly — DEFERRED (needs company-IR scrapers, heavier)
|
||||
### B4. Telecom/backdrop: `te_business_confidence` — DONE, feeds telecom_it + property + healthcare
|
||||
### B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence)
|
||||
Both added to the existing `te_thailand.py` module (2 extra TE pages) — same reviewed pattern, +2 factors, +4 tests.
|
||||
|
||||
## Remaining backlog (needs dedicated scrapers, not single-page snapshots)
|
||||
- REIC property transfer/housing supply (TH-specific, richer than a TE index)
|
||||
- EPPO electricity demand/generation for utilities
|
||||
- PTT/PTTEP/BCP/IRPC quarterly financials (beyond TOP) for energy breadth
|
||||
- NBTC subscriber/data for a true telecom-specific series
|
||||
|
||||
---
|
||||
|
||||
## Task Breakdown
|
||||
### Task A0 — Feasibility spikes (gate, not build)
|
||||
For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under `backend/scripts/spike_<source>.py`; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value.
|
||||
- Exit: a table of "source → scrapable? → value → period".
|
||||
|
||||
### Task A1 — BOT rate/loan + credit factors
|
||||
- Add module `backend/app/bot_rates.py` (or extend existing) with Snapshot + `fetch_<x>()` + `.to_dict()`.
|
||||
- Register FACTORS (policy_rate, loan_growth…) + wire into `banks`/`nonbank_finance` THEMES.
|
||||
- Add `_REFRESH_JOBS` row + dashboard cache line.
|
||||
- Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand).
|
||||
- Verify: full backend suite green; `/api/v1/dashboard` themes show the new factors in `sources`.
|
||||
|
||||
### Task A2 — Household/consumer credit (nonbank_finance)
|
||||
- Same pattern as A1; factor `consumer_credit_yoy` (+1), `consumer_credit_npl` (−1) into `nonbank_finance` (+ maybe banks).
|
||||
- Tests + suite green.
|
||||
|
||||
### Task A3 — Retail sales (retail, consumer_staples)
|
||||
- Collector on TE Thailand retail-sales page; factor `retail_sales_yoy` (+1) into retail + consumer_staples.
|
||||
- Tests + suite green.
|
||||
|
||||
### Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance)
|
||||
- Collector; factor `consumer_confidence` (+1). Use TE fallback if UTCC is paywalled.
|
||||
- Tests + suite green.
|
||||
|
||||
### Task A5 — Frontend macro chips / theme cards
|
||||
- Add the new read values to `App.vue` theme cards / macro chips (single-colour, no new library, follow existing `theme-read-value` pattern).
|
||||
- `npm run build` green; visual check at 320×568 and 500×768 (mobile single-column).
|
||||
|
||||
### Task B1..B5 — Phase B (only after Phase A accepted)
|
||||
- Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance (definition of done)
|
||||
1. Every new FACTOR appears in ≥1 `THEMES[].factors` and is referenced by dashboard `_build_sources` (auto from registry) — **no source shows in the table without feeding a surprise**.
|
||||
2. `test_every_factor_value_key_resolves_to_a_fetched_field` still passes for every new factor (value_key is a real fetched field).
|
||||
3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score).
|
||||
4. Full backend suite green (was 352) + `npm run build` green.
|
||||
5. Scheduler `_REFRESH_JOBS` + dashboard cache include every new source.
|
||||
6. `/api/v1/dashboard` sources table length tracks FACTORS count; new theme reads appear on the dashboard.
|
||||
|
||||
## Open questions for G0
|
||||
- Priority/order of A1–A4 (recommended order above; user may reorder).
|
||||
- Source substitution if a spike fails: fallback list provided per task.
|
||||
- Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed?
|
||||
@@ -0,0 +1,50 @@
|
||||
# 2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug
|
||||
|
||||
## Scope
|
||||
Two-part request from the owner:
|
||||
1. **More + genuinely-used data sources per theme** — the owner felt the source
|
||||
count was too few and insisted any newly-fetched data must feed the analysis
|
||||
(theme surprise), not just the provenance table.
|
||||
2. **Frontend padding/margin issues** — several panels had misaligned spacing.
|
||||
|
||||
## What changed (all verified)
|
||||
| Area | Change | Evidence |
|
||||
|---|---|---|
|
||||
| `factors.py` | Wired 2 **fetched-but-unused** BOT fields into the registry: `macro_core_inflation` (core inflation, sign −1) and `macro_unemployment` (sign −1) → they now actually move themes | Before, `macro_thai.to_dict()` emitted 7 fields but only 5 were registered factors — 2 were dead display data |
|
||||
| `themes.py` | Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights) | direction tests |
|
||||
| **Sign bug fix** | Found + fixed a **double-negative**: several factors carry `sign: -1` (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in `sign`. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it. | `test_bearish_factors_move_score_the_right_way` |
|
||||
| **NEW `te_thailand.py`** | New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence | live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8 |
|
||||
| `themes.py` | Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence) | direction tests |
|
||||
| `scheduler.py` / `dashboard.py` | Registered `te_thailand` in `_REFRESH_JOBS` + dashboard cached-fetch + `_build_sources` provenance | dashboard live build: 8 sources |
|
||||
| `frontend/App.vue` | Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP) | build clean |
|
||||
| `frontend/style.css` | **padding/margin + undefined-variable fix** — added missing design tokens (`--card/--foreground/--accent/--font/...`) that the theme/modal components reference but `:root` never declared (they rendered transparent/wrong-color); zeroed section-panel padding so `.signal-header` is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels) | build clean |
|
||||
|
||||
## Key decisions
|
||||
- **One new module** (`te_thailand.py`) covers 4 plan items (A1 rate/credit, A2
|
||||
consumer credit, A3 retail sales, A4 confidence) because all four series live
|
||||
on just two TradingEconomics pages — avoids 4 fragile scrapers.
|
||||
- **Every source must feed a surprise**: tightened `test_every_factor_value_key_resolves_to_a_fetched_field` so a factor whose `value_key` the collector never emits fails the suite — no dead factors.
|
||||
- Sign convention locked: `sign` = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly.
|
||||
|
||||
## Verified (evidence)
|
||||
- Full backend suite: **360 tests OK** (up from 352 — 8 new `test_te_thailand`).
|
||||
- Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise −0.165
|
||||
(correctly negative from retail-sales −14.5% YoY) vs old BOT-only (positive).
|
||||
- Frontend `npm run build` clean.
|
||||
- Independent code review: see delegation verdict (fail-closed).
|
||||
|
||||
## Phase B (same day, follow-up "ทำต่อได้เลย")
|
||||
Extended the same `te_thailand` module with 2 more TradingEconomics pages:
|
||||
- `te_property_prices` (residential property prices +1.26% YoY) → **property** theme
|
||||
- `te_business_confidence` (46.7) → **telecom_it + property + healthcare**
|
||||
+2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now
|
||||
**362 tests OK**; frontend build clean; live fetch confirmed (1.26 / 46.7).
|
||||
Cleared a stale daily-cache `te_thailand` entry so the new fields show immediately.
|
||||
|
||||
## Backlog (needs dedicated scrapers, not single-page snapshots)
|
||||
REIC property supply, EPPO electricity (utilities), PTT/PTTEP/BCP quarterly breadth
|
||||
(energy), NBTC subscriber/data (telecom). Marked deferred in the plan.
|
||||
|
||||
- Note: a **concurrent process** also landed `thai_trade.py` (external-sector
|
||||
exports/imports/current-account) and external_* factors mid-session; its 3
|
||||
initially-broken tests were fixed to reach the 360-green baseline here.
|
||||
Reference in New Issue
Block a user