kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: data-source expansion plan + engineering log (te_thailand, sign fix, ui fix)

Browse Source
This commit is contained in:
Kunthawat Greethong
2026-08-29 09:19:06 +07:00
parent 9516a7a8e8
commit dbb787c50a
11 changed files with 1536 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

138
.hermes/plans/2026-08-29_data-source-expansion-phase-2.md Normal file
View File

@@ -0,0 +1,138 @@
# Data Source Expansion Plan — Phase 2 (more Thai sources per theme)
> **For Hermes:** Execute task-by-task. One task in progress at a time; commit by phase; run `requesting-code-review` before delivery. Every new source MUST end wired into `FACTORS` (one dict entry) **and** referenced by at least one `THEMES[].factors` row — so it feeds `compute_theme_surprises()`, never just the source table. Follow the existing collector template (`backend/app/thai_trade.py`, `auto_credit.py`). Run the full backend suite + `npm run build` after each phase.
**Goal:** Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table.
**Architecture (unchanged, declarative):**
```
new source → collector module (fetch_<x>() → Snapshot.to_dict())
→ FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span}
→ THEMES[<theme>].factors [{key, weight}]
→ compute_theme_surprises() picks it up with zero scoring-fn change
```
Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by `thai_trade`.
**Current state (verified, baseline green: 352/352 tests):**
- 7 live sources: `bot_tourism`, `auto_credit`, `auto_npl`, `bank_npl`, `energy_thai`, `macro_thai`, `thai_trade`.
- 17 FACTORS registered. All `value_key` resolve to a real fetched field (locked by `test_every_factor_value_key_resolves_to_a_fetched_field`).
- Sign convention FIXED this session: `sign` lives only in the factor; theme weights are positive magnitude. Regression-locked by `test_bearish_factors_move_score_the_right_way` (higher NPL ⇒ lower score, etc.).
- `thai_trade.py` (TradingEconomics current-account) is the reference collector: server-rendered HTML, `_fetch`/`parse`/`fetch_<x>`/`to_dict`, registered in scheduler `_REFRESH_JOBS` + dashboard `_fetch_with_cache` + FACTORS.
---
## Confirmed Decisions (from user: "วางแผนได้เลย")
| Decision | Chosen | Rationale |
|---|---|---|
| Scope | Phase A (macro-deepening) first, Phase B (sector) next | Fastest correctness win, lowest scrap fragility |
| New-source gate | Feasibility spike per source BEFORE building collector | Don't ship a wrong/stale/unscrapable series (same gate `thai_trade.py` used) |
| Integration | Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache | User rule: new data must feed analysis, not just be fetched |
| Test | One value-key-resolution + one direction test per new source | Locks "used in analysis" invariant |
## Pending user decision (gate G0 — before Phase A code)
1. **Which sources to prioritize** — recommendation in priority order below. User may reorder/substitute.
2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page).
---
## Current-State Gap Matrix (per theme, what feeds it today)
| Theme | Sources today | Gap / weakest link |
|---|---|---|
| banks | BOT macro (invest/inflation/NPL) — 3 fields, all BOT | No rate/loan-setting input; NPL only "financial sector" proxy |
| retail | BOT macro (consumption/inflation/unemployment) + TE imports | No direct retail-sales / consumer-mood series |
| consumer_staples | BOT macro + TE imports | Same as retail |
| telecom_it | BOT macro (consumption/investment) + TE exports | No telecom-specific (subs/data) series |
| property | BOT macro (invest/consumption/inflation) + TE imports | No real-estate-specific (transfer/mortgage) series |
| nonbank_finance | BOT macro + auto NPL + unemployment | No household-credit / consumer-loan series |
| healthcare | BOT macro (consumption/unemployment) | No healthcare/tourism-medical series |
| utilities | BOT macro (mfg) + TOP margin | No electricity-demand / generation series |
| tourism | BOT tourism + macro consumption + TE current acct | Covered well; optional: hotel occupancy |
| auto_credit | TE vehicle + BOT auto NPL | Covered well |
| refining_energy / exploration / petrochem | TOP + macro mfg/inflation + TE exports | Only TOP company; no commodity/oil price |
---
## Phase A — deepen macro-proxy themes (priority order)
Each is a collector + FACTORS entry ×N + THEME wiring + tests.
### A1. BOT policy/loan-rate + credit — banks, nonbank_finance
- Target: BOT monetary-policy / rate page (e.g. `bot.or.th` policy rate) + BOT credit/loan-growth report.
- Factors: `bank_policy_rate` (sign −1 for banks? higher rate squeezes demand), `bank_loan_growth_yoy` (sign +1).
- Feasibility: spike must confirm a scrapeable numeric series on a BOT page.
### A2. Household / consumer credit — nonbank_finance (+ banks)
- Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report.
- Factor: `consumer_credit_yoy` (sign +1), `consumer_credit_npl` (sign −1).
- Feasibility: spike on BOT statistics page.
### A3. Retail sales index — retail, consumer_staples
- Target: BOT or TradingEconomics "retail sales" Thailand YoY.
- Factor: `retail_sales_yoy` (sign +1).
- Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit).
### A4. Consumer confidence — retail, consumer_staples, nonbank_finance
- Target: UTCC / Kasikorn Research consumer-confidence index (free HTML).
- Factor: `consumer_confidence` (sign +1).
- Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence".
## Phase B — sector-specific (IMPLEMENTED 2026-08-29 via TradingEconomics single-page snapshots)
### B1. Property: `te_property_prices` (residential property prices % YoY) — DONE, feeds property theme
### B2. Utilities: EPPO electricity — DEFERRED (no clean single-page TE snapshot; needs EPPO scraper)
### B3. Energy breadth: PTT/PTTEP/BCP quarterly — DEFERRED (needs company-IR scrapers, heavier)
### B4. Telecom/backdrop: `te_business_confidence` — DONE, feeds telecom_it + property + healthcare
### B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence)
Both added to the existing `te_thailand.py` module (2 extra TE pages) — same reviewed pattern, +2 factors, +4 tests.
## Remaining backlog (needs dedicated scrapers, not single-page snapshots)
- REIC property transfer/housing supply (TH-specific, richer than a TE index)
- EPPO electricity demand/generation for utilities
- PTT/PTTEP/BCP/IRPC quarterly financials (beyond TOP) for energy breadth
- NBTC subscriber/data for a true telecom-specific series
---
## Task Breakdown
### Task A0 — Feasibility spikes (gate, not build)
For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under `backend/scripts/spike_<source>.py`; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value.
- Exit: a table of "source → scrapable? → value → period".
### Task A1 — BOT rate/loan + credit factors
- Add module `backend/app/bot_rates.py` (or extend existing) with Snapshot + `fetch_<x>()` + `.to_dict()`.
- Register FACTORS (policy_rate, loan_growth…) + wire into `banks`/`nonbank_finance` THEMES.
- Add `_REFRESH_JOBS` row + dashboard cache line.
- Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand).
- Verify: full backend suite green; `/api/v1/dashboard` themes show the new factors in `sources`.
### Task A2 — Household/consumer credit (nonbank_finance)
- Same pattern as A1; factor `consumer_credit_yoy` (+1), `consumer_credit_npl` (−1) into `nonbank_finance` (+ maybe banks).
- Tests + suite green.
### Task A3 — Retail sales (retail, consumer_staples)
- Collector on TE Thailand retail-sales page; factor `retail_sales_yoy` (+1) into retail + consumer_staples.
- Tests + suite green.
### Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance)
- Collector; factor `consumer_confidence` (+1). Use TE fallback if UTCC is paywalled.
- Tests + suite green.
### Task A5 — Frontend macro chips / theme cards
- Add the new read values to `App.vue` theme cards / macro chips (single-colour, no new library, follow existing `theme-read-value` pattern).
- `npm run build` green; visual check at 320×568 and 500×768 (mobile single-column).
### Task B1..B5 — Phase B (only after Phase A accepted)
- Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests.
---
## Acceptance (definition of done)
1. Every new FACTOR appears in ≥1 `THEMES[].factors` and is referenced by dashboard `_build_sources` (auto from registry) — **no source shows in the table without feeding a surprise**.
2. `test_every_factor_value_key_resolves_to_a_fetched_field` still passes for every new factor (value_key is a real fetched field).
3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score).
4. Full backend suite green (was 352) + `npm run build` green.
5. Scheduler `_REFRESH_JOBS` + dashboard cache include every new source.
6. `/api/v1/dashboard` sources table length tracks FACTORS count; new theme reads appear on the dashboard.
## Open questions for G0
- Priority/order of A1–A4 (recommended order above; user may reorder).
- Source substitution if a spike fails: fallback list provided per task.
- Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed?