176 lines
12 KiB
Markdown
176 lines
12 KiB
Markdown
# Data Source Expansion Plan — Phase 2 (more Thai sources per theme)
|
||
|
||
> **For Hermes:** Execute task-by-task. One task in progress at a time; commit by phase; run `requesting-code-review` before delivery. Every new source MUST end wired into `FACTORS` (one dict entry) **and** referenced by at least one `THEMES[].factors` row — so it feeds `compute_theme_surprises()`, never just the source table. Follow the existing collector template (`backend/app/thai_trade.py`, `auto_credit.py`). Run the full backend suite + `npm run build` after each phase.
|
||
|
||
**Goal:** Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table.
|
||
|
||
**Architecture (unchanged, declarative):**
|
||
```
|
||
new source → collector module (fetch_<x>() → Snapshot.to_dict())
|
||
→ FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span}
|
||
→ THEMES[<theme>].factors [{key, weight}]
|
||
→ compute_theme_surprises() picks it up with zero scoring-fn change
|
||
```
|
||
Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by `thai_trade`.
|
||
|
||
**Current state (verified, baseline green: 352/352 tests):**
|
||
- 7 live sources: `bot_tourism`, `auto_credit`, `auto_npl`, `bank_npl`, `energy_thai`, `macro_thai`, `thai_trade`.
|
||
- 17 FACTORS registered. All `value_key` resolve to a real fetched field (locked by `test_every_factor_value_key_resolves_to_a_fetched_field`).
|
||
- Sign convention FIXED this session: `sign` lives only in the factor; theme weights are positive magnitude. Regression-locked by `test_bearish_factors_move_score_the_right_way` (higher NPL ⇒ lower score, etc.).
|
||
- `thai_trade.py` (TradingEconomics current-account) is the reference collector: server-rendered HTML, `_fetch`/`parse`/`fetch_<x>`/`to_dict`, registered in scheduler `_REFRESH_JOBS` + dashboard `_fetch_with_cache` + FACTORS.
|
||
|
||
---
|
||
|
||
## Confirmed Decisions (from user: "วางแผนได้เลย")
|
||
| Decision | Chosen | Rationale |
|
||
|---|---|---|
|
||
| Scope | Phase A (macro-deepening) first, Phase B (sector) next | Fastest correctness win, lowest scrap fragility |
|
||
| New-source gate | Feasibility spike per source BEFORE building collector | Don't ship a wrong/stale/unscrapable series (same gate `thai_trade.py` used) |
|
||
| Integration | Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache | User rule: new data must feed analysis, not just be fetched |
|
||
| Test | One value-key-resolution + one direction test per new source | Locks "used in analysis" invariant |
|
||
|
||
## Pending user decision (gate G0 — before Phase A code)
|
||
1. **Which sources to prioritize** — recommendation in priority order below. User may reorder/substitute.
|
||
2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page).
|
||
|
||
---
|
||
|
||
## Current-State Gap Matrix (per theme, what feeds it today)
|
||
| Theme | Sources today | Gap / weakest link |
|
||
|---|---|---|
|
||
| banks | BOT macro (invest/inflation/NPL) — 3 fields, all BOT | No rate/loan-setting input; NPL only "financial sector" proxy |
|
||
| retail | BOT macro (consumption/inflation/unemployment) + TE imports | No direct retail-sales / consumer-mood series |
|
||
| consumer_staples | BOT macro + TE imports | Same as retail |
|
||
| telecom_it | BOT macro (consumption/investment) + TE exports | No telecom-specific (subs/data) series |
|
||
| property | BOT macro (invest/consumption/inflation) + TE imports | No real-estate-specific (transfer/mortgage) series |
|
||
| nonbank_finance | BOT macro + auto NPL + unemployment | No household-credit / consumer-loan series |
|
||
| healthcare | BOT macro (consumption/unemployment) | No healthcare/tourism-medical series |
|
||
| utilities | BOT macro (mfg) + TOP margin | No electricity-demand / generation series |
|
||
| tourism | BOT tourism + macro consumption + TE current acct | Covered well; optional: hotel occupancy |
|
||
| auto_credit | TE vehicle + BOT auto NPL | Covered well |
|
||
| refining_energy / exploration / petrochem | TOP + macro mfg/inflation + TE exports | Only TOP company; no commodity/oil price |
|
||
|
||
---
|
||
|
||
## Phase A — deepen macro-proxy themes (priority order)
|
||
Each is a collector + FACTORS entry ×N + THEME wiring + tests.
|
||
|
||
### A1. BOT policy/loan-rate + credit — banks, nonbank_finance
|
||
- Target: BOT monetary-policy / rate page (e.g. `bot.or.th` policy rate) + BOT credit/loan-growth report.
|
||
- Factors: `bank_policy_rate` (sign −1 for banks? higher rate squeezes demand), `bank_loan_growth_yoy` (sign +1).
|
||
- Feasibility: spike must confirm a scrapeable numeric series on a BOT page.
|
||
|
||
### A2. Household / consumer credit — nonbank_finance (+ banks)
|
||
- Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report.
|
||
- Factor: `consumer_credit_yoy` (sign +1), `consumer_credit_npl` (sign −1).
|
||
- Feasibility: spike on BOT statistics page.
|
||
|
||
### A3. Retail sales index — retail, consumer_staples
|
||
- Target: BOT or TradingEconomics "retail sales" Thailand YoY.
|
||
- Factor: `retail_sales_yoy` (sign +1).
|
||
- Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit).
|
||
|
||
### A4. Consumer confidence — retail, consumer_staples, nonbank_finance
|
||
- Target: UTCC / Kasikorn Research consumer-confidence index (free HTML).
|
||
- Factor: `consumer_confidence` (sign +1).
|
||
- Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence".
|
||
|
||
## Phase B — sector-specific (IMPLEMENTED 2026-08-29 via single-page snapshots)
|
||
### B1. Property: `te_property_prices` (residential property prices % YoY) — DONE, feeds property theme
|
||
### B3. Energy breadth: `energy_irpc` (IRPC net margin, 3M26 +10.27%) — DONE, feeds refining_energy/exploration/utilities
|
||
### B4. Telecom/backdrop: `te_business_confidence` — DONE, feeds telecom_it + property + healthcare
|
||
### B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence)
|
||
Added to `te_thailand.py` + new `energy_irpc.py`, same reviewed pattern. Full suite 368.
|
||
|
||
## Deferred / blocked by feasibility (2026-08-29 spike results — all JS-rendered or anti-bot)
|
||
- **REIC** (property transfer): JS SPA, data loads via XHR — not plain-HTML scrapable.
|
||
Would need browser_exec or `har-derived-api-client` (XHR reverse-engineering).
|
||
- **EPPO** (utilities electricity): WordPress/JS pages, no static numeric table.
|
||
- **NBTC** (telecom data): HTTP 403 anti-bot block.
|
||
- **PTTEP** (energy): JS shell (no server-rendered tables); PTT/BCP URLs 404/DNS.
|
||
Only **IRPC** among the energy names exposed a server-rendered financial table.
|
||
These are NOT quick plain-HTML collectors — they need a browser/XHR approach or a
|
||
logged-in/authorized session. Do them as a separate effort if the analysis needs
|
||
them, not as simple additions to this collector family.
|
||
|
||
## Status updates (2026-08-29, follow-up asks)
|
||
- **Flexible scoring**: board no longer crashes on any single source failure —
|
||
`_fetch_with_cache` degrades (returns {} → theme drops that source; previous
|
||
good value kept as stale by the daily cache). Verified all-sources-down builds
|
||
13 themes.
|
||
- **Per-source calc detail**: `symbolDetail.factor_sources` shows, per theme, each
|
||
factor's source → raw → normalized → weight → contribution (audit trail for the
|
||
owner to tune weights). Also exposed in `/api/v1/dashboard` as `factor_sources`.
|
||
- **HAR feasibility (deferred sources)**: captured REIC via `har-derived-api-client`
|
||
(Playwright drove the JS SPA → HAR → derived XHR `POST /Home/Web_All_Num_View`).
|
||
Method WORKS and endpoint is derivable, but the homepage XHR returned an empty
|
||
body — real property data needs a deeper interaction (navigate to a Transfer
|
||
page and click to load its data). A full REIC collector is a larger follow-up,
|
||
not a quick add. NBTC's 403 is an IP/fingerprint anti-bot block that HAR replay
|
||
(plain HTTP) likely canNOT bypass — skip NBTC unless a session/credential exists.
|
||
|
||
## REIC deep-dive verdict (2026-08-29 — spike gate result)
|
||
THOROUGHLY tested via Playwright + HAR:
|
||
- No reusable JSON XHR — `Web_All_Num_View` returns an **empty body** (status 200,
|
||
size 0) on both the homepage and the Transfer page.
|
||
- The homepage shows "โอนกรรมสิทธิ์อสังหาริมทรัพย์ ทั่วประเทศ มิ.ย.69 และ Q2/69" as a
|
||
**label/link only**, not an inline numeric value.
|
||
- The actual transfer numbers (17.6% growth, unit counts) live on **login/member-
|
||
gated detail pages** or JS-rendered charts that don't put the raw number in the
|
||
DOM without a session.
|
||
- Transfer sub-page `/Product/Transfer/1/71/1` returns a near-empty body (673B).
|
||
Verdict: a low-cost REIC collector is **NOT feasible** (would require full
|
||
Playwright-in-Docker on every refresh + a membership login). Property theme already
|
||
has `te_property_prices` (TE residential +1.26% YoY) as a clean real source, so
|
||
REIC is passed on rather than forcing a fragile collector. Recorded for reference.
|
||
|
||
|
||
|
||
---
|
||
|
||
## Task Breakdown
|
||
### Task A0 — Feasibility spikes (gate, not build)
|
||
For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under `backend/scripts/spike_<source>.py`; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value.
|
||
- Exit: a table of "source → scrapable? → value → period".
|
||
|
||
### Task A1 — BOT rate/loan + credit factors
|
||
- Add module `backend/app/bot_rates.py` (or extend existing) with Snapshot + `fetch_<x>()` + `.to_dict()`.
|
||
- Register FACTORS (policy_rate, loan_growth…) + wire into `banks`/`nonbank_finance` THEMES.
|
||
- Add `_REFRESH_JOBS` row + dashboard cache line.
|
||
- Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand).
|
||
- Verify: full backend suite green; `/api/v1/dashboard` themes show the new factors in `sources`.
|
||
|
||
### Task A2 — Household/consumer credit (nonbank_finance)
|
||
- Same pattern as A1; factor `consumer_credit_yoy` (+1), `consumer_credit_npl` (−1) into `nonbank_finance` (+ maybe banks).
|
||
- Tests + suite green.
|
||
|
||
### Task A3 — Retail sales (retail, consumer_staples)
|
||
- Collector on TE Thailand retail-sales page; factor `retail_sales_yoy` (+1) into retail + consumer_staples.
|
||
- Tests + suite green.
|
||
|
||
### Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance)
|
||
- Collector; factor `consumer_confidence` (+1). Use TE fallback if UTCC is paywalled.
|
||
- Tests + suite green.
|
||
|
||
### Task A5 — Frontend macro chips / theme cards
|
||
- Add the new read values to `App.vue` theme cards / macro chips (single-colour, no new library, follow existing `theme-read-value` pattern).
|
||
- `npm run build` green; visual check at 320×568 and 500×768 (mobile single-column).
|
||
|
||
### Task B1..B5 — Phase B (only after Phase A accepted)
|
||
- Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests.
|
||
|
||
---
|
||
|
||
## Acceptance (definition of done)
|
||
1. Every new FACTOR appears in ≥1 `THEMES[].factors` and is referenced by dashboard `_build_sources` (auto from registry) — **no source shows in the table without feeding a surprise**.
|
||
2. `test_every_factor_value_key_resolves_to_a_fetched_field` still passes for every new factor (value_key is a real fetched field).
|
||
3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score).
|
||
4. Full backend suite green (was 352) + `npm run build` green.
|
||
5. Scheduler `_REFRESH_JOBS` + dashboard cache include every new source.
|
||
6. `/api/v1/dashboard` sources table length tracks FACTORS count; new theme reads appear on the dashboard.
|
||
|
||
## Open questions for G0
|
||
- Priority/order of A1–A4 (recommended order above; user may reorder).
|
||
- Source substitution if a spike fails: fallback list provided per task.
|
||
- Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed?
|