# Data Source Expansion Plan — Phase 2 (more Thai sources per theme)

> **For Hermes:** Execute task-by-task. One task in progress at a time; commit by phase; run `requesting-code-review` before delivery. Every new source MUST end wired into `FACTORS` (one dict entry) **and** referenced by at least one `THEMES[].factors` row — so it feeds `compute_theme_surprises()`, never just the source table. Follow the existing collector template (`backend/app/thai_trade.py`, `auto_credit.py`). Run the full backend suite + `npm run build` after each phase.

**Goal:** Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table.

**Architecture (unchanged, declarative):**
```
new source → collector module (fetch_<x>() → Snapshot.to_dict())
          → FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span}
          → THEMES[<theme>].factors [{key, weight}]
          → compute_theme_surprises() picks it up with zero scoring-fn change
```
Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by `thai_trade`.

**Current state (verified, baseline green: 352/352 tests):**
- 7 live sources: `bot_tourism`, `auto_credit`, `auto_npl`, `bank_npl`, `energy_thai`, `macro_thai`, `thai_trade`.
- 17 FACTORS registered. All `value_key` resolve to a real fetched field (locked by `test_every_factor_value_key_resolves_to_a_fetched_field`).
- Sign convention FIXED this session: `sign` lives only in the factor; theme weights are positive magnitude. Regression-locked by `test_bearish_factors_move_score_the_right_way` (higher NPL ⇒ lower score, etc.).
- `thai_trade.py` (TradingEconomics current-account) is the reference collector: server-rendered HTML, `_fetch`/`parse`/`fetch_<x>`/`to_dict`, registered in scheduler `_REFRESH_JOBS` + dashboard `_fetch_with_cache` + FACTORS.

---

## Confirmed Decisions (from user: "วางแผนได้เลย")
| Decision | Chosen | Rationale |
|---|---|---|
| Scope | Phase A (macro-deepening) first, Phase B (sector) next | Fastest correctness win, lowest scrap fragility |
| New-source gate | Feasibility spike per source BEFORE building collector | Don't ship a wrong/stale/unscrapable series (same gate `thai_trade.py` used) |
| Integration | Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache | User rule: new data must feed analysis, not just be fetched |
| Test | One value-key-resolution + one direction test per new source | Locks "used in analysis" invariant |

## Pending user decision (gate G0 — before Phase A code)
1. **Which sources to prioritize** — recommendation in priority order below. User may reorder/substitute.
2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page).

---

## Current-State Gap Matrix (per theme, what feeds it today)
| Theme | Sources today | Gap / weakest link |
|---|---|---|
| banks | BOT macro (invest/inflation/NPL) — 3 fields, all BOT | No rate/loan-setting input; NPL only "financial sector" proxy |
| retail | BOT macro (consumption/inflation/unemployment) + TE imports | No direct retail-sales / consumer-mood series |
| consumer_staples | BOT macro + TE imports | Same as retail |
| telecom_it | BOT macro (consumption/investment) + TE exports | No telecom-specific (subs/data) series |
| property | BOT macro (invest/consumption/inflation) + TE imports | No real-estate-specific (transfer/mortgage) series |
| nonbank_finance | BOT macro + auto NPL + unemployment | No household-credit / consumer-loan series |
| healthcare | BOT macro (consumption/unemployment) | No healthcare/tourism-medical series |
| utilities | BOT macro (mfg) + TOP margin | No electricity-demand / generation series |
| tourism | BOT tourism + macro consumption + TE current acct | Covered well; optional: hotel occupancy |
| auto_credit | TE vehicle + BOT auto NPL | Covered well |
| refining_energy / exploration / petrochem | TOP + macro mfg/inflation + TE exports | Only TOP company; no commodity/oil price |

---

## Phase A — deepen macro-proxy themes (priority order)
Each is a collector + FACTORS entry ×N + THEME wiring + tests.

### A1. BOT policy/loan-rate + credit — banks, nonbank_finance
- Target: BOT monetary-policy / rate page (e.g. `bot.or.th` policy rate) + BOT credit/loan-growth report.
- Factors: `bank_policy_rate` (sign −1 for banks? higher rate squeezes demand), `bank_loan_growth_yoy` (sign +1).
- Feasibility: spike must confirm a scrapeable numeric series on a BOT page.

### A2. Household / consumer credit — nonbank_finance (+ banks)
- Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report.
- Factor: `consumer_credit_yoy` (sign +1), `consumer_credit_npl` (sign −1).
- Feasibility: spike on BOT statistics page.

### A3. Retail sales index — retail, consumer_staples
- Target: BOT or TradingEconomics "retail sales" Thailand YoY.
- Factor: `retail_sales_yoy` (sign +1).
- Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit).

### A4. Consumer confidence — retail, consumer_staples, nonbank_finance
- Target: UTCC / Kasikorn Research consumer-confidence index (free HTML).
- Factor: `consumer_confidence` (sign +1).
- Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence".

## Phase B — sector-specific (IMPLEMENTED 2026-08-29 via single-page snapshots)
### B1. Property: `te_property_prices` (residential property prices % YoY) — DONE, feeds property theme
### B3. Energy breadth: `energy_irpc` (IRPC net margin, 3M26 +10.27%) — DONE, feeds refining_energy/exploration/utilities
### B4. Telecom/backdrop: `te_business_confidence` — DONE, feeds telecom_it + property + healthcare
### B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence)
Added to `te_thailand.py` + new `energy_irpc.py`, same reviewed pattern. Full suite 368.

## Deferred / blocked by feasibility (2026-08-29 spike results — all JS-rendered or anti-bot)
- **REIC** (property transfer): JS SPA, data loads via XHR — not plain-HTML scrapable.
  Would need browser_exec or `har-derived-api-client` (XHR reverse-engineering).
- **EPPO** (utilities electricity): WordPress/JS pages, no static numeric table.
- **NBTC** (telecom data): HTTP 403 anti-bot block.
- **PTTEP** (energy): JS shell (no server-rendered tables); PTT/BCP URLs 404/DNS.
  Only **IRPC** among the energy names exposed a server-rendered financial table.
These are NOT quick plain-HTML collectors — they need a browser/XHR approach or a
logged-in/authorized session. Do them as a separate effort if the analysis needs
them, not as simple additions to this collector family.

## Status updates (2026-08-29, follow-up asks)
- **Flexible scoring**: board no longer crashes on any single source failure —
  `_fetch_with_cache` degrades (returns {} → theme drops that source; previous
  good value kept as stale by the daily cache). Verified all-sources-down builds
  13 themes.
- **Per-source calc detail**: `symbolDetail.factor_sources` shows, per theme, each
  factor's source → raw → normalized → weight → contribution (audit trail for the
  owner to tune weights). Also exposed in `/api/v1/dashboard` as `factor_sources`.
- **HAR feasibility (deferred sources)**: captured REIC via `har-derived-api-client`
  (Playwright drove the JS SPA → HAR → derived XHR `POST /Home/Web_All_Num_View`).
  Method WORKS and endpoint is derivable, but the homepage XHR returned an empty
  body — real property data needs a deeper interaction (navigate to a Transfer
  page and click to load its data). A full REIC collector is a larger follow-up,
  not a quick add. NBTC's 403 is an IP/fingerprint anti-bot block that HAR replay
  (plain HTTP) likely canNOT bypass — skip NBTC unless a session/credential exists.

## REIC deep-dive verdict (2026-08-29 — spike gate result)
THOROUGHLY tested via Playwright + HAR:
- No reusable JSON XHR — `Web_All_Num_View` returns an **empty body** (status 200,
  size 0) on both the homepage and the Transfer page.
- The homepage shows "โอนกรรมสิทธิ์อสังหาริมทรัพย์ ทั่วประเทศ มิ.ย.69 และ Q2/69" as a
  **label/link only**, not an inline numeric value.
- The actual transfer numbers (17.6% growth, unit counts) live on **login/member-
  gated detail pages** or JS-rendered charts that don't put the raw number in the
  DOM without a session.
- Transfer sub-page `/Product/Transfer/1/71/1` returns a near-empty body (673B).
Verdict: a low-cost REIC collector is **NOT feasible** (would require full
Playwright-in-Docker on every refresh + a membership login). Property theme already
has `te_property_prices` (TE residential +1.26% YoY) as a clean real source, so
REIC is passed on rather than forcing a fragile collector. Recorded for reference.



---

## Task Breakdown
### Task A0 — Feasibility spikes (gate, not build)
For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under `backend/scripts/spike_<source>.py`; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value.
- Exit: a table of "source → scrapable? → value → period".

### Task A1 — BOT rate/loan + credit factors
- Add module `backend/app/bot_rates.py` (or extend existing) with Snapshot + `fetch_<x>()` + `.to_dict()`.
- Register FACTORS (policy_rate, loan_growth…) + wire into `banks`/`nonbank_finance` THEMES.
- Add `_REFRESH_JOBS` row + dashboard cache line.
- Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand).
- Verify: full backend suite green; `/api/v1/dashboard` themes show the new factors in `sources`.

### Task A2 — Household/consumer credit (nonbank_finance)
- Same pattern as A1; factor `consumer_credit_yoy` (+1), `consumer_credit_npl` (−1) into `nonbank_finance` (+ maybe banks).
- Tests + suite green.

### Task A3 — Retail sales (retail, consumer_staples)
- Collector on TE Thailand retail-sales page; factor `retail_sales_yoy` (+1) into retail + consumer_staples.
- Tests + suite green.

### Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance)
- Collector; factor `consumer_confidence` (+1). Use TE fallback if UTCC is paywalled.
- Tests + suite green.

### Task A5 — Frontend macro chips / theme cards
- Add the new read values to `App.vue` theme cards / macro chips (single-colour, no new library, follow existing `theme-read-value` pattern).
- `npm run build` green; visual check at 320×568 and 500×768 (mobile single-column).

### Task B1..B5 — Phase B (only after Phase A accepted)
- Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests.

---

## Acceptance (definition of done)
1. Every new FACTOR appears in ≥1 `THEMES[].factors` and is referenced by dashboard `_build_sources` (auto from registry) — **no source shows in the table without feeding a surprise**.
2. `test_every_factor_value_key_resolves_to_a_fetched_field` still passes for every new factor (value_key is a real fetched field).
3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score).
4. Full backend suite green (was 352) + `npm run build` green.
5. Scheduler `_REFRESH_JOBS` + dashboard cache include every new source.
6. `/api/v1/dashboard` sources table length tracks FACTORS count; new theme reads appear on the dashboard.

## Open questions for G0
- Priority/order of A1–A4 (recommended order above; user may reorder).
- Source substitution if a spike fails: fallback list provided per task.
- Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed?