kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
set50-system/.hermes/plans/2026-08-29_data-source-expansion-phase-2.md

12 KiB
Raw Permalink Blame History

Data Source Expansion Plan — Phase 2 (more Thai sources per theme)

For Hermes: Execute task-by-task. One task in progress at a time; commit by phase; run requesting-code-review before delivery. Every new source MUST end wired into FACTORS (one dict entry) and referenced by at least one THEMES[].factors row — so it feeds compute_theme_surprises(), never just the source table. Follow the existing collector template (backend/app/thai_trade.py, auto_credit.py). Run the full backend suite + npm run build after each phase.

Goal: Add genuinely new Thai data sources for the themes that today rely on a single BOT macro proxy (banks, retail, property, telecom_it, nonbank_finance, healthcare, utilities/energy), each wired into the factor engine so it changes theme surprises — not merely the provenance table.

Architecture (unchanged, declarative):

new source → collector module (fetch_<x>() → Snapshot.to_dict())
          → FACTORS entry {name_th, source, frequency, fetch, value_key, sign, weight, center, span}
          → THEMES[<theme>].factors [{key, weight}]
          → compute_theme_surprises() picks it up with zero scoring-fn change

Adding a factor = 1 registry entry + (optionally) a theme factor line. No scoring random. This is already proven by thai_trade.

Current state (verified, baseline green: 352/352 tests):

  • 7 live sources: bot_tourism, auto_credit, auto_npl, bank_npl, energy_thai, macro_thai, thai_trade.
  • 17 FACTORS registered. All value_key resolve to a real fetched field (locked by test_every_factor_value_key_resolves_to_a_fetched_field).
  • Sign convention FIXED this session: sign lives only in the factor; theme weights are positive magnitude. Regression-locked by test_bearish_factors_move_score_the_right_way (higher NPL ⇒ lower score, etc.).
  • thai_trade.py (TradingEconomics current-account) is the reference collector: server-rendered HTML, _fetch/parse/fetch_<x>/to_dict, registered in scheduler _REFRESH_JOBS + dashboard _fetch_with_cache + FACTORS.

Confirmed Decisions (from user: "วางแผนได้เลย")

Decision Chosen Rationale
Scope Phase A (macro-deepening) first, Phase B (sector) next Fastest correctness win, lowest scrap fragility
New-source gate Feasibility spike per source BEFORE building collector Don't ship a wrong/stale/unscrapable series (same gate thai_trade.py used)
Integration Every new FACTOR must appear in ≥1 THEMES factor line + scheduler + dashboard cache User rule: new data must feed analysis, not just be fetched
Test One value-key-resolution + one direction test per new source Locks "used in analysis" invariant

Pending user decision (gate G0 — before Phase A code)

  1. Which sources to prioritize — recommendation in priority order below. User may reorder/substitute.
  2. Confirm each target URL is acceptable (some BOT/REIC/EPPO pages are heavy; a couple may need a different page).

Current-State Gap Matrix (per theme, what feeds it today)

Theme Sources today Gap / weakest link
banks BOT macro (invest/inflation/NPL) — 3 fields, all BOT No rate/loan-setting input; NPL only "financial sector" proxy
retail BOT macro (consumption/inflation/unemployment) + TE imports No direct retail-sales / consumer-mood series
consumer_staples BOT macro + TE imports Same as retail
telecom_it BOT macro (consumption/investment) + TE exports No telecom-specific (subs/data) series
property BOT macro (invest/consumption/inflation) + TE imports No real-estate-specific (transfer/mortgage) series
nonbank_finance BOT macro + auto NPL + unemployment No household-credit / consumer-loan series
healthcare BOT macro (consumption/unemployment) No healthcare/tourism-medical series
utilities BOT macro (mfg) + TOP margin No electricity-demand / generation series
tourism BOT tourism + macro consumption + TE current acct Covered well; optional: hotel occupancy
auto_credit TE vehicle + BOT auto NPL Covered well
refining_energy / exploration / petrochem TOP + macro mfg/inflation + TE exports Only TOP company; no commodity/oil price

Phase A — deepen macro-proxy themes (priority order)

Each is a collector + FACTORS entry ×N + THEME wiring + tests.

A1. BOT policy/loan-rate + credit — banks, nonbank_finance

  • Target: BOT monetary-policy / rate page (e.g. bot.or.th policy rate) + BOT credit/loan-growth report.
  • Factors: bank_policy_rate (sign −1 for banks? higher rate squeezes demand), bank_loan_growth_yoy (sign +1).
  • Feasibility: spike must confirm a scrapeable numeric series on a BOT page.

A2. Household / consumer credit — nonbank_finance (+ banks)

  • Target: BOT consumer-loan (สินเชื่อส่วนบุคคล/บัตรเครดิต) report.
  • Factor: consumer_credit_yoy (sign +1), consumer_credit_npl (sign −1).
  • Feasibility: spike on BOT statistics page.

A3. Retail sales index — retail, consumer_staples

  • Target: BOT or TradingEconomics "retail sales" Thailand YoY.
  • Factor: retail_sales_yoy (sign +1).
  • Feasibility: TE has a Thailand retail-sales page (same inc as auto_credit).

A4. Consumer confidence — retail, consumer_staples, nonbank_finance

  • Target: UTCC / Kasikorn Research consumer-confidence index (free HTML).
  • Factor: consumer_confidence (sign +1).
  • Feasibility: spike — some sources require login; fallback to TradingEconomics "consumer confidence".

Phase B — sector-specific (IMPLEMENTED 2026-08-29 via single-page snapshots)

B1. Property: te_property_prices (residential property prices % YoY) — DONE, feeds property theme

B3. Energy breadth: energy_irpc (IRPC net margin, 3M26 +10.27%) — DONE, feeds refining_energy/exploration/utilities

B4. Telecom/backdrop: te_business_confidence — DONE, feeds telecom_it + property + healthcare

B5. Healthcare: consumer/business backdrop wired in — DONE (macro + business confidence)

Added to te_thailand.py + new energy_irpc.py, same reviewed pattern. Full suite 368.

Deferred / blocked by feasibility (2026-08-29 spike results — all JS-rendered or anti-bot)

  • REIC (property transfer): JS SPA, data loads via XHR — not plain-HTML scrapable. Would need browser_exec or har-derived-api-client (XHR reverse-engineering).
  • EPPO (utilities electricity): WordPress/JS pages, no static numeric table.
  • NBTC (telecom data): HTTP 403 anti-bot block.
  • PTTEP (energy): JS shell (no server-rendered tables); PTT/BCP URLs 404/DNS. Only IRPC among the energy names exposed a server-rendered financial table. These are NOT quick plain-HTML collectors — they need a browser/XHR approach or a logged-in/authorized session. Do them as a separate effort if the analysis needs them, not as simple additions to this collector family.

Status updates (2026-08-29, follow-up asks)

  • Flexible scoring: board no longer crashes on any single source failure — _fetch_with_cache degrades (returns {} → theme drops that source; previous good value kept as stale by the daily cache). Verified all-sources-down builds 13 themes.
  • Per-source calc detail: symbolDetail.factor_sources shows, per theme, each factor's source → raw → normalized → weight → contribution (audit trail for the owner to tune weights). Also exposed in /api/v1/dashboard as factor_sources.
  • HAR feasibility (deferred sources): captured REIC via har-derived-api-client (Playwright drove the JS SPA → HAR → derived XHR POST /Home/Web_All_Num_View). Method WORKS and endpoint is derivable, but the homepage XHR returned an empty body — real property data needs a deeper interaction (navigate to a Transfer page and click to load its data). A full REIC collector is a larger follow-up, not a quick add. NBTC's 403 is an IP/fingerprint anti-bot block that HAR replay (plain HTTP) likely canNOT bypass — skip NBTC unless a session/credential exists.

REIC deep-dive verdict (2026-08-29 — spike gate result)

THOROUGHLY tested via Playwright + HAR:

  • No reusable JSON XHR — Web_All_Num_View returns an empty body (status 200, size 0) on both the homepage and the Transfer page.
  • The homepage shows "โอนกรรมสิทธิ์อสังหาริมทรัพย์ ทั่วประเทศ มิ.ย.69 และ Q2/69" as a label/link only, not an inline numeric value.
  • The actual transfer numbers (17.6% growth, unit counts) live on login/member- gated detail pages or JS-rendered charts that don't put the raw number in the DOM without a session.
  • Transfer sub-page /Product/Transfer/1/71/1 returns a near-empty body (673B). Verdict: a low-cost REIC collector is NOT feasible (would require full Playwright-in-Docker on every refresh + a membership login). Property theme already has te_property_prices (TE residential +1.26% YoY) as a clean real source, so REIC is passed on rather than forcing a fragile collector. Recorded for reference.

Task Breakdown

Task A0 — Feasibility spikes (gate, not build)

For each candidate URL, fetch + confirm a stable numeric series parses. Write a throwaway script under backend/scripts/spike_<source>.py; record OK/FAIL + exact as_of in the plan log. Only pass a source to A1..A4 if its spike yields a current, non-stale value.

  • Exit: a table of "source → scrapable? → value → period".

Task A1 — BOT rate/loan + credit factors

  • Add module backend/app/bot_rates.py (or extend existing) with Snapshot + fetch_<x>() + .to_dict().
  • Register FACTORS (policy_rate, loan_growth…) + wire into banks/nonbank_finance THEMES.
  • Add _REFRESH_JOBS row + dashboard cache line.
  • Tests: parse test (fixture HTML), value-key-resolution, direction test (higher loan growth ⇒ higher banks surprise; higher rate policy ⇒ lower demand).
  • Verify: full backend suite green; /api/v1/dashboard themes show the new factors in sources.

Task A2 — Household/consumer credit (nonbank_finance)

  • Same pattern as A1; factor consumer_credit_yoy (+1), consumer_credit_npl (−1) into nonbank_finance (+ maybe banks).
  • Tests + suite green.

Task A3 — Retail sales (retail, consumer_staples)

  • Collector on TE Thailand retail-sales page; factor retail_sales_yoy (+1) into retail + consumer_staples.
  • Tests + suite green.

Task A4 — Consumer confidence (retail, consumer_staples, nonbank_finance)

  • Collector; factor consumer_confidence (+1). Use TE fallback if UTCC is paywalled.
  • Tests + suite green.

Task A5 — Frontend macro chips / theme cards

  • Add the new read values to App.vue theme cards / macro chips (single-colour, no new library, follow existing theme-read-value pattern).
  • npm run build green; visual check at 320×568 and 500×768 (mobile single-column).

Task B1..B5 — Phase B (only after Phase A accepted)

  • Same per-source pattern; each with spike, collector, FACTORS+THEME wiring, scheduler row, dashboard cache, tests.

Acceptance (definition of done)

  1. Every new FACTOR appears in ≥1 THEMES[].factors and is referenced by dashboard _build_sources (auto from registry) — no source shows in the table without feeding a surprise.
  2. test_every_factor_value_key_resolves_to_a_fetched_field still passes for every new factor (value_key is a real fetched field).
  3. Direction tests assert the intended sign for each new factor (e.g. higher loan growth ⇒ higher bank theme score).
  4. Full backend suite green (was 352) + npm run build green.
  5. Scheduler _REFRESH_JOBS + dashboard cache include every new source.
  6. /api/v1/dashboard sources table length tracks FACTORS count; new theme reads appear on the dashboard.

Open questions for G0

  • Priority/order of A1–A4 (recommended order above; user may reorder).
  • Source substitution if a spike fails: fallback list provided per task.
  • Should B1–B5 (Phase B) be planned into this same milestone or a separate one after Phase A ships and is reviewed?
Reference in New Issue View Git Blame Copy Permalink