set50-system/.hermes/plans/2026-08-29_additional-data-sources-per-theme.md

Architecture Plan — Additional Data Sources per Theme + Use-in-Analysis Enforcement

Status: SUPERSEDED / largely IMPLEMENTED (2026-08-29). The use-in-analysis invariant + sign fix are done and committed (fcc0da9); the external-sector gap (this plan's BOT report-60 target) is covered by thai_trade.py (TradingEconomics current-account/exports/imports); commercial-loan coverage is provided by te_loan_growth (TE loans-to-fin-corp) and te_consumer_credit. See 2026-08-29_data-source-expansion-phase-2.md for the actual delivered work and the remaining backlog (REIC/EPPO/NBTC/energy-breadth). Author: Macky. Date: 2026-08-29.

Goal

Close the "few sources per theme" gap AND lock the user's rule — every data point fetched must feed analysis, not just display. This is the follow-up the declarative factor engine (2026-08-26 plan) explicitly deferred: "no new collectors yet; sources are added as follow-up."

Two deliverables:

1. Wiring convention that guarantees use-in-analysis — a build/test pattern so a new source cannot be added without being consumed by compute_theme_surprises.
2. Two new high-feasibility Thai sources, wired end-to-end, verified against live pages: BOT Balance of Payments (report 60) and BOT Commercial-bank loans (reportID confirmed at build). A medium-feasibility source (REIC house-price/transfer) is scoped but flagged for a feasibility spike before committing.

Why this is needed (evidence — verified against live sources this session)

• 6 of 13 themes (telecom_it, property, healthcare, consumer_staples, petrochem_materials, retail) rest on one BOT macro page. Source count is genuinely thin.
• Found + fixed (this session, prior commit pending): core_inflation_yoy and unemployment_pct were fetched but never used; worse, every sign:-1 factor (NPL/inflation/unemployment) used a negative theme weight → double-negative → higher NPL raised the theme score. Both fixed in factors.py/themes.py with regression tests. This plan prevents recurrence and adds genuinely new providers, not just new fields on the same page.
• Feasibility verified live: BOT ReportPage.aspx?reportID=60 (Balance of Payments) responds to the exact dgExcel form-POST flow already implemented in bot_tourism.py (200, parseable — exports/imports/current-account series extracted this session). REIC homepage is a portal — data lives on subpages/endpoints (medium feasibility).

Why the current architecture makes this cheap

Adding a factor already needs no scoring-function change (factors.py registry + a theme factor line). The new work is only:

• a collector module exposing .to_dict(),
• registry + theme wiring,
• a scheduler _REFRESH_JOBS entry (so it collects + appears in source-health log),
• a _source_label provenance entry,
• tests.

Decision Gates (approve before code)

• G0 — Approve the 2 source choices + the mandatory-use test contract (below). Exit → build B0.
• G1 — Collector parses a frozen fixture; factor value extracted; used-in-analysis test passes. Exit → wire.
• G2 — /api/v1/dashboard shows the new source row + source_summary factor_keys increases; board surprises change with it. Exit → done.

New collectors

B0 — Thailand External Sector / Trade (thai_trade.py) [high feasibility — VERIFIED current]

• PIVOT from BOT BOP report 60: probing live this session showed the BOT statistics portal's reportID=60 only carries data through 2011 (its drpToYear options cap at 2011xxxx) — stale, not fit for a 2026 platform. Per the plan's gate ("do not ship a factor reading a wrong/stale series") we source the external sector from TradingEconomics Thailand trade, which is live through June 2026 and reuses the exact auto_credit.py scraping pattern (same site/infobox):
  • https://tradingeconomics.com/thailand/current-account → current-account balance (USD mn, June 2026 verified)
  • https://tradingeconomics.com/thailand/exports → exports (USD mn, 2026 verified)
  • https://tradingeconomics.com/thailand/imports → imports (USD mn, 2026 verified)
• Factors (declarative additions):
  • external_current_account — current-account balance → sign +1 (strong external position supportive). Feeds tourism, exploration, petrochem_materials, retail.
  • external_exports — exports USD mn → sign +1. Feeds petrochem_materials, exploration, telecom_it.
  • external_imports — imports USD mn (domestic demand) → sign +1. Feeds retail, consumer_staples, property.
• Frequency: monthly. Scheduler cadence: monthly.
• Test: frozen fixture (one value sentence) → parse → assert current account / exports / imports; used-in-analysis test asserts a change in the fed themes.

B1 — BOT Commercial-bank loans (bot_bank_loans.py) [high feasibility — flow VERIFIED, reportID TBD]

• Same dgExcel adapter, new reportID. The exact BOT report for commercial-bank loans/deposits is confirmed accessible in principle (report 60 proved the flow); its numeric reportID is discovered at build time from BOT's statistics index / report search (do NOT hardcode an unverified ID).
• Fields expected: total loans outstanding, deposits, NPL (commercial banks), loan growth YoY as derived factor.
• Factors:
  • bank_loan_growth — commercial-bank loan growth YoY → sign +1. Feeds banks (primary), nonbank_finance, property (mortgage-linked).
  • bank_deposit_growth — deposit growth → sign +1 (funding base). Feeds banks.
• Frequency: monthly (BOT reports these monthly). Scheduler cadence: monthly.
• Gate: if the found reportID/data is not cleanly loan-related, fall back to BOT "credit by sector" report and re-scope the factor — do not ship a factor reading a wrong series.

B2 — REIC house-price / transfer index (reic_housing.py) [MEDIUM feasibility — SPIKE FIRST]

• Homepage is a portal (verified). The house-price index / residential transfer data live on a subpage or a data endpoint (likely JSON/PDF). Spike task before G0 sign-off on this one: locate the real data URL, confirm it is scrapeable without an API key.
• If feasible → factor reic_housing_price_index / reic_transfer_value_yoy → sign +1 → feeds property.
• If the only clean access is a PDF chart with no numeric series → defer (do not ship a display-only scrape). This is the exact "แค่ดึงข้อมูลมาเฉย ๆ" the user forbids.

Wiring checklist per source (all must land)

1. backend/app/<module>.py — dataclass + to_dict() + fetch_<module>() (+ frozen-fixture parse fn).
2. factors.py — FACTORS entries (source/frequency/fetch/value_key/sign/center/span).
3. themes.py — factor lines with positive weights on the sign-correct themes (the sign lives in the factor).
4. scheduler.py — _REFRESH_JOBS entry (module/fn/fetch_module/frequency) so it collects + shows in the health log.
5. dashboard.py — _build_sources _source_label map entry (provenance table).
6. __init__.py — no endpoint change needed (dashboard drives the UI); verify /api/v1/dashboard picks it up (G2).

Use-in-analysis enforcement (the user's rule, made a test)

Two contracts, added as tests so "fetched but not used" can never silently return:

• Contract A (value_key resolves): every FACTORS.value_key must be a real field the registered fetch module emits. Already added (test_every_factor_value_key_resolves_to_a_fetched_field). Extend it as new modules land.
• Contract B (new factor moves scores): for each new factor, a test feeds a neutral vs a hot value and asserts the target theme surprise changes in the right direction (bearish factors lower it). Pattern already added (test_new_macro_factors_actually_move_scores, test_bearish_factors_move_score_the_right_way). One such test per new factor.

This dual contract is the build guarantee that fulfills "มีแหล่งข้อมูลใหม่ → ต้องใช้ในการวิเคราะห์จริง".

Data Model sketch

# factors.py
"macro_current_account": {"source":"BOT BOP","frequency":"monthly",
                          "fetch":"bot_bop","value_key":"current_account_balance_mb",
                          "sign":1, "center":0.0, "span":80000.0},
"bank_loan_growth": {"source":"BOT CB loans","frequency":"monthly",
                     "fetch":"bot_bank_loans","value_key":"loan_growth_yoy",
                     "sign":1, "center":5.0, "span":10.0},

# themes.py — positive weights; direction is in the factor
"banks": { "factors": [ {"key":"macro_investment","weight":1.0},
                        {"key":"bank_loan_growth","weight":0.8},
                        ... ] },

Non-Goals (v1)

• No new factors on the existing single BOT macro page beyond the two already wired (core inflation, unemployment) — avoid the "one page, many pseudo-independent factors" trap.
• No live trading / MT5 changes. No LLM in scoring. No API-key sources (free/scrapeable only).
• Do not ship a REIC scrape unless a clean numeric series exists.

Acceptance (definition of done)

1. Thailand external-sector (TradingEconomics trade) collector + 3 factors land, verified against a frozen fixture and a live fetch (current through 2026 — not the stale BOT-60 series).
2. BOT commercial-bank-loan collector + factors land (reportID confirmed; else scoped fallback).
3. REIC either lands with a real series or is explicitly deferred (no display-only scrape).
4. Every new factor has Contract A + Contract B tests green.
5. Full backend suite (352 → target ≥ 366) + npm run build pass.
6. /api/v1/dashboard sources + source_summary.factor_keys increase; board surprises visibly move with a hot value.