Files
set50-system/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md

4.5 KiB
Raw Blame History

2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug

Scope

Two-part request from the owner:

  1. More + genuinely-used data sources per theme — the owner felt the source count was too few and insisted any newly-fetched data must feed the analysis (theme surprise), not just the provenance table.
  2. Frontend padding/margin issues — several panels had misaligned spacing.

What changed (all verified)

Area Change Evidence
factors.py Wired 2 fetched-but-unused BOT fields into the registry: macro_core_inflation (core inflation, sign 1) and macro_unemployment (sign 1) → they now actually move themes Before, macro_thai.to_dict() emitted 7 fields but only 5 were registered factors — 2 were dead display data
themes.py Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights) direction tests
Sign bug fix Found + fixed a double-negative: several factors carry sign: -1 (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in sign. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it. test_bearish_factors_move_score_the_right_way
NEW te_thailand.py New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8
themes.py Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence) direction tests
scheduler.py / dashboard.py Registered te_thailand in _REFRESH_JOBS + dashboard cached-fetch + _build_sources provenance dashboard live build: 8 sources
frontend/App.vue Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP) build clean
frontend/style.css padding/margin + undefined-variable fix — added missing design tokens (--card/--foreground/--accent/--font/...) that the theme/modal components reference but :root never declared (they rendered transparent/wrong-color); zeroed section-panel padding so .signal-header is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels) build clean

Key decisions

  • One new module (te_thailand.py) covers 4 plan items (A1 rate/credit, A2 consumer credit, A3 retail sales, A4 confidence) because all four series live on just two TradingEconomics pages — avoids 4 fragile scrapers.
  • Every source must feed a surprise: tightened test_every_factor_value_key_resolves_to_a_fetched_field so a factor whose value_key the collector never emits fails the suite — no dead factors.
  • Sign convention locked: sign = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly.

Verified (evidence)

  • Full backend suite: 360 tests OK (up from 352 — 8 new test_te_thailand).
  • Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise 0.165 (correctly negative from retail-sales 14.5% YoY) vs old BOT-only (positive).
  • Frontend npm run build clean.
  • Independent code review: see delegation verdict (fail-closed).

Phase B (same day, follow-up "ทำต่อได้เลย")

Extended the same te_thailand module with 2 more TradingEconomics pages:

  • te_property_prices (residential property prices +1.26% YoY) → property theme
  • te_business_confidence (46.7) → telecom_it + property + healthcare +2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now 362 tests OK; frontend build clean; live fetch confirmed (1.26 / 46.7). Cleared a stale daily-cache te_thailand entry so the new fields show immediately.

Backlog (needs dedicated scrapers, not single-page snapshots)

REIC property supply, EPPO electricity (utilities), PTT/PTTEP/BCP quarterly breadth (energy), NBTC subscriber/data (telecom). Marked deferred in the plan.

  • Note: a concurrent process also landed thai_trade.py (external-sector exports/imports/current-account) and external_* factors mid-session; its 3 initially-broken tests were fixed to reach the 360-green baseline here.