set50-system/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md

# 2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug

## Scope
Two-part request from the owner:
1. **More + genuinely-used data sources per theme** — the owner felt the source
   count was too few and insisted any newly-fetched data must feed the analysis
   (theme surprise), not just the provenance table.
2. **Frontend padding/margin issues** — several panels had misaligned spacing.

## What changed (all verified)
| Area | Change | Evidence |
|---|---|---|
| `factors.py` | Wired 2 **fetched-but-unused** BOT fields into the registry: `macro_core_inflation` (core inflation, sign −1) and `macro_unemployment` (sign −1) → they now actually move themes | Before, `macro_thai.to_dict()` emitted 7 fields but only 5 were registered factors — 2 were dead display data |
| `themes.py` | Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights) | direction tests |
| **Sign bug fix** | Found + fixed a **double-negative**: several factors carry `sign: -1` (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in `sign`. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it. | `test_bearish_factors_move_score_the_right_way` |
| **NEW `te_thailand.py`** | New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence | live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8 |
| `themes.py` | Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence) | direction tests |
| `scheduler.py` / `dashboard.py` | Registered `te_thailand` in `_REFRESH_JOBS` + dashboard cached-fetch + `_build_sources` provenance | dashboard live build: 8 sources |
| `frontend/App.vue` | Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP) | build clean |
| `frontend/style.css` | **padding/margin + undefined-variable fix** — added missing design tokens (`--card/--foreground/--accent/--font/...`) that the theme/modal components reference but `:root` never declared (they rendered transparent/wrong-color); zeroed section-panel padding so `.signal-header` is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels) | build clean |

## Key decisions
- **One new module** (`te_thailand.py`) covers 4 plan items (A1 rate/credit, A2
  consumer credit, A3 retail sales, A4 confidence) because all four series live
  on just two TradingEconomics pages — avoids 4 fragile scrapers.
- **Every source must feed a surprise**: tightened `test_every_factor_value_key_resolves_to_a_fetched_field` so a factor whose `value_key` the collector never emits fails the suite — no dead factors.
- Sign convention locked: `sign` = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly.

## Verified (evidence)
- Full backend suite: **360 tests OK** (up from 352 — 8 new `test_te_thailand`).
- Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise −0.165
  (correctly negative from retail-sales −14.5% YoY) vs old BOT-only (positive).
- Frontend `npm run build` clean.
- Independent code review: see delegation verdict (fail-closed).

## Phase B (same day, follow-up "ทำต่อได้เลย")
Extended the same `te_thailand` module with 2 more TradingEconomics pages:
- `te_property_prices` (residential property prices +1.26% YoY) → **property** theme
- `te_business_confidence` (46.7) → **telecom_it + property + healthcare**
+2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now
**362 tests OK**; frontend build clean; live fetch confirmed (1.26 / 46.7).
Cleared a stale daily-cache `te_thailand` entry so the new fields show immediately.

## Phase C (same day, follow-up "ทำ phase B ต่อได้เลย" — energy breadth)
Feasibility spike of the remaining Phase B sources found most are JS-rendered or
anti-bot (recorded in plan), but **IRPC** performance-highlights is server-rendered
and clean. Added:
- `backend/app/energy_irpc.py` — collector parsing IRPC net-profit/EBITDA/ROE
  margin rows [2024,2025,3M26], latest period (3M26: net margin **+10.27%**)
- factor `energy_irpc_net_margin` (sign +1) wired into refining_energy /
  exploration / utilities (2nd real Thai refiner beyond TOP)
- scheduler job + dashboard fetch + `_build_sources` row (now **9 sources**)
- tests: parse (incl. paren-negatives), value-key resolution, direction
- frontend theme card shows "กำไรสุทธิ IRPC 10.27%"
Full suite **368 tests OK**; frontend build clean; live dashboard refining_energy
surprise 0.764 driven partly by IRPC margin.
Independent review `deleg_474555e4` **passed: true** (parser/paren/value-key/wiring
all correct, no security/logic errors). Suggestion applied: comment noting
ebitda/roe are display-only. Author also flagged that a single upstream page
failure (strict `_fetch_with_cache` → DashboardError) takes down the whole board —
consistent with the documented no-fixture-fallback policy; left as an open
decision, not silently changed.

Deferred (feasibility blocked): REIC (JS SPA/XHR), EPPO (JS/WordPress), NBTC (403
anti-bot), PTTEP (JS shell), PTT/BCP (404/DNS). Need browser/XHR approach, not
plain-HTML — recorded in the plan as a separate effort.

## Phase D (same day — 3 follow-up asks)
1. **Flexible scoring (Q2)**: `_fetch_with_cache` now degrades instead of raising
   `DashboardError` — source down + no cache = return {} (theme drops that
   source's factors); previously-good value present = daily cache returns stale.
   Board no longer crashes on any single upstream failure (verified: all-sources-
   down still builds 13 themes). `DashboardError` class now unused by build().
2. **Per-source calc detail (Q3)**: new `themes.factor_source_breakdown(fetched,
   theme)` surfaced in dashboard as `fetch_data` + `factor_sources`, and in the
   per-symbol modal as `symbolDetail.factor_sources` — the owner sees, per theme,
   each factor's source → raw → normalized → weight → contribution (e.g. retail:
   te_thailand ยอดขายปลีก -14.5 → -1.0 × 0.7 = -0.7). Backend test added.
3. **HAR feasibility for deferred sources (Q1)**: captured REIC via
   `har-derived-api-client` (Playwright drive → HAR → derived XHR endpoint
   `POST /Home/Web_All_Num_View`). Spike shows the method WORKS (browser drives
   the JS SPA, XHR endpoint derivable) BUT the homepage XHR returned an empty
   body — actual property data needs a deeper interaction (a real Transfer page
   click), so a full REIC collector is a larger follow-up, not a quick add.
Finishing: commit Q2+Q3 with suite 369 green; Q1 recorded as feasible-but-
needs-deeper-capture and left as a decision for the owner.

- Note: a **concurrent process** also landed `thai_trade.py` (external-sector
  exports/imports/current-account) and external_* factors mid-session; its 3
  initially-broken tests were fixed to reach the 360-green baseline here.

## Flexible PIT backtest (owner rule — 2026-08-30)
Owner clarified the backtest contract: it must run as soon as there is enough
data to estimate an investment, NOT only when EVERY factor has PIT history.
Some sources may not provide deep history — scoring is deliberately flexible
(a theme scores from whatever subset of factors was knowable that day).

Implemented (commit 2026-08-30):
- `backtest_readiness.evaluate_readiness`: readiness is now a *usable* window
  (price + Siamchart + >=1 released factor), not all-factors-present. Missing
  factors are still reported in `missing` for transparency but no longer block.
  `recommended_start` = oldest executable price (the oldest history held);
  `recommended_end` = last complete trading day (yesterday Bangkok).
- `pit_scorer.theme_surprise_report`: flexible — skips any factor with no release
  by `as_of` (no blanket theme block); `blocked` only when NO factor has a value.
  `pit_meta.partial_pit` reflects themes that scored from a partial subset.
- Verified: `/api/v1/backtest/readiness` → ready=true, recommended 2024-01-03 →
  2026-08-29. `POST /api/v1/backtest/run` (default window) → 201 full result
  (final_equity 1,117,243.95 on 1M), no 400 from missing factors.
- Tests: test_backtest_readiness updated to earliest-runnable semantics; full
  suite 369 green.

## Capital-allocation selection rework (owner rule — 2026-08-31)
Owner clarified what the 3 buckets mean and flagged that the old selection
(rank by `combined` = 60/40 theme+siamchart where siamchart was EPS-growth
dominated) picked names like BTS whose EPS was up 137% but whose PRICE was flat/
falling — NOT the owner's "ทำกำไร" (price likely to rise in 3-6 months).

New definition implemented:
- "ทำกำไร" is now measured by a **price-trend (momentum) score**, not EPS growth.
  Added `themes.price_trend_score()` blending ~3/6/12-month momentum (trading
  days), z-scored across the universe (heavy weight on 3/6m per the 3-6 month
  tenure).
- Buckets 1 & 2 rank by momentum, **gated on `theme_signal > 0`** (mean surprise
  of the symbol's themes). theme_signal=None (backtest path) is not gated, so
  the PIT backtest still allocates.
- Bucket 3 unchanged: highest dividend yield, ignores score/momentum.
- Suggestion endpoint passes real momentum + theme_signal from the live board.
- Verified: suggestion now picks CRC+BEM (dividend, rising price) in bucket 1;
  PTT/MINT (falling price, theme positive) slide to bucket 3 by yield, not
  bucket 1. Full suite 372 green (3 new momentum/gate tests).

## Price-snapshot load fix + momentum gate (owner "เพื่อทดสอบ logic" — 2026-08-31)
- `load_price_snapshot` was picking the LAST snapshot by FILENAME (lexicographic),
  which selected a stale 9-symbol collection over the full 50-symbol universe
  (`2024-01-01..` sorts after `2023-08-28..`). Now it picks the snapshot with the
  latest `source.retrieved_at`, so the full 50-symbol SET50 universe loads.
- Suggestion now allocates across all 50 names; verified it picks rising-price,
  theme-positive dividend names in B1 (BGRIM, TTB), non-dividend rising in B2
  (BANPU), and yield-top names in B3 (ADVANC, SCB, LH).
- Added momentum>0 gate to the profit buckets (a falling-price name must NOT be
  picked as "ทำกำไร"), while keeping momentum/theme_signal as Optional so the
  PIT backtest path (which doesn't provide them) still allocates.
- Regression tests: load_price_snapshot picks most-recently-retrieved; negative
  momentum excluded from profit buckets. Full suite 374 green.

## Refactor: momentum is a factor INSIDE the Siamchart formula (owner rule — 2026-08-31)
The owner insisted momentum must be entered INTO the score formula (a first-class
factor), not bolted on outside it. Refactored `themes.py` from 3 loose weight
constants + inline formula repeated in 3 places to a single declarative source:
- `_SIAMCHART_WEIGHTS = {eps_growth:1.5, dividend_yield:2.0, momentum:0.5}` (config).
- `siamchart_raw_score(g,d,m)` = single source of the formula; momentum is a term
  INSIDE it. All three call sites (build_siamchart_score, symbol_breakdown raw,
  symbol_breakdown population) now use it — no more duplicated arithmetic.
- Purely a refactor: outputs unchanged (0.6/0.4 weighting intact), momentum
  already weighted 0.5 inside the formula.
- Added tests: raw score formula incl. momentum term; momentum raises siamchart
  score given identical fundamentals. Full suite 376 green.

## Profit buckets now require a positive combined score (owner rule — 2026-08-31)
The owner flagged that BGRIM/TTB still got recommended in bucket 1 even though
their combined score was negative (-0.029 / -0.066): the profit pool only gated
on momentum>0 + theme>0, so a stock with good price trend but weak fundamentals
(EPS/yield lagging the cohort) could still be suggested.
- `allocate_capital` profit_pool now ALSO requires `combined_score > 0` — a stock
  whose overall score is negative is never recommended in buckets 1/2, while
  still ranking by momentum within that gated pool.
- Verified: suggestion now picks PTTGC (combined +0.209, mom +1.85) in bucket 1
  instead of BGRIM/TTB; BANPU in bucket 2 (combined +0.70); ADVANC/SCB/LH in
  bucket 3 by yield. Full suite 377 green.