set50-system/docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md

2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug

Scope

Two-part request from the owner:

1. More + genuinely-used data sources per theme — the owner felt the source count was too few and insisted any newly-fetched data must feed the analysis (theme surprise), not just the provenance table.
2. Frontend padding/margin issues — several panels had misaligned spacing.

What changed (all verified)

Area	Change	Evidence
factors.py	Wired 2 fetched-but-unused BOT fields into the registry: macro_core_inflation (core inflation, sign −1) and macro_unemployment (sign −1) → they now actually move themes	Before, macro_thai.to_dict() emitted 7 fields but only 5 were registered factors — 2 were dead display data
themes.py	Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights)	direction tests
Sign bug fix	Found + fixed a double-negative: several factors carry sign: -1 (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in sign. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it.	test_bearish_factors_move_score_the_right_way
NEW te_thailand.py	New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence	live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8
themes.py	Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence)	direction tests
scheduler.py / dashboard.py	Registered te_thailand in _REFRESH_JOBS + dashboard cached-fetch + _build_sources provenance	dashboard live build: 8 sources
frontend/App.vue	Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP)	build clean
frontend/style.css	padding/margin + undefined-variable fix — added missing design tokens (--card/--foreground/--accent/--font/...) that the theme/modal components reference but :root never declared (they rendered transparent/wrong-color); zeroed section-panel padding so .signal-header is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels)	build clean

Key decisions

• One new module (te_thailand.py) covers 4 plan items (A1 rate/credit, A2 consumer credit, A3 retail sales, A4 confidence) because all four series live on just two TradingEconomics pages — avoids 4 fragile scrapers.
• Every source must feed a surprise: tightened test_every_factor_value_key_resolves_to_a_fetched_field so a factor whose value_key the collector never emits fails the suite — no dead factors.
• Sign convention locked: sign = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly.

Verified (evidence)

• Full backend suite: 360 tests OK (up from 352 — 8 new test_te_thailand).
• Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise −0.165 (correctly negative from retail-sales −14.5% YoY) vs old BOT-only (positive).
• Frontend npm run build clean.
• Independent code review: see delegation verdict (fail-closed).

Phase B (same day, follow-up "ทำต่อได้เลย")

Extended the same te_thailand module with 2 more TradingEconomics pages:

• te_property_prices (residential property prices +1.26% YoY) → property theme
• te_business_confidence (46.7) → telecom_it + property + healthcare +2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now 362 tests OK; frontend build clean; live fetch confirmed (1.26 / 46.7). Cleared a stale daily-cache te_thailand entry so the new fields show immediately.

Phase C (same day, follow-up "ทำ phase B ต่อได้เลย" — energy breadth)

Feasibility spike of the remaining Phase B sources found most are JS-rendered or anti-bot (recorded in plan), but IRPC performance-highlights is server-rendered and clean. Added:

• backend/app/energy_irpc.py — collector parsing IRPC net-profit/EBITDA/ROE margin rows [2024,2025,3M26], latest period (3M26: net margin +10.27%)
• factor energy_irpc_net_margin (sign +1) wired into refining_energy / exploration / utilities (2nd real Thai refiner beyond TOP)
• scheduler job + dashboard fetch + _build_sources row (now 9 sources)
• tests: parse (incl. paren-negatives), value-key resolution, direction
• frontend theme card shows "กำไรสุทธิ IRPC 10.27%" Full suite 368 tests OK; frontend build clean; live dashboard refining_energy surprise 0.764 driven partly by IRPC margin. Independent review deleg_474555e4 passed: true (parser/paren/value-key/wiring all correct, no security/logic errors). Suggestion applied: comment noting ebitda/roe are display-only. Author also flagged that a single upstream page failure (strict _fetch_with_cache → DashboardError) takes down the whole board — consistent with the documented no-fixture-fallback policy; left as an open decision, not silently changed.

Deferred (feasibility blocked): REIC (JS SPA/XHR), EPPO (JS/WordPress), NBTC (403 anti-bot), PTTEP (JS shell), PTT/BCP (404/DNS). Need browser/XHR approach, not plain-HTML — recorded in the plan as a separate effort.

Phase D (same day — 3 follow-up asks)

1. Flexible scoring (Q2): _fetch_with_cache now degrades instead of raising DashboardError — source down + no cache = return {} (theme drops that source's factors); previously-good value present = daily cache returns stale. Board no longer crashes on any single upstream failure (verified: all-sources- down still builds 13 themes). DashboardError class now unused by build().
2. Per-source calc detail (Q3): new themes.factor_source_breakdown(fetched, theme) surfaced in dashboard as fetch_data + factor_sources, and in the per-symbol modal as symbolDetail.factor_sources — the owner sees, per theme, each factor's source → raw → normalized → weight → contribution (e.g. retail: te_thailand ยอดขายปลีก -14.5 → -1.0 × 0.7 = -0.7). Backend test added.
3. HAR feasibility for deferred sources (Q1): captured REIC via har-derived-api-client (Playwright drive → HAR → derived XHR endpoint POST /Home/Web_All_Num_View). Spike shows the method WORKS (browser drives the JS SPA, XHR endpoint derivable) BUT the homepage XHR returned an empty body — actual property data needs a deeper interaction (a real Transfer page click), so a full REIC collector is a larger follow-up, not a quick add. Finishing: commit Q2+Q3 with suite 369 green; Q1 recorded as feasible-but- needs-deeper-capture and left as a decision for the owner.

• Note: a concurrent process also landed thai_trade.py (external-sector exports/imports/current-account) and external_* factors mid-session; its 3 initially-broken tests were fixed to reach the 360-green baseline here.

Flexible PIT backtest (owner rule — 2026-08-30)

Owner clarified the backtest contract: it must run as soon as there is enough data to estimate an investment, NOT only when EVERY factor has PIT history. Some sources may not provide deep history — scoring is deliberately flexible (a theme scores from whatever subset of factors was knowable that day).

Implemented (commit 2026-08-30):

• backtest_readiness.evaluate_readiness: readiness is now a usable window (price + Siamchart + >=1 released factor), not all-factors-present. Missing factors are still reported in missing for transparency but no longer block. recommended_start = oldest executable price (the oldest history held); recommended_end = last complete trading day (yesterday Bangkok).
• pit_scorer.theme_surprise_report: flexible — skips any factor with no release by as_of (no blanket theme block); blocked only when NO factor has a value. pit_meta.partial_pit reflects themes that scored from a partial subset.
• Verified: /api/v1/backtest/readiness → ready=true, recommended 2024-01-03 → 2026-08-29. POST /api/v1/backtest/run (default window) → 201 full result (final_equity 1,117,243.95 on 1M), no 400 from missing factors.
• Tests: test_backtest_readiness updated to earliest-runnable semantics; full suite 369 green.

Capital-allocation selection rework (owner rule — 2026-08-31)

Owner clarified what the 3 buckets mean and flagged that the old selection (rank by combined = 60/40 theme+siamchart where siamchart was EPS-growth dominated) picked names like BTS whose EPS was up 137% but whose PRICE was flat/ falling — NOT the owner's "ทำกำไร" (price likely to rise in 3-6 months).

New definition implemented:

• "ทำกำไร" is now measured by a price-trend (momentum) score, not EPS growth. Added themes.price_trend_score() blending ~3/6/12-month momentum (trading days), z-scored across the universe (heavy weight on 3/6m per the 3-6 month tenure).
• Buckets 1 & 2 rank by momentum, gated on theme_signal > 0 (mean surprise of the symbol's themes). theme_signal=None (backtest path) is not gated, so the PIT backtest still allocates.
• Bucket 3 unchanged: highest dividend yield, ignores score/momentum.
• Suggestion endpoint passes real momentum + theme_signal from the live board.
• Verified: suggestion now picks CRC+BEM (dividend, rising price) in bucket 1; PTT/MINT (falling price, theme positive) slide to bucket 3 by yield, not bucket 1. Full suite 372 green (3 new momentum/gate tests).

Price-snapshot load fix + momentum gate (owner "เพื่อทดสอบ logic" — 2026-08-31)

• load_price_snapshot was picking the LAST snapshot by FILENAME (lexicographic), which selected a stale 9-symbol collection over the full 50-symbol universe (2024-01-01.. sorts after 2023-08-28..). Now it picks the snapshot with the latest source.retrieved_at, so the full 50-symbol SET50 universe loads.
• Suggestion now allocates across all 50 names; verified it picks rising-price, theme-positive dividend names in B1 (BGRIM, TTB), non-dividend rising in B2 (BANPU), and yield-top names in B3 (ADVANC, SCB, LH).
• Added momentum>0 gate to the profit buckets (a falling-price name must NOT be picked as "ทำกำไร"), while keeping momentum/theme_signal as Optional so the PIT backtest path (which doesn't provide them) still allocates.
• Regression tests: load_price_snapshot picks most-recently-retrieved; negative momentum excluded from profit buckets. Full suite 374 green.

Refactor: momentum is a factor INSIDE the Siamchart formula (owner rule — 2026-08-31)

The owner insisted momentum must be entered INTO the score formula (a first-class factor), not bolted on outside it. Refactored themes.py from 3 loose weight constants + inline formula repeated in 3 places to a single declarative source:

• _SIAMCHART_WEIGHTS = {eps_growth:1.5, dividend_yield:2.0, momentum:0.5} (config).
• siamchart_raw_score(g,d,m) = single source of the formula; momentum is a term INSIDE it. All three call sites (build_siamchart_score, symbol_breakdown raw, symbol_breakdown population) now use it — no more duplicated arithmetic.
• Purely a refactor: outputs unchanged (0.6/0.4 weighting intact), momentum already weighted 0.5 inside the formula.
• Added tests: raw score formula incl. momentum term; momentum raises siamchart score given identical fundamentals. Full suite 376 green.