5b8a10c730
Owner: momentum must enter the formula (a factor), not be bolted on outside it.
- Single declarative source _SIAMCHART_WEIGHTS = {eps_growth:1.5, dividend_yield:2.0, momentum:0.5}.
- New siamchart_raw_score(g,d,m) = single source of the formula; momentum is an
explicit term inside it. All 3 call sites (build_siamchart_score + both
symbol_breakdown spots) now share it — no duplicated arithmetic.
- Pure refactor: outputs unchanged (weights identical). Tests added for the
momentum-inside-formula rule + momentum raising the score. Full suite 376 green.
159 lines
12 KiB
Markdown
159 lines
12 KiB
Markdown
# 2026-08-29 — Data-source expansion (Phase A) + frontend padding fix + sign bug
|
||
|
||
## Scope
|
||
Two-part request from the owner:
|
||
1. **More + genuinely-used data sources per theme** — the owner felt the source
|
||
count was too few and insisted any newly-fetched data must feed the analysis
|
||
(theme surprise), not just the provenance table.
|
||
2. **Frontend padding/margin issues** — several panels had misaligned spacing.
|
||
|
||
## What changed (all verified)
|
||
| Area | Change | Evidence |
|
||
|---|---|---|
|
||
| `factors.py` | Wired 2 **fetched-but-unused** BOT fields into the registry: `macro_core_inflation` (core inflation, sign −1) and `macro_unemployment` (sign −1) → they now actually move themes | Before, `macro_thai.to_dict()` emitted 7 fields but only 5 were registered factors — 2 were dead display data |
|
||
| `themes.py` | Deepened banks/retail/consumer_staples/healthcare/nonbank_finance with the new factors (positive weights) | direction tests |
|
||
| **Sign bug fix** | Found + fixed a **double-negative**: several factors carry `sign: -1` (NPL, inflation, unemployment) while theme weights were ALSO negative → higher NPL/inflation RAISED the theme score (inverted). Flipped all 16 theme weights to positive; direction now lives only in `sign`. Proven: auto-NPL 3.95→7.0 previously raised auto_credit 0.64→0.799; now lowers it. | `test_bearish_factors_move_score_the_right_way` |
|
||
| **NEW `te_thailand.py`** | New TradingEconomics collector parsing 2 pages (interest-rate + consumer-confidence) → 6 factors: interest rate, loans-to-fin-corp, consumer credit, household-debt/GDP, retail-sales-YoY, consumer confidence | live fetch: rate 1.0%, retail -14.5% YoY, confidence 51.8 |
|
||
| `themes.py` | Wired new TE factors into banks (rate/loan), retail + consumer_staples (retail sales + confidence), nonbank_finance (credit + debt + confidence) | direction tests |
|
||
| `scheduler.py` / `dashboard.py` | Registered `te_thailand` in `_REFRESH_JOBS` + dashboard cached-fetch + `_build_sources` provenance | dashboard live build: 8 sources |
|
||
| `frontend/App.vue` | Theme cards now show the new reads (banks→interest rate, retail→retail sales YoY, nonbank→household debt %GDP) | build clean |
|
||
| `frontend/style.css` | **padding/margin + undefined-variable fix** — added missing design tokens (`--card/--foreground/--accent/--font/...`) that the theme/modal components reference but `:root` never declared (they rendered transparent/wrong-color); zeroed section-panel padding so `.signal-header` is the single top-spacing source (was double 22px+22px on theme/lineage/health/sim/backtest panels) | build clean |
|
||
|
||
## Key decisions
|
||
- **One new module** (`te_thailand.py`) covers 4 plan items (A1 rate/credit, A2
|
||
consumer credit, A3 retail sales, A4 confidence) because all four series live
|
||
on just two TradingEconomics pages — avoids 4 fragile scrapers.
|
||
- **Every source must feed a surprise**: tightened `test_every_factor_value_key_resolves_to_a_fetched_field` so a factor whose `value_key` the collector never emits fails the suite — no dead factors.
|
||
- Sign convention locked: `sign` = factor direction, theme weights = positive magnitude. Regression test locks NPL/inflation/unemployment move correctly.
|
||
|
||
## Verified (evidence)
|
||
- Full backend suite: **360 tests OK** (up from 352 — 8 new `test_te_thailand`).
|
||
- Live dashboard build: 13 themes, 8 sources, new TE row; retail surprise −0.165
|
||
(correctly negative from retail-sales −14.5% YoY) vs old BOT-only (positive).
|
||
- Frontend `npm run build` clean.
|
||
- Independent code review: see delegation verdict (fail-closed).
|
||
|
||
## Phase B (same day, follow-up "ทำต่อได้เลย")
|
||
Extended the same `te_thailand` module with 2 more TradingEconomics pages:
|
||
- `te_property_prices` (residential property prices +1.26% YoY) → **property** theme
|
||
- `te_business_confidence` (46.7) → **telecom_it + property + healthcare**
|
||
+2 FACTORS, +2 theme wrings, +4 tests (parse + direction for both). Full suite now
|
||
**362 tests OK**; frontend build clean; live fetch confirmed (1.26 / 46.7).
|
||
Cleared a stale daily-cache `te_thailand` entry so the new fields show immediately.
|
||
|
||
## Phase C (same day, follow-up "ทำ phase B ต่อได้เลย" — energy breadth)
|
||
Feasibility spike of the remaining Phase B sources found most are JS-rendered or
|
||
anti-bot (recorded in plan), but **IRPC** performance-highlights is server-rendered
|
||
and clean. Added:
|
||
- `backend/app/energy_irpc.py` — collector parsing IRPC net-profit/EBITDA/ROE
|
||
margin rows [2024,2025,3M26], latest period (3M26: net margin **+10.27%**)
|
||
- factor `energy_irpc_net_margin` (sign +1) wired into refining_energy /
|
||
exploration / utilities (2nd real Thai refiner beyond TOP)
|
||
- scheduler job + dashboard fetch + `_build_sources` row (now **9 sources**)
|
||
- tests: parse (incl. paren-negatives), value-key resolution, direction
|
||
- frontend theme card shows "กำไรสุทธิ IRPC 10.27%"
|
||
Full suite **368 tests OK**; frontend build clean; live dashboard refining_energy
|
||
surprise 0.764 driven partly by IRPC margin.
|
||
Independent review `deleg_474555e4` **passed: true** (parser/paren/value-key/wiring
|
||
all correct, no security/logic errors). Suggestion applied: comment noting
|
||
ebitda/roe are display-only. Author also flagged that a single upstream page
|
||
failure (strict `_fetch_with_cache` → DashboardError) takes down the whole board —
|
||
consistent with the documented no-fixture-fallback policy; left as an open
|
||
decision, not silently changed.
|
||
|
||
Deferred (feasibility blocked): REIC (JS SPA/XHR), EPPO (JS/WordPress), NBTC (403
|
||
anti-bot), PTTEP (JS shell), PTT/BCP (404/DNS). Need browser/XHR approach, not
|
||
plain-HTML — recorded in the plan as a separate effort.
|
||
|
||
## Phase D (same day — 3 follow-up asks)
|
||
1. **Flexible scoring (Q2)**: `_fetch_with_cache` now degrades instead of raising
|
||
`DashboardError` — source down + no cache = return {} (theme drops that
|
||
source's factors); previously-good value present = daily cache returns stale.
|
||
Board no longer crashes on any single upstream failure (verified: all-sources-
|
||
down still builds 13 themes). `DashboardError` class now unused by build().
|
||
2. **Per-source calc detail (Q3)**: new `themes.factor_source_breakdown(fetched,
|
||
theme)` surfaced in dashboard as `fetch_data` + `factor_sources`, and in the
|
||
per-symbol modal as `symbolDetail.factor_sources` — the owner sees, per theme,
|
||
each factor's source → raw → normalized → weight → contribution (e.g. retail:
|
||
te_thailand ยอดขายปลีก -14.5 → -1.0 × 0.7 = -0.7). Backend test added.
|
||
3. **HAR feasibility for deferred sources (Q1)**: captured REIC via
|
||
`har-derived-api-client` (Playwright drive → HAR → derived XHR endpoint
|
||
`POST /Home/Web_All_Num_View`). Spike shows the method WORKS (browser drives
|
||
the JS SPA, XHR endpoint derivable) BUT the homepage XHR returned an empty
|
||
body — actual property data needs a deeper interaction (a real Transfer page
|
||
click), so a full REIC collector is a larger follow-up, not a quick add.
|
||
Finishing: commit Q2+Q3 with suite 369 green; Q1 recorded as feasible-but-
|
||
needs-deeper-capture and left as a decision for the owner.
|
||
|
||
- Note: a **concurrent process** also landed `thai_trade.py` (external-sector
|
||
exports/imports/current-account) and external_* factors mid-session; its 3
|
||
initially-broken tests were fixed to reach the 360-green baseline here.
|
||
|
||
## Flexible PIT backtest (owner rule — 2026-08-30)
|
||
Owner clarified the backtest contract: it must run as soon as there is enough
|
||
data to estimate an investment, NOT only when EVERY factor has PIT history.
|
||
Some sources may not provide deep history — scoring is deliberately flexible
|
||
(a theme scores from whatever subset of factors was knowable that day).
|
||
|
||
Implemented (commit 2026-08-30):
|
||
- `backtest_readiness.evaluate_readiness`: readiness is now a *usable* window
|
||
(price + Siamchart + >=1 released factor), not all-factors-present. Missing
|
||
factors are still reported in `missing` for transparency but no longer block.
|
||
`recommended_start` = oldest executable price (the oldest history held);
|
||
`recommended_end` = last complete trading day (yesterday Bangkok).
|
||
- `pit_scorer.theme_surprise_report`: flexible — skips any factor with no release
|
||
by `as_of` (no blanket theme block); `blocked` only when NO factor has a value.
|
||
`pit_meta.partial_pit` reflects themes that scored from a partial subset.
|
||
- Verified: `/api/v1/backtest/readiness` → ready=true, recommended 2024-01-03 →
|
||
2026-08-29. `POST /api/v1/backtest/run` (default window) → 201 full result
|
||
(final_equity 1,117,243.95 on 1M), no 400 from missing factors.
|
||
- Tests: test_backtest_readiness updated to earliest-runnable semantics; full
|
||
suite 369 green.
|
||
|
||
## Capital-allocation selection rework (owner rule — 2026-08-31)
|
||
Owner clarified what the 3 buckets mean and flagged that the old selection
|
||
(rank by `combined` = 60/40 theme+siamchart where siamchart was EPS-growth
|
||
dominated) picked names like BTS whose EPS was up 137% but whose PRICE was flat/
|
||
falling — NOT the owner's "ทำกำไร" (price likely to rise in 3-6 months).
|
||
|
||
New definition implemented:
|
||
- "ทำกำไร" is now measured by a **price-trend (momentum) score**, not EPS growth.
|
||
Added `themes.price_trend_score()` blending ~3/6/12-month momentum (trading
|
||
days), z-scored across the universe (heavy weight on 3/6m per the 3-6 month
|
||
tenure).
|
||
- Buckets 1 & 2 rank by momentum, **gated on `theme_signal > 0`** (mean surprise
|
||
of the symbol's themes). theme_signal=None (backtest path) is not gated, so
|
||
the PIT backtest still allocates.
|
||
- Bucket 3 unchanged: highest dividend yield, ignores score/momentum.
|
||
- Suggestion endpoint passes real momentum + theme_signal from the live board.
|
||
- Verified: suggestion now picks CRC+BEM (dividend, rising price) in bucket 1;
|
||
PTT/MINT (falling price, theme positive) slide to bucket 3 by yield, not
|
||
bucket 1. Full suite 372 green (3 new momentum/gate tests).
|
||
|
||
## Price-snapshot load fix + momentum gate (owner "เพื่อทดสอบ logic" — 2026-08-31)
|
||
- `load_price_snapshot` was picking the LAST snapshot by FILENAME (lexicographic),
|
||
which selected a stale 9-symbol collection over the full 50-symbol universe
|
||
(`2024-01-01..` sorts after `2023-08-28..`). Now it picks the snapshot with the
|
||
latest `source.retrieved_at`, so the full 50-symbol SET50 universe loads.
|
||
- Suggestion now allocates across all 50 names; verified it picks rising-price,
|
||
theme-positive dividend names in B1 (BGRIM, TTB), non-dividend rising in B2
|
||
(BANPU), and yield-top names in B3 (ADVANC, SCB, LH).
|
||
- Added momentum>0 gate to the profit buckets (a falling-price name must NOT be
|
||
picked as "ทำกำไร"), while keeping momentum/theme_signal as Optional so the
|
||
PIT backtest path (which doesn't provide them) still allocates.
|
||
- Regression tests: load_price_snapshot picks most-recently-retrieved; negative
|
||
momentum excluded from profit buckets. Full suite 374 green.
|
||
|
||
## Refactor: momentum is a factor INSIDE the Siamchart formula (owner rule — 2026-08-31)
|
||
The owner insisted momentum must be entered INTO the score formula (a first-class
|
||
factor), not bolted on outside it. Refactored `themes.py` from 3 loose weight
|
||
constants + inline formula repeated in 3 places to a single declarative source:
|
||
- `_SIAMCHART_WEIGHTS = {eps_growth:1.5, dividend_yield:2.0, momentum:0.5}` (config).
|
||
- `siamchart_raw_score(g,d,m)` = single source of the formula; momentum is a term
|
||
INSIDE it. All three call sites (build_siamchart_score, symbol_breakdown raw,
|
||
symbol_breakdown population) now use it — no more duplicated arithmetic.
|
||
- Purely a refactor: outputs unchanged (0.6/0.4 weighting intact), momentum
|
||
already weighted 0.5 inside the formula.
|
||
- Added tests: raw score formula incl. momentum term; momentum raises siamchart
|
||
score given identical fundamentals. Full suite 376 green.
|