Committing the prior uncommitted working-tree state that predates this session's
data-source work (was already modified/untracked at session start) so the tree
is clean before push. Includes: event-study + research report integrity/forward
observation work, prices tests, research hash migration script, and the
2026-08-23/24 engineering-log + test-evidence notes. Verified green as part of
the full 362-test suite.
- define missing design tokens (--card/--foreground/--accent/--font/--text-2)
that theme/modal components referenced but :root never declared (they rendered
transparent/wrong color)
- zero section-panel padding so .signal-header is the single top-spacing source
(was double 22px+22px on theme/lineage/health/sim/backtest panels)
- theme cards now surface the new source reads (banks rate, retail sales YoY,
nonbank household debt, property prices, telecom business confidence)
Add self-contained Docker deployment for EasyPanel / docker-compose:
- Dockerfile: python:3.11-slim + nginx. Serves the PREBUILT Vue SPA
(frontend/dist, committed) via nginx and reverse-proxies /api to the Flask
backend on 127.0.0.1:5000. No node/npm in the image (avoids Vite/npm
flakiness in Docker builds). VOLUME /app/backend/data for persistence
(factor vintages, dividend ledger, forward runs, prices). HEALTHCHECK hits
/api/v1/dashboard/summary. Exposes :80.
- deploy/nginx.conf: SPA fallback + /api proxy to 127.0.0.1:5000 + gzip.
- deploy/entrypoint.sh: starts Flask (HOST=0.0.0.0:5000) + nginx foreground,
forwards signals.
- docker-compose.yml: single service 'set50', port 8080:80, mounts
./backend/data for persistence, healthcheck.
- .dockerignore excludes .git/.venv/backend/data/node_modules/docs.
- .gitignore now tracks frontend/dist/ (prebuilt bundle required by the
image); backend/data stays untracked.
- Backend runtime verified: test_client GET /api/v1/dashboard/summary=200
(the HEALTHCHECK path). entrypoint sh -n, compose YAML parse, and all
Dockerfile-referenced files exist. NOTE: no local docker here, so the
image itself was not built — that happens on EasyPanel.
Backtest result card and saved-run history now show an explicit dividend
badge per run: green 'ตามวันจริง' for dated_ledger (real ex-date x qty cash
flow), amber 'Proxy (ต่อหุ้น)'/'Proxy' for estimates, plus a footnote that
matches the actual dividend_method. Frontend build passes (bundle
index-DDItjwYy.js); served bundle contains the new strings.
Automatically keep the real dated dividend ledger fresh inside the app's own
refresh loop (this app runs on its own server, independent of Hermes):
- backend/app/scheduler.py: AppDataScheduler gained a cooldown-gated
_maybe_refresh_dated_dividends() that fetches real dated dividend history
(siamchart /stock-info) into data/dividends/ledger.json at most once per
dividend_cooldown_seconds (default 6h) — dividend history changes only a
few times a year, so we never hammer the source every refresh tick. The
fetch is non-fatal: a network failure leaves the previous ledger intact.
- backend/app/__init__.py: passes DIVIDEND_REFRESH_COOLDOWN_SECONDS to the
scheduler (default 21600s).
- tests: cooldown fires once then skips, and refetches after it elapses (2)
— full backend 294 passed.
Close the last deferred PIT milestone by collecting REAL per-stock dated
dividend cash-flow history from Siamchart, upgrading the dividend ledger
from DPS estimates to dated_ledger.
- backend/app/siamchart.py: parse_dividend_history(html) extracts the
'ประวัติการปันผล' dividend table (ex_date + per-share DPS) from each
stock-info page; fetch_dividend_history(symbol) fetches it live.
- backend/app/dividend_ledger.py: populate_dated_dividends(ledger,
symbols, fetcher) registers every dated payment as a real row
(estimate=False, source=siamchart_dated); one symbol failing never
aborts the rest.
- backend/app/__init__.py: DividendLedger persisted at
data/dividends/ledger.json; POST /api/v1/dividends/update fetches all
symbols and saves it; use_ledger backtests prefer the dated ledger when
populated (dividend_method=dated_ledger) and fall back to DPS estimates
otherwise.
- tests: parser (4) + populate (2) — full backend 292 passed.
Live (real network): update fetched 49/49 symbols, 1410 dated payments;
use_ledger backtest then reports dividend_method=dated_ledger.
The 'eval(' static-scan hit is ast.literal_eval (safe literal parse, no
code execution), not eval().
Add a strict holdout/walk-forward + baseline gate to factor-weight learning,
per the P4 guardrail: learned weights are never auto-applied until minimum
sample, holdout/walk-forward, and baseline comparison all pass.
- backend/app/weight_learning.py:
- FactorLearning gained ic_train / ic_holdout / validated / gate_notes.
- apply_validation_gate(...) splits a chronological IC series into train +
holdout and only marks validated=True when: total >= MIN_SAMPLE_PERIODS,
each window >= its minimum, train AND holdout IC are positive (beat the
BASELINE_IC=0) and agree in sign, and the pooled |t| > MIN_IC_TSTAT.
- apply_weight_update now keeps new_weight == old_weight for any factor
that is not validated (no auto-apply); only validated factors move.
- learn_momentum_gated(...) builds PIT momentum ICs then applies the gate.
- backend/app/__init__.py: /api/v1/learning/momentum uses the gated learner
and surfaces ic_train/ic_holdout/validated/gate_notes.
- tests: gate (16) via rewritten suite — full backend 286 passed.
Live probe on current price archive: validated=false with
gate_note 'IC not above baseline (0.0711/-0.1143)' — momentum is not
validated, weight stays unchanged (new_weight=None).
Forward mode in the simulation tab now calls the durable /api/v1/forward
lifecycle instead of the single-pass /api/v1/simulation (which only differed
by a mode string). runSimulation branches to /api/v1/forward for forward mode,
loads runs on mount, and renders a forward panel listing each paper run with
its status (frozen/executed/marked/matured), non-PIT badge, holdings and a
Mark / Mature action per run. Frontend build passes (bundle index-INuvPOCF.js).
Replace the cosmetic 'forward' mode (which was the same single-pass backtest
with a mode string) with a genuine forward paper-portfolio lifecycle:
- backend/app/forward_test.py: ForwardTestStore — durable, thread-safe JSON
store of forward runs with an explicit status lifecycle:
frozen (signals snapshotted, immutable) -> executed (fills 50/20/30
buckets at post-freeze prices) -> marked (mark-to-market equity series ->
matured (net_return finalised).
Frozen signals can never be re-read/rewritten after creation, so later data
cannot retroactively change what the run decided.
- backend/app/__init__.py: GET /api/v1/forward (+<id>), POST /api/v1/forward
(create+execute, with use_pit to freeze PIT or current-board scores),
POST /<id>/mark, POST /<id>/mature. ForwardTestStore wired as an extension
backed by data/forward/runs.json (survives restarts).
- tests: lifecycle store (7) — full backend suite 280 passed. Live probe:
create->execute (2xx, real holdings), list, mark, mature all work and the
run persists.
Honest scope: the score source at CREATE time may be the current board
(non_pit=true, tagged); paper-only, no MT5 send. A PIT scorer only marks a run
non_pit=false when its scores assert pit_meta.pit=true.
Add an append-only, hash-chained store of every collected Siamchart
fundamental snapshot so the 40% fundamental dimension can be reconstructed
at a historical date instead of always reading the latest snapshot:
- backend/app/siamchart_vintages.py: SiamchartVintageStore persists each
snapshot under its retrieved_at with a SHA-256 canonical hash chain
(tamper/reorder detectable); snapshot_at(as_of) returns the newest
snapshot whose retrieved_at <= as_of (anti-look-ahead), and fails closed
(returns {}) when none is knowable yet. Deduplicates identical
retrieved_at+body persists.
- backend/app/pit_scorer.py: PitScoreProvider accepts siamchart_store; when
wired, siamchart_factor_view reads the snapshot knowable at as_of
(pit_grade='pit') instead of the current snapshot (pit_grade='current').
score_board no longer forces partial_pit when a store is present — the
fundamental dimension is PIT; the theme dimension still fails closed
(pit=false) unless every theme factor has a released PIT value by as_of.
- backend/app/__init__.py: /api/v1/backtest use_pit seeds the first vintage
from the current snapshot (idempotent) and wires the store.
- tests: store (6) + scorer-with-store anti-look-ahead (1) — full backend
suite 273 passed.
Honest scope: snapshots are stored whole and reconstructible forward;
EPS year-keys inside a snapshot are not tied to calendar years, so EPS
growth stays latest-vs-prior (not fiscal-year-pinned). No history before the
first collected snapshot exists.
Map dividend_method (dated_ledger / dps_annual_proxy / final_holdings_yield
_proxy) to an honest Thai label. Show a Proxy badge whenever the dividend is
an estimate (everything but dated_ledger) on both the result card and the
saved-run history rows, and set the footnote to match the actual method.
Frontend build passes.
Replace the single final-holdings yield proxy with a per-symbol dated
dividend ledger for the backtest engine:
- backend/app/dividend_ledger.py: DividendLedger store (ex_date,
record_date, pay_date, per_share, source, estimate flag) with validation
and persistence; credit_dividends credits per_share * qty once a payment is
due (on/after ex-date and pay date); build_dps_ledger builds estimate rows
from siamchart ratios.DPS (per-share, price-independent) as a step up from
the yield-percentage proxy.
- backend/app/backtest.py: run_backtest accepts dividend_ledger; when set,
dividend_income comes from the ledger and dividend_method reports
'dated_ledger' (real rows) or 'dps_annual_proxy' (estimate). No ledger ->
legacy final_holdings_yield_proxy preserved and labelled.
- backend/app/__init__.py: /api/v1/backtest accepts use_ledger, wiring the
DPS-built ledger.
- tests: ledger store/credit (9) + backtest ledger integration (2 new) —
full backend suite 266 passed. Live probe: use_ledger flips dividend_method
to dps_annual_proxy with per-share income (4151.0) vs proxy (5041.96).
Honest scope: DPS rows are estimates (no ex-date history in snapshot yet);
real dated cash flows require collecting per-stock dividend history, which
upgrades a symbol to dated_ledger when present.
Append the PIT-work session to HANDOFF and engineering-log: factor
vintages store, partial PIT score provider, honest leakage gating, /
api/v1/backtest use_pit wiring, 255-test verification, and the honest
scope (no pre-2026-08-27 factor history; siamchart fundamental partial).
Add a point-in-time (PIT) factor/data store and a score provider so the
backtest engine can rebuild per-symbol scores from data actually knowable
at a given date, instead of silently reusing the live board:
- backend/app/factor_vintages.py: append-only, provenance-complete store
(observed_at/released_at/retrieved_at) with a SHA-256 canonical hash chain.
value_at(as_of) only ever returns rows whose released_at <= as_of (real,
testable anti-look-ahead); no value by as_of fails closed (returns None).
- backend/app/pit_scorer.py: PitScoreProvider computes theme surprises from
PIT factor values only, and a partial siamchart fundamental view (EPS
growth from the 5-year series; current ratios marked partial). score_board
attaches pit_meta so callers can tell PIT from fallback.
- backend/app/backtest.py: _resolve_scores now sets leakage_guard ONLY when
the supplied score_fn's meta asserts pit_meta.pit=true; an arbitrary
callable with no PIT proof is no longer treated as PIT (closes the
'supplied fn => PIT' hole).
- backend/app/__init__.py: /api/v1/backtest accepts use_pit, wiring the PIT
provider; _load_siamchart_snapshot loads the SET50 fundamental snapshot.
- tests: factor store (9), pit scorer (5), backtest leakage-guard gating (2
new + 1 corrected) — full backend suite 255 passed. Empty store fail-closes
(leakage_guard=false) as proven by a live route probe.
Honest scope: theme dimension is PIT from this store forward; siamchart
fundamental remains partial (current ratios) and is flagged as such. No
historical factor data before today exists, so pre-today backtests remain
non-PIT by construction.
Closes reviewer suggestion (deleg_5dd358e3): adds coverage for the factor
readiness endpoint (n_points / learnable / last_value / ordering) and the
min_points 400 validation. FACTOR_HISTORY_DIR is now configurable via app
config so tests (and deploy) can point the history store at a chosen path
instead of a hardcoded data dir. 236 tests pass.
A. Cross-theme comparability:
- compute_theme_surprises now weight-normalizes by total |weight| (weighted
average), so every theme surprise on same [-1,1] scale regardless of factor
count/weight (retail 0.189->0.145; auto_credit 1.0->0.64).
B. Historical factor store (enables learning macro/demographic factors):
- New factor_history.py: append-only per-factor JSONL, dedupes unchanged
values, rejects non-finite, records every FACTORS value each scheduler run.
- scheduler.py: jobs carry fetch_module; refresh_all records factor history
(non-fatal); added bank_npl job.
- GET /api/v1/learning/factors?min_points= reports n_points/learnable per
factor so users see when P4 learning unlocks (validated query parsing).
- weight_learning: generic learn_factor_series() aggregator (momentum reuses).
Independent review deleg_5dd358e3 passed=true (empty security/logic arrays);
its two robustness suggestions applied (finite guard in record(), clean 400 on
bad min_points). 234 tests pass; Vite build passes.
P0-B (registry is the single source of truth for scoring):
- FACTORS now carries center/span normalization spec; unused hand-written
per-theme surprise blocks in dashboard.py replaced by one registry-driven
compute_theme_surprises() (themes.py).
- THEMES['banks'] adds bank_npl weight so NPL is genuinely blended.
- factor_value/normalize hardened against NaN/inf (finite guards).
- Board re-ranks (TRUE/GULF up, TOP->3) per registry weights; 3 new tests
incl. 'changing a registry weight changes output'.
P3 (point-in-time backtest):
- run_backtest is now a real multi-rebalance engine (reallocates every window,
reconciles holdings, marks to market) instead of allocate-once+break.
- Added leakage_guard (False unless a PIT score_fn is supplied), planned vs
actual rebalances, and momentum_at() true 12-1 (skips last month, PIT).
P4 (factor-weight learning):
- weight_learning.py: cross-sectional Spearman IC, forward-return builder,
IC aggregation + t-stat, and apply_weight_update (new = clip(old*(1+shrink*IC))).
- GET /api/v1/learning/momentum endpoint. Live result: momentum IC=0.012
t=0.132 over 22 periods -> momentum has no reliable predictive power here.
Macro/demographic factors blocked (no historical factor vintages yet).
Two independent review gates passed (deleg_fe6f45cd, deleg_718218f8): empty
security/logic arrays; their non-blocking suggestions applied (finite guards,
dedupe leakage_guard resolution). 226 tests pass; Vite build passes.
- P1: /api/v1/simulation now uses the canonical board score (default_scores)
instead of a divergent 3-theme recompute -> 'จำลอง' can't disagree with board
(live check: sim top pick PTT == top board combined 1.600). Removes binary
auto/en signs, restores quality+momentum+dividend screen consistency.
- P2: dashboard emits source_summary{factor_keys, rows}; frontend shows
'N ปัจจัย · M แหล่ง' so the 7-vs-5 count confusion is impossible.
- P5: removed dead themes.list_themes()/Theme/build_theme_scores/_map_index and
the tests that locked them; added tests/conftest.py so pytest needs no PYTHONPATH.
- docs: audit-and-plan-2026-08-26.md (full P0-P5 plan) + engineering-log entry.
- 203 backend tests pass; Vite build passes. Independent reviewer: no security or
logic blockers (minor error-leak suggestion applied: 503 message no longer leaks
exception detail).
- backtest.py: buy-and-hold backtest over [start,end] — allocates 50/20/30 at first available rebalance date, marks to market to end, accrues dividend, reports {final_value, price_pnl, dividend_income, net_return, trades, holdings}
- Fixed double-spend bug (was allocating full capital every rebalance -> negative cash)
- dashboard.default_scores(): per-symbol combined/dividend/yield baseline for backtest
- POST /api/v1/backtest + GET /api/v1/backtest/runs (results persisted in app state -> survive refresh)
- Frontend: backtest section w/ start/end/capital/freq inputs + P&L KPIs + run history table
- Honest note: uses current combined scores as static baseline (non-PIT); PIT score_fn pluggable
- Verified: 1M -> 1.088M (+8.80%) over 2024-06..2026-06; history persists across refresh
