set50-system/docs/engineering-log.md

Engineering Log — SET50 Alternative Data Platform

Current status

Milestone	Status	Evidence	Next action
M0 repo foundation	complete	Flask API, Vue/Vite shell	keep research/paper guardrails
M1 BOT Tourism adapter	complete	18 tests, live BOT fetch, raw/snapshot persistence	validate multiple vintages
M2 vintage collector	complete	25 tests, manifest idempotency, live collector and point-in-time API	collect independent releases
M2.3 event-study gate	complete/blocked	30 tests, pure engine and truthful 409 readiness API	add point-in-time price provider
M2.4 price snapshot adapter	complete/blocked	38 tests, live 9-symbol Yahoo snapshot, revised-history gate	evaluate point-in-time price source
M2.5 research runner + durable paper ledger	complete/blocked	54 tests, immutable blocked report, restart-safe paper path, live API/UI workflow	collect independent releases and point-in-time prices
M2.5 integrity hardening	complete	79 tests, canonical VintageStore replay, report and manifest-entry identity/hash binding, explicit resumable legacy migration, fail-closed manifest/snapshot reads/writes/encoding/I/O, semantic replay-shape validation, exact-schema independent review passed	protect point-in-time gate with a trusted deployment secret if threat model expands
M2.6 paper auth policy	complete/blocked	85 tests, explicit loopback-only demo mode, protected token mode preserved, UI warning and startup guard	keep demo local; use token mode for shared/network access
M2.7 PIT price archive contract	complete/blocked	93 tests, explicit pit-daily-v1 contract, per-bar known-at validation, market-timezone no-lookahead join, immutable manifest binding; live provider evidence not yet established	obtain a provider archive with release-time evidence and keep revised history false
M2.8 price-provider feasibility	complete/blocked	public evidence matrix for SET Historical Data, SETSMART, SMART Marketplace, ICE SET, LSEG Tick History, Databento, and EDI; EDI is the closest conditional price-feed lead but no candidate proves the full PIT price contract	capture forward observations while obtaining one complete provider evidence packet separately
M2.9 forward price observations + research modes	complete/blocked	immutable raw/snapshot reuse, append-only observation IDs, semantic series/raw lineage, contract-backed history counts/timing, UTC predecessor ordering, same-raw/normalized mismatch rejection, malformed-manifest rejection, first-capture temporal guard, exploratory descriptive runner, validated PIT gate preserved; 125 backend tests; final independent review deleg_e5407553 and focused follow-up review deleg_0e2282c8 passed with empty blocker arrays	collect independent BOT releases and obtain provider PIT evidence; do not promote revised history
Tourism deterministic signal	complete	live foreign-arrivals YoY surprise	add occupancy/airport metric
Internal paper ledger	complete	atomic local JSON persistence and restart test	shared store before multi-worker deployment
Dashboard	complete	Vite build + served source check with live-sign copy	visual browser capture after permission is available
General backtest accounting	complete/blocked	flat-price RED/GREEN invariants; 238 backend tests; Vite build; dividend proxy and non-PIT UI disclosures	replace final-holdings yield proxy with dated dividend cash flows; add provenance-validated PIT scorer
LLM analysis	deferred	intentionally no LLM dependency in M0	add after signal lineage is stable
Webhook receiver	deferred	contract only, no external receiver	choose after core app is usable
MT5 bridge	deferred	not started	paper bridge after webhook decision

Guardrails

• Research and paper modes only.
• No live orders, external webhook receiver, broker credentials, or MT5 connection.
• NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED].
• Deterministic signal is authoritative; LLM will remain downstream.
• Fixture and provisional BOT sources are explicitly labelled; neither is investment-ready without validation.
• target_weight is recorded in the internal paper ledger; it is not an order.
• Local paper demo mode is explicit, loopback-only, visibly warned, and never a live-order authorization path.
• Shared/network paper writes require protected token mode and a server-side session.

Verification

• Backend: 93 unittest tests pass, including paper-auth mode, malformed-config, bind-host, PIT archive, immutable-write, and market-timezone no-lookahead regressions.
• Independent M1 review: PASSED; no concrete security or logic blockers.
• Reviewer suggestions: set PAPER_COOKIE_SECURE=1 outside local HTTP; replace in-memory sessions before multi-worker deployment.
• M1 reviewer backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
• Frontend: npm run build passes with Vite.
• Backend health endpoint returns HTTP 200 JSON.
• Dashboard served HTML contains the current title, Vue mount point and Vite entry.
• Paper ledger POST and readback work through the live API.
• BOT source live fetch parsed 138 monthly periods and persisted raw HTML plus normalized snapshot.
• Data-health reports source_mode=bot, status=provisional, and replayable=true.
• Live vintage replay returned the same theme surprise as the current dashboard summary.
• Vintage collector preserved one live vintage_id with seen_count=4 and point-in-time API excluded it before published_at.
• Independent M2 review: PASSED; no concrete security or logic blockers.
• Event-study readiness gate correctly returns HTTP 409 with 1/12 independent vintages; no backtest result is fabricated.
• Independent M2.3 review: PASSED; no concrete security or logic blockers.
• Price snapshot normalized 9 symbols with adjusted close and provider-derived trading dates; quality is explicitly revised_vendor_history and point_in_time=false.
• Backtest gate requires both independent vintages and point-in-time prices.
• Independent M2.4 review: PASSED; no concrete security or logic blockers.
• Research runner persists blocked/ready reports with frozen input IDs, hashes, configuration and gate reasons.
• Revision-aware readiness counts (source_id, published_at) once and runner selects the latest revision only.
• Event-study defaults to next trading-session execution and handles non-trading event dates without skipping the first session.
• Paper ledger persists atomically under ignored backend/data/paper/ledger.json when configured.
• Live API 0.5.0 created and replayed the same blocked Tourism research report; UI served the research-run panel and human-readable gate reason.
• Independent M2.5 review: PASSED; no concrete security or logic blockers.
• Integrity hardening: normalized snapshots and raw payloads are bound to manifest metadata; malformed boolean flags, stale manifests, and cached-ready replay after tampering are rejected.
• Canonical hash algorithm is explicit: sha256-json-canonical-v1 using sorted-key compact UTF-8 JSON after excluding only the normalized hash field.
• Integrity scope is local artifact/corruption detection. A hostile machine owner who can rewrite code, manifests, raw files and runtime environment is outside this local research app's threat model.
• Independent integrity-hardening review: PASSED under the local threat model via the schema-corrected exact verdict; prior pre-fix findings are recorded as remediated.
• Replay now uses the canonical VintageStore.load_snapshot() validation path; tampered normalized snapshots return HTTP 422 instead of being recomputed.
• Research reports carry a canonical content hash; manifest entries carry their own hash and bind the report file/hash/metadata. Tampered reports, manifest hashes/metadata, and malformed report shapes fail closed.
• Research input lineage now records normalized snapshot hashes and hash algorithms alongside raw payload hashes.
• Manifest JSON shape and every list_runs()/latest() entry are validated before metadata is returned or used for selection.
• Persist validates every existing manifest entry before returning or writing a new report, and refuses malformed entries without leaving an orphan report.
• Direct report loads also bind the manifest entry identity to the requested run_id.
• Invalid UTF-8 in persisted manifest/report files is converted to ResearchRunError instead of escaping as a decode exception.
• VintageStore rejects non-object snapshots, invalid UTF-8 manifests/snapshots, and raw-payload read failures with VintageStoreError instead of leaking attribute/decode/I/O exceptions.
• Tourism replay computation rejects non-list/non-object observation/exposure shapes as controlled semantic failures; malformed exposure replay returns HTTP 422 instead of leaking AttributeError.
• Latest BOT collector/readiness check fetched the same source_id/published_at; seen_count=16 but independent releases remain 1, and the backtest gate remains HTTP 409 with insufficient_vintages.
• Browser visual capture was blocked by Chrome remote-debugging permission; no permission dialog was clicked.
• Paper auth policy: PAPER_AUTH_MODE=demo permits local paper writes without a session token only on loopback; PAPER_AUTH_MODE=token remains fail-closed without a configured server token.
• Frontend surfaces the demo warning returned by /api/v1/auth/paper; no token is embedded in the bundle.
• Independent paper-auth review initially found malformed non-string token configuration being coerced and an ambiguous disabled-session UI state; fixed with strict token typing, explicit enabled status, UI gating, and regressions.
• Fresh final paper-auth review returned schema-valid passed=true with empty security-concern and logic-error arrays; non-blocking suggestions remain for request-token type, frontend integration, and startup precedence coverage.
• The separate future PIT-price-contract delegation ended interrupted without a complete recommendation; no implementation was accepted, and revised vendor history remains point_in_time=false/blocked.
• The original staged integrity-review result (deleg_43348a3b) reported passed=true/findings=[] but violated the required four-key verdict schema; its corrected follow-up is recorded below.
• Schema-correction review deleg_a1c721d2 returned exactly {"passed":true,"security_concerns":[],"logic_errors":[],"suggestions":[]}; the staged integrity-remediation scope is now approved under the stated local threat model. Interrupted parallel review deleg_e2d54c30 contributes no evidence.
• PIT price contract: point_in_time=true now requires quality=point_in_time_archive, archive_contract=pit-daily-v1, provider release/evidence metadata, per-series IANA timezone, per-bar session_date/known_at/OHLC/adjusted-close/volume, and immutable manifest/hash binding. The event-study runner rejects missing or market-locally future-known prices when PIT mode is enabled.
• PIT verification: 93 backend tests, compileall, Vite build, npm audit (0 vulnerabilities), diff checks, and credential-pattern scan passed. The live revised Yahoo snapshot remains point_in_time=false; /api/v1/backtest/tourism?min_events=1 remains HTTP 409 blocked with price_series_not_point_in_time.
• Final bounded independent PIT review deleg_c08f6dd8 returned the exact required verdict {"passed":true,"security_concerns":[],"logic_errors":[],"suggestions":[]}. Earlier reviewer timeouts were treated as non-approving and contributed no evidence.
• Price-provider feasibility pass recorded in docs/engineering-log/2026-08-24-price-provider-feasibility.md: public SET/SETSMART/SMART Marketplace pages establish historical/API availability but not release-time PIT semantics; ICE SET is a strong commercial candidate; LSEG S3 Direct has the strongest public PIT claim but still needs SET-specific coverage and vintage evidence; Databento confirms SET venue presence and PIT corporate-action records but not PIT price-vintage semantics. No provider currently passes pit-daily-v1 from public evidence.
• No price adapter or configuration promotion was made during the feasibility pass. The next adapter must be gated on a raw sample, provider release metadata, known-at definition, correction/revision example, symbol coverage, and immutable replay evidence. Citation ledger verification for the feasibility document passed with evidence quotes for all 11 cited sources.
• M2.9 forward price observation layer records every retrieval against a stable source/period scope, preserves immutable raw/snapshot files, binds each observation ID to its prior hash and diff, tracks first/last seen times, rejects tampered observation records, and rejects out-of-order writes before creating snapshot/raw artifacts. Repeated unchanged payloads do not create a new snapshot or a new research run; runtime observation timestamps are excluded from the run fingerprint.
• Price integrity remediation requires explicit non-PIT normalized schema/series-to-raw lineage, validates PIT numeric input without leaking OverflowError, parses predecessor order in UTC, and enforces observation counts/timing for contract-backed entries. Pre-observation legacy manifests remain readable when they have no observation contract; new contract-backed entries are fully cross-checked.
• Research now has explicit validated and exploratory modes. Validated mode retains the PIT/known-at fail-closed gate. Exploratory mode may use revised vendor history only with require_price_known_at=false, returns result_scope=non_pit_descriptive_only and top-level status=descriptive_only, and carries limitations; it cannot promote point_in_time or produce a validated backtest claim.
• /api/v1/prices/observations exposes the local observation audit trail; /api/v1/prices/health reports observation count, last observation time, and revision status. The UI's research button explicitly requests exploratory mode and maps machine-readable status/reason codes to human-readable labels.
• Current M2.9 verification: 125 backend tests, compileall, Vite build, npm audit (0 vulnerabilities), diff checks, static credential/dangerous-pattern scan, and current-data Flask health smoke test passed. Provider release semantics remain unproven and the live revised Yahoo snapshot remains point_in_time=false.
• Fresh final M2.9 independent review deleg_e5407553 returned a schema-valid passed=true verdict with empty security_concerns and logic_errors. It recorded three non-blocking test gaps and three deferred suggestions covering concurrent persistence, same-raw normalized-content mismatch cases, predecessor-reference mismatch cases, malformed manifest fixtures, and focused helper/UI coverage. No commit or push has been made.
• M2.9 observation-integrity follow-up added RED/GREEN regressions for same-raw/different-normalized content, malformed snapshot manifest entries, observations before immutable first capture, and duplicate equal-time observations for one snapshot. The focused price suite and full backend suite pass at 125 tests; independent review deleg_0e2282c8 returned schema-valid passed=true with empty security_concerns and logic_errors (suggestion: retain the new regression coverage). Process-level locking and focused helper/UI coverage remain deferred.
• Audit + fix pass (2026-08-26, plan at docs/audit-and-plan-2026-08-26.md): triple-confirmed the declarative FACTORS/THEMES framework is by-passed by hand-written scoring in dashboard._theme_surprises, and that /api/v1/simulation recomputed a divergent 3-theme path. Fixed P1 (simulation reuses the canonical board via default_scores — live check: sim top pick PTT == top board combined 1.600), P2 (dashboard now emits source_summary{factor_keys:11, rows:6}; frontend shows "N ปัจจัย · M แหล่ง"), and P5-partial (removed dead list_themes/Theme/build_theme_scores/_map_index + the tests that locked them; added backend/tests/conftest.py so pytest needs no PYTHONPATH). Deferred P0 (registry-driven re-baseline) and P3/P4 (point-in-time backtest + factor-weight learning) pending explicit scope/baseline sign-off. Full backend suite: 203 tests pass; Vite build passes. This work is own-engine review gated before commit.
• P0-B + P3 + P4 (commit 8db3d48, reviews deleg_fe6f45cd + deleg_718218f8 both passed=true): the declarative FACTORS/THEMES registry is now the single source of truth (compute_theme_surprises reads registry; hand-written per-theme blocks removed; FACTORS carries center/span normalization spec; bank NPL wired into banks). P3 rebuilt run_backtest as a real multi-rebalance PIT engine (leakage_guard, momentum_at true 12-1). P4 added factor-weight learning (Spearman IC -> apply_weight_update) + /api/v1/learning/momentum. Live result: momentum IC=0.012, t=0.132 over 22 periods (no reliable predictive power in this SET50 window). 226 tests pass.
• Follow-up (A+B): theme surprises now weight-normalized by total |weight| so cross-theme magnitudes are comparable (retail 0.189->0.145). Added append-only FactorHistory store (data/factor_history/<key>.jsonl) that records every FACTORS value each scheduler run, wired into refresh_all (non-fatal), plus /api/v1/learning/factors readiness endpoint. Macro/demographic factors start at n=1 and become learnable (P4) as history accumulates. 233 tests pass.
• Backtest accounting remediation (2026-08-27): deterministic flat-price tests exposed that ending holdings value was reported as price_pnl and the dividend proxy was excluded from final value/net return. The corrected identity is final_value = capital + price_pnl + dividend_income; API now emits dividend_method=final_holdings_yield_proxy, and both result/history UI paths label the proxy and warn when leakage_guard=false. RED failures reproduced price_pnl=20,000/49,000; GREEN verification: 239 backend tests, compileall, Vite build, npm audit 0 high-severity vulnerabilities, served-bundle disclosure check, static scan and diff check passed. Fresh final independent review deleg_10918fed passed with empty blocker arrays. Evidence: docs/engineering-log/2026-08-27-backtest-accounting-remediation.md and docs/test-evidence/2026-08-27-backtest-accounting-remediation.md. The dividend model remains a proxy and the default public backtest remains descriptive non-PIT.
• PIT factor store + partial score provider (2026-08-27, commit 1f630be): added factor_vintages.py (append-only PIT factor store: observed/released/retrieved_at + SHA-256 hash chain; value_at(as_of) anti-look-ahead, fail-closed when nothing released by as_of) and pit_scorer.py (PitScoreProvider.score_board(as_of) + make_pit_score_fn). backtest._resolve_scores now sets leakage_guard=True only when the supplied score_fn asserts pit_meta.pit=True (closes "any callable ⇒ PIT"); /api/v1/backtest accepts use_pit. Empty factor store fail-closes (live probe leakage_guard=false). Full backend 255 passed (was 239). Honest scope: theme dimension PIT from this store forward; no factor history before 2026-08-27 so pre-today backtests stay non-PIT; siamchart fundamental partial (EPS 5-yr series PIT-grade, ratios current → partial_pit). Next: dated dividend cash-flow ledger.
• Dated dividend cash-flow ledger (2026-08-27, commits 068dff2 + 5b8b6d1): added dividend_ledger.py (per-symbol dated store: ex_date/record_date/pay_date/per_share/source/estimate flag; credit_dividends credits per_share×qty when payment due, fail-closed with no entry; build_dps_ledger builds per-share estimate rows from siamchart ratios.DPS). run_backtest accepts dividend_ledger and reports dividend_method = dated_ledger (real rows) | dps_annual_proxy (estimate) | final_holdings_yield_proxy (legacy). /api/v1/backtest accepts use_ledger. Frontend maps method to an honest Thai label + Proxy badge on result card and history rows. Full backend 266 passed (was 255). Honest scope: DPS rows are estimates (no ex-date history in snapshot yet); real dated cash flows require collecting per-stock dividend history, which upgrades a symbol to dated_ledger.
• PIT siamchart vintage store (2026-08-27, commit 887e9c9): added siamchart_vintages.py (append-only hash-chained store of every collected siamchart snapshot; snapshot_at(as_of) returns the newest snapshot with retrieved_at <= as_of — anti-look-ahead, fail-closed). PitScoreProvider accepts siamchart_store; when wired the fundamental dimension reads the snapshot knowable at as_of (pit_grade='pit') and partial_pit no longer forces true. /api/v1/backtest use_pit seeds the first vintage from the current snapshot. Full backend 273 passed (was 266). Honest scope: snapshot reconstruction forward only; EPS year-keys not tied to calendar years (growth = latest-vs-prior).
• Real forward-test lifecycle (2026-08-27, commits 6d9c283 + 80c6d79): the cosmetic "forward" (same single-pass backtest, different mode string) is replaced with a durable, frozen-signal paper-portfolio lifecycle. forward_test.py ForwardTestStore (thread-safe JSON store) with status flow: frozen (signals snapshotted immutable) → executed (fills 50/20/30 at post-freeze prices) → marked (mark-to-market equity series) → matured (net_return). New routes: GET /api/v1/forward(+<id>), POST /api/v1/forward (create+execute, use_pit freeze), POST /<id>/mark, POST /<id>/mature; store at data/forward/runs.json (survives restarts). UI simulation tab: forward calls /api/v1/forward, loads runs, shows status/non-PIT/holdings + Mark/Mature per run. Full backend 280 passed (was 273). Honest scope: score source at CREATE may be current board (non_pit=true tagged); paper-only.
• Factor-learning validation gate (2026-08-27, commit ae814c3): closed the P4 "no auto-apply" loop — weight_learning.py now splits a chronological IC series into train + holdout via apply_validation_gate, and a factor is validated=True only when total sample >= 12, each window >= its min, train & holdout IC both beat baseline (BASELINE_IC=0) and agree in sign, and pooled |t| > 1.0. apply_weight_update keeps the weight unchanged for any unvalidated factor (no auto-apply); only validated factors move. learn_momentum_gated wired into /api/v1/learning/momentum, surfacing ic_train/ic_holdout/validated/gate_notes. Full backend 286 passed (was 280). Live probe: momentum validated=false, gate_note "IC not above baseline (0.0711/-0.1143)" — weight unchanged.
• Real dated dividend-history collector (2026-08-27, commit f9973e8): siamchart.py gained parse_dividend_history/fetch_dividend_history (reads the "ประวัติการปันผล" ex-date + DPS table per stock-info page); dividend_ledger.py gained populate_dated_dividends (registers dated rows, estimate=False). POST /api/v1/dividends/update fetches all snapshot symbols and persists a dated ledger at data/dividends/ledger.json; use_ledger backtests prefer the dated ledger (dividend_method=dated_ledger) and fall back to DPS estimates when unpopulated. Full backend 292 passed (was 286). Live network fetch: 49/49 symbols, 1410 dated payments; use_ledger → dated_ledger. Note: this sandbox HAS outbound network (curl/https to siamchart 200) — contrary to earlier assumption.
• Auto-refresh dated dividend ledger (2026-08-27, commit 03195dc): AppDataScheduler now calls _maybe_refresh_dated_dividends() in refresh_all — a cooldown-gated (default 6h) fetch of real dated dividend history into data/dividends/ledger.json, so the ledger stays fresh without manual /api/v1/dividends/update and without hammering the source every tick. DIVIDEND_REFRESH_COOLDOWN_SECONDS env toggle. Full backend 294 passed (was 292).