105 lines
22 KiB
Markdown
105 lines
22 KiB
Markdown
# Engineering Log — SET50 Alternative Data Platform
|
||
|
||
## Current status
|
||
|
||
| Milestone | Status | Evidence | Next action |
|
||
|---|---|---|---|
|
||
| M0 repo foundation | complete | Flask API, Vue/Vite shell | keep research/paper guardrails |
|
||
| M1 BOT Tourism adapter | complete | 18 tests, live BOT fetch, raw/snapshot persistence | validate multiple vintages |
|
||
| M2 vintage collector | complete | 25 tests, manifest idempotency, live collector and point-in-time API | collect independent releases |
|
||
| M2.3 event-study gate | complete/blocked | 30 tests, pure engine and truthful 409 readiness API | add point-in-time price provider |
|
||
| M2.4 price snapshot adapter | complete/blocked | 38 tests, live 9-symbol Yahoo snapshot, revised-history gate | evaluate point-in-time price source |
|
||
| M2.5 research runner + durable paper ledger | complete/blocked | 54 tests, immutable blocked report, restart-safe paper path, live API/UI workflow | collect independent releases and point-in-time prices |
|
||
| M2.5 integrity hardening | complete | 79 tests, canonical VintageStore replay, report and manifest-entry identity/hash binding, explicit resumable legacy migration, fail-closed manifest/snapshot reads/writes/encoding/I/O, semantic replay-shape validation, exact-schema independent review passed | protect point-in-time gate with a trusted deployment secret if threat model expands |
|
||
| M2.6 paper auth policy | complete/blocked | 85 tests, explicit loopback-only demo mode, protected token mode preserved, UI warning and startup guard | keep demo local; use token mode for shared/network access |
|
||
| M2.7 PIT price archive contract | complete/blocked | 93 tests, explicit `pit-daily-v1` contract, per-bar known-at validation, market-timezone no-lookahead join, immutable manifest binding; live provider evidence not yet established | obtain a provider archive with release-time evidence and keep revised history false |
|
||
| M2.8 price-provider feasibility | complete/blocked | public evidence matrix for SET Historical Data, SETSMART, SMART Marketplace, ICE SET, LSEG Tick History, Databento, and EDI; EDI is the closest conditional price-feed lead but no candidate proves the full PIT price contract | capture forward observations while obtaining one complete provider evidence packet separately |
|
||
| M2.9 forward price observations + research modes | complete/blocked | immutable raw/snapshot reuse, append-only observation IDs, semantic series/raw lineage, contract-backed history counts/timing, UTC predecessor ordering, same-raw/normalized mismatch rejection, malformed-manifest rejection, first-capture temporal guard, exploratory descriptive runner, validated PIT gate preserved; 125 backend tests; final independent review `deleg_e5407553` and focused follow-up review `deleg_0e2282c8` passed with empty blocker arrays | collect independent BOT releases and obtain provider PIT evidence; do not promote revised history |
|
||
| Tourism deterministic signal | complete | live foreign-arrivals YoY surprise | add occupancy/airport metric |
|
||
| Internal paper ledger | complete | atomic local JSON persistence and restart test | shared store before multi-worker deployment |
|
||
| Dashboard | complete | Vite build + served source check with live-sign copy | visual browser capture after permission is available |
|
||
| General backtest accounting | complete/blocked | flat-price RED/GREEN invariants; 238 backend tests; Vite build; dividend proxy and non-PIT UI disclosures | replace final-holdings yield proxy with dated dividend cash flows; add provenance-validated PIT scorer |
|
||
| LLM analysis | deferred | intentionally no LLM dependency in M0 | add after signal lineage is stable |
|
||
| Webhook receiver | deferred | contract only, no external receiver | choose after core app is usable |
|
||
| MT5 bridge | deferred | not started | paper bridge after webhook decision |
|
||
|
||
## Guardrails
|
||
|
||
- Research and paper modes only.
|
||
- No live orders, external webhook receiver, broker credentials, or MT5 connection.
|
||
- NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED].
|
||
- Deterministic signal is authoritative; LLM will remain downstream.
|
||
- Fixture and provisional BOT sources are explicitly labelled; neither is investment-ready without validation.
|
||
- `target_weight` is recorded in the internal paper ledger; it is not an order.
|
||
- Local paper demo mode is explicit, loopback-only, visibly warned, and never a live-order authorization path.
|
||
- Shared/network paper writes require protected token mode and a server-side session.
|
||
|
||
## Verification
|
||
|
||
- Backend: 93 unittest tests pass, including paper-auth mode, malformed-config, bind-host, PIT archive, immutable-write, and market-timezone no-lookahead regressions.
|
||
- Independent M1 review: **PASSED**; no concrete security or logic blockers.
|
||
- Reviewer suggestions: set `PAPER_COOKIE_SECURE=1` outside local HTTP; replace in-memory sessions before multi-worker deployment.
|
||
- M1 reviewer backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
|
||
- Frontend: `npm run build` passes with Vite.
|
||
- Backend health endpoint returns HTTP 200 JSON.
|
||
- Dashboard served HTML contains the current title, Vue mount point and Vite entry.
|
||
- Paper ledger POST and readback work through the live API.
|
||
- BOT source live fetch parsed 138 monthly periods and persisted raw HTML plus normalized snapshot.
|
||
- Data-health reports `source_mode=bot`, `status=provisional`, and `replayable=true`.
|
||
- Live vintage replay returned the same theme surprise as the current dashboard summary.
|
||
- Vintage collector preserved one live `vintage_id` with `seen_count=4` and point-in-time API excluded it before `published_at`.
|
||
- Independent M2 review: **PASSED**; no concrete security or logic blockers.
|
||
- Event-study readiness gate correctly returns HTTP 409 with 1/12 independent vintages; no backtest result is fabricated.
|
||
- Independent M2.3 review: **PASSED**; no concrete security or logic blockers.
|
||
- Price snapshot normalized 9 symbols with adjusted close and provider-derived trading dates; quality is explicitly `revised_vendor_history` and `point_in_time=false`.
|
||
- Backtest gate requires both independent vintages and point-in-time prices.
|
||
- Independent M2.4 review: **PASSED**; no concrete security or logic blockers.
|
||
- Research runner persists blocked/ready reports with frozen input IDs, hashes, configuration and gate reasons.
|
||
- Revision-aware readiness counts `(source_id, published_at)` once and runner selects the latest revision only.
|
||
- Event-study defaults to next trading-session execution and handles non-trading event dates without skipping the first session.
|
||
- Paper ledger persists atomically under ignored `backend/data/paper/ledger.json` when configured.
|
||
- Live API `0.5.0` created and replayed the same blocked Tourism research report; UI served the research-run panel and human-readable gate reason.
|
||
- Independent M2.5 review: **PASSED**; no concrete security or logic blockers.
|
||
- Integrity hardening: normalized snapshots and raw payloads are bound to manifest metadata; malformed boolean flags, stale manifests, and cached-ready replay after tampering are rejected.
|
||
- Canonical hash algorithm is explicit: `sha256-json-canonical-v1` using sorted-key compact UTF-8 JSON after excluding only the normalized hash field.
|
||
- Integrity scope is local artifact/corruption detection. A hostile machine owner who can rewrite code, manifests, raw files and runtime environment is outside this local research app's threat model.
|
||
- Independent integrity-hardening review: **PASSED** under the local threat model via the schema-corrected exact verdict; prior pre-fix findings are recorded as remediated.
|
||
- Replay now uses the canonical `VintageStore.load_snapshot()` validation path; tampered normalized snapshots return HTTP 422 instead of being recomputed.
|
||
- Research reports carry a canonical content hash; manifest entries carry their own hash and bind the report file/hash/metadata. Tampered reports, manifest hashes/metadata, and malformed report shapes fail closed.
|
||
- Research input lineage now records normalized snapshot hashes and hash algorithms alongside raw payload hashes.
|
||
- Manifest JSON shape and every `list_runs()`/`latest()` entry are validated before metadata is returned or used for selection.
|
||
- Persist validates every existing manifest entry before returning or writing a new report, and refuses malformed entries without leaving an orphan report.
|
||
- Direct report loads also bind the manifest entry identity to the requested `run_id`.
|
||
- Invalid UTF-8 in persisted manifest/report files is converted to `ResearchRunError` instead of escaping as a decode exception.
|
||
- VintageStore rejects non-object snapshots, invalid UTF-8 manifests/snapshots, and raw-payload read failures with `VintageStoreError` instead of leaking attribute/decode/I/O exceptions.
|
||
- Tourism replay computation rejects non-list/non-object observation/exposure shapes as controlled semantic failures; malformed exposure replay returns HTTP 422 instead of leaking `AttributeError`.
|
||
- Latest BOT collector/readiness check fetched the same `source_id`/`published_at`; `seen_count=16` but independent releases remain `1`, and the backtest gate remains HTTP 409 with `insufficient_vintages`.
|
||
- Browser visual capture was blocked by Chrome remote-debugging permission; no permission dialog was clicked.
|
||
- Paper auth policy: `PAPER_AUTH_MODE=demo` permits local paper writes without a session token only on loopback; `PAPER_AUTH_MODE=token` remains fail-closed without a configured server token.
|
||
- Frontend surfaces the demo warning returned by `/api/v1/auth/paper`; no token is embedded in the bundle.
|
||
- Independent paper-auth review initially found malformed non-string token configuration being coerced and an ambiguous disabled-session UI state; fixed with strict token typing, explicit `enabled` status, UI gating, and regressions.
|
||
- Fresh final paper-auth review returned schema-valid `passed=true` with empty security-concern and logic-error arrays; non-blocking suggestions remain for request-token type, frontend integration, and startup precedence coverage.
|
||
- The separate future PIT-price-contract delegation ended interrupted without a complete recommendation; no implementation was accepted, and revised vendor history remains `point_in_time=false`/blocked.
|
||
- The original staged integrity-review result (`deleg_43348a3b`) reported `passed=true`/`findings=[]` but violated the required four-key verdict schema; its corrected follow-up is recorded below.
|
||
- Schema-correction review `deleg_a1c721d2` returned exactly `{"passed":true,"security_concerns":[],"logic_errors":[],"suggestions":[]}`; the staged integrity-remediation scope is now approved under the stated local threat model. Interrupted parallel review `deleg_e2d54c30` contributes no evidence.
|
||
- PIT price contract: `point_in_time=true` now requires `quality=point_in_time_archive`, `archive_contract=pit-daily-v1`, provider release/evidence metadata, per-series IANA timezone, per-bar `session_date`/`known_at`/OHLC/adjusted-close/volume, and immutable manifest/hash binding. The event-study runner rejects missing or market-locally future-known prices when PIT mode is enabled.
|
||
- PIT verification: 93 backend tests, compileall, Vite build, npm audit (0 vulnerabilities), diff checks, and credential-pattern scan passed. The live revised Yahoo snapshot remains `point_in_time=false`; `/api/v1/backtest/tourism?min_events=1` remains HTTP 409 `blocked` with `price_series_not_point_in_time`.
|
||
- Final bounded independent PIT review `deleg_c08f6dd8` returned the exact required verdict `{"passed":true,"security_concerns":[],"logic_errors":[],"suggestions":[]}`. Earlier reviewer timeouts were treated as non-approving and contributed no evidence.
|
||
- Price-provider feasibility pass recorded in `docs/engineering-log/2026-08-24-price-provider-feasibility.md`: public SET/SETSMART/SMART Marketplace pages establish historical/API availability but not release-time PIT semantics; ICE SET is a strong commercial candidate; LSEG S3 Direct has the strongest public PIT claim but still needs SET-specific coverage and vintage evidence; Databento confirms SET venue presence and PIT corporate-action records but not PIT price-vintage semantics. No provider currently passes `pit-daily-v1` from public evidence.
|
||
- No price adapter or configuration promotion was made during the feasibility pass. The next adapter must be gated on a raw sample, provider release metadata, known-at definition, correction/revision example, symbol coverage, and immutable replay evidence. Citation ledger verification for the feasibility document passed with evidence quotes for all 11 cited sources.
|
||
- M2.9 forward price observation layer records every retrieval against a stable source/period scope, preserves immutable raw/snapshot files, binds each observation ID to its prior hash and diff, tracks first/last seen times, rejects tampered observation records, and rejects out-of-order writes before creating snapshot/raw artifacts. Repeated unchanged payloads do not create a new snapshot or a new research run; runtime observation timestamps are excluded from the run fingerprint.
|
||
- Price integrity remediation requires explicit non-PIT normalized schema/series-to-raw lineage, validates PIT numeric input without leaking `OverflowError`, parses predecessor order in UTC, and enforces observation counts/timing for contract-backed entries. Pre-observation legacy manifests remain readable when they have no observation contract; new contract-backed entries are fully cross-checked.
|
||
- Research now has explicit `validated` and `exploratory` modes. Validated mode retains the PIT/known-at fail-closed gate. Exploratory mode may use revised vendor history only with `require_price_known_at=false`, returns `result_scope=non_pit_descriptive_only` and top-level `status=descriptive_only`, and carries limitations; it cannot promote `point_in_time` or produce a validated backtest claim.
|
||
- `/api/v1/prices/observations` exposes the local observation audit trail; `/api/v1/prices/health` reports observation count, last observation time, and revision status. The UI's research button explicitly requests exploratory mode and maps machine-readable status/reason codes to human-readable labels.
|
||
- Current M2.9 verification: 125 backend tests, compileall, Vite build, npm audit (0 vulnerabilities), diff checks, static credential/dangerous-pattern scan, and current-data Flask health smoke test passed. Provider release semantics remain unproven and the live revised Yahoo snapshot remains `point_in_time=false`.
|
||
- Fresh final M2.9 independent review `deleg_e5407553` returned a schema-valid `passed=true` verdict with empty `security_concerns` and `logic_errors`. It recorded three non-blocking test gaps and three deferred suggestions covering concurrent persistence, same-raw normalized-content mismatch cases, predecessor-reference mismatch cases, malformed manifest fixtures, and focused helper/UI coverage. No commit or push has been made.
|
||
- M2.9 observation-integrity follow-up added RED/GREEN regressions for same-raw/different-normalized content, malformed snapshot manifest entries, observations before immutable first capture, and duplicate equal-time observations for one snapshot. The focused price suite and full backend suite pass at 125 tests; independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty `security_concerns` and `logic_errors` (suggestion: retain the new regression coverage). Process-level locking and focused helper/UI coverage remain deferred.
|
||
- Audit + fix pass (2026-08-26, plan at `docs/audit-and-plan-2026-08-26.md`): triple-confirmed the declarative `FACTORS`/`THEMES` framework is by-passed by hand-written scoring in `dashboard._theme_surprises`, and that `/api/v1/simulation` recomputed a divergent 3-theme path. Fixed P1 (simulation reuses the canonical board via `default_scores` — live check: sim top pick PTT == top board combined 1.600), P2 (dashboard now emits `source_summary{factor_keys:11, rows:6}`; frontend shows "N ปัจจัย · M แหล่ง"), and P5-partial (removed dead `list_themes`/`Theme`/`build_theme_scores`/`_map_index` + the tests that locked them; added `backend/tests/conftest.py` so pytest needs no `PYTHONPATH`). Deferred P0 (registry-driven re-baseline) and P3/P4 (point-in-time backtest + factor-weight learning) pending explicit scope/baseline sign-off. Full backend suite: **203 tests pass**; Vite build passes. This work is own-engine review gated before commit.
|
||
- P0-B + P3 + P4 (commit `8db3d48`, reviews `deleg_fe6f45cd` + `deleg_718218f8` both `passed=true`): the declarative FACTORS/THEMES registry is now the single source of truth (`compute_theme_surprises` reads registry; hand-written per-theme blocks removed; FACTORS carries center/span normalization spec; bank NPL wired into banks). P3 rebuilt `run_backtest` as a real multi-rebalance PIT engine (`leakage_guard`, `momentum_at` true 12-1). P4 added factor-weight learning (Spearman IC -> `apply_weight_update`) + `/api/v1/learning/momentum`. Live result: momentum IC=0.012, t=0.132 over 22 periods (no reliable predictive power in this SET50 window). 226 tests pass.
|
||
- Follow-up (A+B): theme surprises now weight-normalized by total |weight| so cross-theme magnitudes are comparable (retail 0.189->0.145). Added append-only `FactorHistory` store (`data/factor_history/<key>.jsonl`) that records every FACTORS value each scheduler run, wired into `refresh_all` (non-fatal), plus `/api/v1/learning/factors` readiness endpoint. Macro/demographic factors start at n=1 and become learnable (P4) as history accumulates. 233 tests pass.
|
||
- Backtest accounting remediation (2026-08-27): deterministic flat-price tests exposed that ending holdings value was reported as `price_pnl` and the dividend proxy was excluded from final value/net return. The corrected identity is `final_value = capital + price_pnl + dividend_income`; API now emits `dividend_method=final_holdings_yield_proxy`, and both result/history UI paths label the proxy and warn when `leakage_guard=false`. RED failures reproduced `price_pnl=20,000/49,000`; GREEN verification: 239 backend tests, compileall, Vite build, npm audit 0 high-severity vulnerabilities, served-bundle disclosure check, static scan and diff check passed. Fresh final independent review `deleg_10918fed` passed with empty blocker arrays. Evidence: `docs/engineering-log/2026-08-27-backtest-accounting-remediation.md` and `docs/test-evidence/2026-08-27-backtest-accounting-remediation.md`. The dividend model remains a proxy and the default public backtest remains descriptive non-PIT.
|
||
- PIT factor store + partial score provider (2026-08-27, commit `1f630be`): added `factor_vintages.py` (append-only PIT factor store: observed/released/retrieved_at + SHA-256 hash chain; `value_at(as_of)` anti-look-ahead, fail-closed when nothing released by as_of) and `pit_scorer.py` (`PitScoreProvider.score_board(as_of)` + `make_pit_score_fn`). `backtest._resolve_scores` now sets `leakage_guard=True` only when the supplied score_fn asserts `pit_meta.pit=True` (closes "any callable ⇒ PIT"); `/api/v1/backtest` accepts `use_pit`. Empty factor store fail-closes (live probe `leakage_guard=false`). Full backend **255 passed** (was 239). Honest scope: theme dimension PIT from this store forward; no factor history before 2026-08-27 so pre-today backtests stay non-PIT; siamchart fundamental partial (EPS 5-yr series PIT-grade, ratios current → `partial_pit`). Next: dated dividend cash-flow ledger.
|
||
- Dated dividend cash-flow ledger (2026-08-27, commits `068dff2` + `5b8b6d1`): added `dividend_ledger.py` (per-symbol dated store: ex_date/record_date/pay_date/per_share/source/estimate flag; `credit_dividends` credits per_share×qty when payment due, fail-closed with no entry; `build_dps_ledger` builds per-share estimate rows from siamchart ratios.DPS). `run_backtest` accepts `dividend_ledger` and reports `dividend_method` = `dated_ledger` (real rows) | `dps_annual_proxy` (estimate) | `final_holdings_yield_proxy` (legacy). `/api/v1/backtest` accepts `use_ledger`. Frontend maps method to an honest Thai label + Proxy badge on result card and history rows. Full backend **266 passed** (was 255). Honest scope: DPS rows are estimates (no ex-date history in snapshot yet); real dated cash flows require collecting per-stock dividend history, which upgrades a symbol to `dated_ledger`.
|
||
- PIT siamchart vintage store (2026-08-27, commit `887e9c9`): added `siamchart_vintages.py` (append-only hash-chained store of every collected siamchart snapshot; `snapshot_at(as_of)` returns the newest snapshot with `retrieved_at <= as_of` — anti-look-ahead, fail-closed). `PitScoreProvider` accepts `siamchart_store`; when wired the fundamental dimension reads the snapshot knowable at as_of (`pit_grade='pit'`) and `partial_pit` no longer forces true. `/api/v1/backtest use_pit` seeds the first vintage from the current snapshot. Full backend **273 passed** (was 266). Honest scope: snapshot reconstruction forward only; EPS year-keys not tied to calendar years (growth = latest-vs-prior).
|
||
- Real forward-test lifecycle (2026-08-27, commits `6d9c283` + `80c6d79`): the cosmetic "forward" (same single-pass backtest, different mode string) is replaced with a durable, frozen-signal paper-portfolio lifecycle. `forward_test.py` `ForwardTestStore` (thread-safe JSON store) with status flow: frozen (signals snapshotted immutable) → executed (fills 50/20/30 at post-freeze prices) → marked (mark-to-market equity series) → matured (net_return). New routes: `GET /api/v1/forward(+<id>)`, `POST /api/v1/forward` (create+execute, `use_pit` freeze), `POST /<id>/mark`, `POST /<id>/mature`; store at `data/forward/runs.json` (survives restarts). UI simulation tab: forward calls `/api/v1/forward`, loads runs, shows status/non-PIT/holdings + Mark/Mature per run. Full backend **280 passed** (was 273). Honest scope: score source at CREATE may be current board (`non_pit=true` tagged); paper-only.
|