kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a7daf333b24756354dd5e3514fed66e43bf64964
set50-system/docs/HANDOFF.md

35 KiB
Raw Blame History

Handoff — Tourism Vertical Slice

Project

  • Path: /Users/kunthawat/Gitea/set50-alternative-data-platform
  • Branch: main
  • Verified code commit: pending; final M2.9 independent remediation review passed; no commit or push performed
  • Current milestone: M2.9 forward price observations and explicit research modes complete/blocked; observation-integrity follow-up implemented; exploratory research is usable, validated backtest remains blocked
  • Mode: research + paper only
  • Frontend: Vue 3 + Vite
  • Backend: Flask 0.5.0
  • Current runtime source: BOT Tourism Indicators (TOURISM_SOURCE=bot)

Completed

  • Tourism snapshot schema with source, publication, retrieval, vintage and raw hash fields.
  • Deterministic Tourism Pulse score: standardized surprise × exposure × confidence.
  • BOT ASP.NET form adapter with monthly history parsing and trailing 12-point YoY baseline.
  • Atomic raw HTML and normalized snapshot persistence under ignored backend/data/.
  • Read-only /api/v1/data-health endpoint.
  • Safe /api/v1/replay/tourism?vintage_id=... endpoint with identity and path validation.
  • Immutable VintageStore manifest with first-seen, last-seen, seen-count and revision metadata.
  • One-shot collector: backend/scripts/collect_tourism_vintage.py.
  • Point-in-time vintage query: GET /api/v1/vintages?as_of=<ISO-8601>.
  • Deterministic event-study engine with window, benchmark and cost calculations.
  • Backtest readiness gate: GET /api/v1/backtest/tourism?min_events=12.
  • Yahoo-backed daily price snapshot with SET symbol mapping and adjusted-close bars.
  • Price health: GET /api/v1/prices/health.
  • Forward price observation archive: GET /api/v1/prices/observations; repeated payloads reuse immutable snapshots while recording retrieval observations, first/last-seen times, prior hashes, revision status and structured diffs.
  • Frozen research runner: POST /api/v1/research/tourism/run and GET /api/v1/research/tourism/latest.
  • Research runner modes: explicit validated (PIT fail-closed) and exploratory (revised-history descriptive only); the UI calls exploratory mode explicitly and shows result_scope.
  • Headless runner: backend/scripts/run_tourism_research.py.
  • Durable atomic paper ledger under backend/data/paper/ledger.json when configured.
  • Raw and normalized snapshot integrity binding with explicit sha256-json-canonical-v1 metadata.
  • Replay now loads through the canonical VintageStore.load_snapshot() path and fails closed on tampered snapshots.
  • Research reports carry canonical content hashes; manifest entries carry their own hashes binding report filename, report hash, and metadata.
  • Research reports from pre-hash trusted stores can be migrated explicitly with backend/scripts/migrate_research_report_hashes.py; automatic legacy reads remain disabled.
  • Research input lineage records normalized snapshot hashes and hash algorithms alongside raw payload hashes.
  • PIT daily price contract pit-daily-v1 is implemented: explicit provider release/evidence metadata, per-series IANA timezone, per-bar session date/known-at/OHLC/adjusted-close/volume, and immutable manifest/hash binding are required before point_in_time=true is accepted.
  • PIT event-study mode rejects missing or market-locally future-known prices; the research runner enables this check for PIT snapshots.
  • Public price-provider feasibility is documented in docs/engineering-log/2026-08-24-price-provider-feasibility.md; SET Historical Data, SETSMART, SMART Marketplace, ICE SET, LSEG Tick History/S3 Direct, Databento, and Exchange Data International (EDI) have all been checked, but no candidate has yet supplied enough evidence to pass pit-daily-v1, so no new adapter was added. EDI is the closest conditional price-feed lead because its public docs expose SET coverage, a confirmed ETA, and correction/re-push behavior, but its public contract still lacks immutable release metadata.
  • VintageStore rejects non-object snapshots, invalid UTF-8 manifests/snapshots, and raw-payload read failures with VintageStoreError.
  • Tourism replay computation rejects non-list/non-object observation/exposure shapes as controlled semantic failures, so malformed exposure snapshots fail closed instead of leaking AttributeError.
  • Manifest JSON shape and every list_runs()/latest() entry are validated before metadata is returned or used for selection.
  • Ranked target weights and LONG/SHORT/NEUTRAL classification.
  • English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields.
  • Explicit loopback-only paper demo mode with a visible warning, protected token mode for shared/network use, and internal paper ledger.
  • No external webhook or MT5 integration.

Independent M1 review passed with no concrete security or logic blockers. Non-blocking backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.

Current live vintage

source_id: bot.ec_ei_028_s2
published_at: 2026-07-31T14:30:00+07:00
as_of: 2026-06-30
status: provisional
available_periods: 138
history_points: 12
vintage_id: bot.ec_ei_028_s2-2026-07-31-665981f88b4a
seen_count: 16
raw_payload_hash: 665981f88b4a30c5bd30026cf1e96279c244ad83725558f4136d952b29756a31
theme_surprise: -0.35114754

Verified commands and live checks

PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
Ran 125 tests ... OK

PYTHONPATH=backend .venv/bin/python -m compileall -q backend/app backend/scripts backend/tests
passed

npm run build
Vite build completed successfully.

npm audit --omit=dev --audit-level=high
found 0 vulnerabilities

git diff --check
passed

python3 ~/.hermes/skills/research/grounded-citations/scripts/sources.py verify docs/engineering-log/2026-08-24-price-provider-feasibility.md --strict --evidence
citations OK; 11 cited sources, all with evidence quotes

GET /api/v1/health
HTTP 200; {"mode":"research","status":"ok","version":"0.5.0"}

GET /api/v1/data-health
HTTP 200; source_mode=bot, status=provisional, replayable=true

GET /api/v1/replay/tourism?vintage_id=bot.ec_ei_028_s2-2026-07-31-665981f88b4a
HTTP 200; replay theme surprise matched live summary exactly.

GET /api/v1/vintages?as_of=2026-07-01T00:00:00Z
HTTP 200; count=0

GET /api/v1/vintages?as_of=2026-08-01T00:00:00Z
HTTP 200; count=1; manifest seen_count=16

GET /api/v1/backtest/tourism?min_events=12
HTTP 409; status=blocked, available_events=1, required_events=12, price_series_required=true

GET /api/v1/prices/health
HTTP 200; available=true, quality=revised_vendor_history, point_in_time=false, symbols=9

POST /api/v1/research/tourism/run (validated default)
HTTP 200; report status=blocked, reason=insufficient_vintages, immutable run_id persisted

POST /api/v1/research/tourism/run (mode=exploratory, min_events=1)
HTTP 200; revised-price report runs only as result_scope=non_pit_descriptive_only

GET /api/v1/research/tourism/latest
HTTP 200; same run_id returned on replay, report_hash present

GET /api/v1/prices/observations
HTTP 200 on empty archive; populated archives return bound observation IDs, prior hashes and revision diffs

BOT collector rerun
same published release; seen_count=16; independent releases remain=1

M2.9 remediation status
125 tests and current-data health smoke test pass; observation-integrity follow-up covers same-raw/normalized mismatch, malformed manifest entries, first-capture temporal ordering, and duplicate equal-time observations; focused independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty security/logic blocker arrays; no commit or push performed

Paper writes support two explicit policies: local PAPER_AUTH_MODE=demo requires a loopback bind and allows paper-only writes without a session token while showing a warning; protected PAPER_AUTH_MODE=token uses a server-side token exchange and HttpOnly paper_session cookie. The token is not embedded in the frontend bundle.

Known limitations

  • BOT data is provisional and may be revised. The vintage/hash contract preserves the fetched version, but this is not a final-data guarantee.
  • Only foreign-arrival YoY is live in this slice; occupancy and airport passenger metrics are not yet connected.
  • Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
  • No investment edge, transaction-cost result, or backtest conclusion has been established.
  • One independent source release is not enough for a valid event study; current historical rows are not treated as point-in-time vintages.
  • The event-study engine is deterministic and tested. Its PIT mode now enforces per-bar known-at semantics in the market timezone. Yahoo price history is connected for plumbing, but it is revised vendor history, not point-in-time data.
  • The PIT contract is implemented but provider release-time evidence is still absent. No configuration-only promotion of revised history is allowed; point_in_time remains false.
  • SET Historical Data, SETSMART, and SMART Marketplace document historical/API availability but do not publicly document the release-vintage and known-at semantics required by the PIT contract. ICE SET is a strong commercial candidate; LSEG S3 Direct has the strongest public PIT claim but remains conditional on SET coverage and vintage evidence. Databento lists the SET venue and documents PIT corporate-action history, but its public price-data material does not establish price release IDs, client availability/known_at, or correction-vintage replay. EDI documents confirmed/unconfirmed republish and Repush1/2/3 correction files for SET-covered EOD pricing and publishes a confirmed Thailand ETA, but also states that it has no EOD time-series database; immutable release IDs, actual release timestamps, retention, and retrievable manifests remain unproven. All candidates require a provider evidence packet before implementation.
  • Validated research runner is operational and replayable, but correctly emits a blocked report until both evidence gates pass. Exploratory runner is operational against revised history only with an explicit descriptive-only scope.
  • Snapshot loads fail closed when raw files, normalized payloads or manifest metadata do not match their recorded hashes.
  • Research report and manifest-entry loads fail closed when persisted content or bound metadata is tampered; pre-existing research volumes require integrity metadata migration before they can be loaded.
  • Independent integrity review gate passed for the staged remediation scope under the local threat model: deleg_c227c439 found a persist-wide manifest-validation gap in the prior staged tree; it is now fixed and covered by a regression test. The original deleg_43348a3b result was semantically positive but schema-invalid because it used findings; schema correction deleg_a1c721d2 returned the exact required four-key verdict with passed=true and all arrays empty. Earlier timeouts/stale reviews (deleg_cf1da27d, deleg_feb40653, deleg_87a0f686) and interrupted parallel review deleg_e2d54c30 are non-approving and contribute no additional evidence.
  • The hash boundary protects local artifacts against corruption/partial writes; hostile host-level rewrite of code, manifests and runtime environment is outside this local threat model.
  • Revisions are deduplicated by canonical (source_id, published_at); the latest observed revision is selected once.
  • Event-study default execution is the next trading session, including weekend/holiday event dates.
  • Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead.
  • Paper demo startup rejects non-loopback PAPER_BIND_HOST; token mode remains the required policy for shared/network deployment.
  • Fresh bounded independent review of the paper-auth scope passed with security_concerns=[] and logic_errors=[]; its remaining items are non-blocking test suggestions. Integrity remediation is tracked separately and passed via the schema-corrected exact verdict.
  • M2.9 remediation now covers malformed provider/config totalization, non-PIT series/raw lineage, PIT numeric overflow handling, UTC/equal-time predecessor ordering, same-raw/normalized mismatch rejection, malformed snapshot manifest rejection, first-capture observation timing, contract-backed observation count/timing/reference checks, descriptive-only exploratory status, and human-readable UI reason labels. Focused independent review deleg_0e2282c8 returned schema-valid passed=true with empty security and logic blocker arrays; its only suggestion is to retain regression coverage. Process-level locking and focused helper/UI coverage remain deferred. No commit or push has been made.

Exact next action

Keep collecting independent BOT releases until 12 genuinely distinct (source_id, published_at) releases exist. In parallel request one complete market-data provider evidence packet (EDI first, or ICE/LSEG/SET fallback): raw sample, provider release ID/time, explicit known-at semantics, correction/revision replay example, SET symbol coverage, timezone/session rules, and immutable archive/manifest evidence. Databento is currently a reference/corporate-actions lead only and must not be used to claim PIT price bars. Revised Yahoo history remains point_in_time=false. Do not raise the PIT flag or promote a PIT adapter until that packet passes pit-daily-v1. The operator can run exploratory research now, but only validated mode may support a historical backtest claim. Keep local paper work in explicit demo mode and use token mode before any shared/network deployment. Add a trusted deployment signing key before treating the app as a multi-user or hostile-host service.

NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED].

Session 2026-08-27 — Backtest Accounting Remediation

Branch: main · Base HEAD: b362cc3 · Commit/push: not performed

Completed

  • Added RED/GREEN flat-price accounting invariants in backend/tests/test_backtest.py.
  • Corrected general backtest price P&L to compare ending pre-dividend equity with initial capital.
  • Included the final-holdings dividend-yield proxy in final_value and net_return.
  • Added machine-readable dividend_method=final_holdings_yield_proxy.
  • Updated the Vue backtest panel to label estimated dividends and warn when leakage_guard=false.

Verified

  • RED: flat-price regressions failed with incorrect price_pnl=20,000 and 49,000.
  • GREEN: flat/no-dividend, flat/proxy-dividend, rising/no-dividend, and multi-rebalance accounting checks passed; targeted suite 11 passed.
  • Full backend suite: 239 passed.
  • Python compileall passed.
  • Vite production build passed.
  • npm audit reported 0 high-severity vulnerabilities.
  • Added-line security/dangerous-pattern scan and git diff --check passed.
  • Independent review cycle 1 failed closed on missing proxy/non-PIT disclosure in saved-run history. A fresh fix agent added ประมาณการปันผล (Proxy) and per-row descriptive non-PIT badges. Fresh final review deleg_10918fed inspected the current post-hardening diff and returned passed=true with empty security, logic, and suggestion arrays.

Known limitations

  • dividend_income is still estimated from final holdings and final dividend yield; it is not a dated cash-flow ledger.
  • The default public backtest still applies current scores historically, so leakage_guard=false and the result remains descriptive non-PIT.
  • A supplied scorer toggles leakage_guard=true, but provenance and release-time semantics are not independently certified.
  • Backtest run history remains in process memory.

Exact next action

Implement dated dividend events and a provenance-validated PIT score provider before using results as strategy-performance evidence. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — Single-container Docker packaging (verified + pushed)

Branch: main · HEAD: f9b8cd1 (pushed)

Completed this session (verified + pushed)

  • Dockerfile: python:3.11-slim + nginx. Serves the PREBUILT frontend/dist SPA (committed bundle, no node/npm in the image), reverse-proxies /api → 127.0.0.1:5000, VOLUME [/app/backend/data] for persistence (factor vintages, dividend ledger, forward runs, prices), HEALTHCHECK on /api/v1/dashboard/summary, exposes :80.
  • deploy/nginx.conf: SPA fallback + /api proxy + gzip.
  • deploy/entrypoint.sh: starts Flask (HOST=0.0.0.0:5000) + nginx in foreground, forwards INT/TERM.
  • docker-compose.yml: service set50, port 8080:80, volume ./backend/data, healthcheck.
  • .dockerignore (excludes .git/.venv/backend/data/node_modules/docs) + .gitignore now tracks frontend/dist/ (prebuilt bundle); backend/data/ stays untracked.

Verified

  • Backend runtime: test_client GET /api/v1/dashboard/summary → 200 (the HEALTHCHECK path).
  • sh -n deploy/entrypoint.sh, compose YAML parses, all Dockerfile-referenced files exist on remote origin/main.

Honest caveat

  • No local docker on this machine → the image itself was NOT built/run here. The first real build happens on EasyPanel (or any docker host). If the build fails there, the most likely causes are env-var / volume mapping, which the compose file already covers for self-host.

Exact next action (user-side)

  1. On EasyPanel create a service from this repo (build type Dockerfile — not Nixpacks).
  2. Mount a persistent volume at /app/backend/data (or set DATA_DIR env to a mounted path) so the dividend ledger / factor vintages / forward runs survive redeploys.
  3. Publish port 80 (default). Add a webhook in Gitea → EasyPanel deploy URL on push to main (Gitea Webhook, not Gitea Actions — no managed runners).
  4. Optional: set DIVIDEND_REFRESH_COOLDOWN_SECONDS / REFRESH_INTERVAL_SECONDS.

Session 2026-08-27 — Auto-refresh dated dividend ledger (verified + pushed)

Branch: main · HEAD: 03195dc (pushed)

Completed this session (verified + pushed)

  • AppDataScheduler now calls _maybe_refresh_dated_dividends() in refresh_all — a cooldown-gated (default 6h) fetch of real dated dividend history into data/dividends/ledger.json, so the ledger stays fresh automatically without manual POST /api/v1/dividends/update and without re-hammering Siamchart every refresh tick (dividend history changes only a few times a year).
  • DIVIDEND_REFRESH_COOLDOWN_SECONDS env toggle (default 21600). Network failure is non-fatal: the previous ledger stays intact.
  • Tests: cooldown fires once then skips, refetches after it elapses (2) — full backend 294 passed (was 292).

Verified

  • Backend 294 passed; compileall, git diff --check, static scan clean.

Exact next action

None required — the SET50 alternative-data platform PIT/enabler work is complete. Optional UI-only polish: surface the dated-ledger synthesis (dated vs proxy) more explicitly in the backtest result card. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — Real dated dividend-history collector (verified + pushed)

Branch: main · HEAD: f9973e8 (pushed 59c97b5..f9973e8)

Completed this session (verified + pushed) — the last PIT milestone

  • Dated dividend collector backend/app/siamchart.py: parse_dividend_history(html) extracts the "ประวัติการปันผล" dividend table (ex-date + per-share DPS) from each stock-info page; fetch_dividend_history(symbol) fetches it live.
  • Ledger upgrade backend/app/dividend_ledger.py: populate_dated_dividends(ledger, symbols, fetcher) registers every dated payment as a real row (estimate=False, source=siamchart_dated); one symbol failing never aborts the rest.
  • API backend/app/__init__.py: DividendLedger persisted at data/dividends/ledger.json; POST /api/v1/dividends/update fetches all snapshot symbols and saves; use_ledger backtests prefer the dated ledger (dividend_method=dated_ledger) and fall back to DPS estimates when unpopulated.
  • Tests: parser (4) + populate (2) — full backend 292 passed (was 286).

Verified (real network!)

  • The sandbox HAS outbound network (contrary to the earlier "needs external access" deferral): curl/https to siamchart.com returns 200.
  • Live POST /api/v1/dividends/update: fetched 49/49 symbols, 1410 dated payments in ~11s.
  • Live use_ledger backtest then reports dividend_method=dated_ledger (real ex-date × qty) instead of DPS proxy.

Honest scope / notes

  • Every snapshot symbol now has real dated dividend history (PTT: 45 rows from 2002; ADVANC: 51; AOT: 22; AWC: 6, etc.).
  • use_pit scoring remains non-PIT when a run is fed current-board scores; the dated ledger is orthogonal (cash-flow model), not a PIT-scorer flag.

Exact next action

All requested milestones are complete and pushed. Optional follow-ups: (a) schedule /api/v1/dividends/update into the data refresh loop, (b) surface the dated-ledger synthesis (dated vs proxy) more explicitly in the backtest UI. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — Factor-learning validation gate (verified + pushed)

Branch: main · HEAD: ae814c3 (pushed 9700f6b..ae814c3)

Completed this session (verified + pushed)

  • Validation gate backend/app/weight_learning.py: apply_validation_gate(train_ics, holdout_ics) marks a factor validated=True only when total sample >= MIN_SAMPLE_PERIODS (12), each window >= its minimum, train AND holdout IC both beat baseline (BASELINE_IC=0) and agree in sign, and pooled |t| > MIN_IC_TSTAT (1.0). FactorLearning now carries ic_train/ic_holdout/validated/gate_notes.
  • No auto-apply enforced: apply_weight_update now keeps new_weight == old_weight for any unvalidated factor — weights only move after the gate passes with explicit evidence.
  • Gated learner: learn_momentum_gated(...) builds PIT momentum ICs then splits chronologically into train/holdout and applies the gate. Wired into GET /api/v1/learning/momentum, which now surfaces ic_train, ic_holdout, validated, gate_notes.
  • Tests: rewritten suite + gate (16) — full backend 286 passed (was 280).

Verified

  • Backend full suite 286 passed; compileall, git diff --check, static scan clean.
  • Live probe on the current price archive: momentum validated=false, ic_train=0.0711, ic_holdout=-0.1143, gate_notes=['IC not above baseline (0.0711/-0.1143)'], new_weight=None — the honest outcome: momentum is not validated, weight stays unchanged.

Honest scope / known limitations

  • learn_momentum_gated is the only factor with a real PIT series; macro/demographic factors are still unattributable (no vintages) and stay blocked.
  • The gate is intentionally strict; a validated factor merely suggests a weight delta — applying it to production is a separate, explicit step.

Exact next action

All six planned PIT/enabler milestones are complete and pushed. Remaining (deferred): a real per-stock dividend-history collector (siamchart /stock-info/<SYMBOL>/) to upgrade symbols from dps_annual_proxy to dated_ledger (needs live network access). Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — Real forward-test lifecycle (verified + pushed)

Branch: main · HEAD: 80c6d79 (pushed 6d9c283 + 80c6d79)

Completed this session (verified + pushed)

  • ForwardTestStore backend/app/forward_test.py: durable, thread-safe JSON store of forward paper runs with an explicit lifecycle — frozen (signals snapshotted, immutable) → executed (fills 50/20/30 buckets at post-freeze prices) → marked (mark-to-market equity series) → matured (net_return finalised). Frozen signals are never re-readable/rewritten after creation, so later data cannot retroactively change the run's decision.
  • API backend/app/__init__.py: GET /api/v1/forward(+<id>), POST /api/v1/forward (create+execute, use_pit freeze option), POST /<id>/mark, POST /<id>/mature. Store extension backed by data/forward/runs.json (survives restarts).
  • Frontend frontend/src/App.vue: simulation tab's Forward mode now calls the real /api/v1/forward lifecycle (not the cosmetic /api/v1/simulation with a mode string), loads runs on mount, and renders a forward panel with status badge, non-PIT tag, holdings, and Mark / Mature per run.

Verified

  • Backend full suite 280 passed (was 273); new lifecycle tests (7).
  • compileall, git diff --check, static scan clean.
  • Live route probe (isolated store): create→execute (201, real holdings AWC/CRC/MINT/PTT), list, mark, mature all work; run persists across the probe.

Honest scope / known limitations

  • The score source at CREATE time may be the current board (non_pit=true, tagged). A PIT scorer only marks non_pit=false when its scores assert pit_meta.pit=true.
  • Paper-only; never dispatches a real MT5 order.

Exact next action

Implement the factor-learning validation gate (PIT exposures + forward returns + Spearman IC + holdout/walk-forward + baseline, no auto-apply). Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — PIT Siamchart vintage store (verified + pushed)

Branch: main · HEAD: 887e9c9 (pushed)

Completed this session (verified + pushed)

  • Siamchart vintage store backend/app/siamchart_vintages.py: append-only, hash-chained store of every collected Siamchart fundamental snapshot. snapshot_at(as_of) returns the newest snapshot whose retrieved_at <= as_of (anti-look-ahead, fail-closed when none). Deduplicates identical persists.
  • Fundamental un-partialed backend/app/pit_scorer.py: PitScoreProvider accepts siamchart_store; when wired, siamchart_factor_view(as_of) reads the snapshot knowable at as_of (pit_grade='pit') instead of the current snapshot (pit_grade='current'). score_board sets partial_pit only when no store is present; theme dimension still fails closed (pit=false) unless every theme factor has a released PIT value by as_of.
  • API backend/app/__init__.py: /api/v1/backtest use_pit seeds the first vintage from the on-disk snapshot (idempotent) and wires the store.
  • Tests: store (6) + provider-with-store anti-look-ahead (1) — full backend 273 passed (was 266).

Verified

  • Backend full suite 273 passed; compileall, git diff --check, static scan clean.
  • Live isolated probe: with siamchart store + one released factor, score_board returns partial_pit=false (fundamental PIT via store) and pit=false with a full blocked_theme list (themes still lack PIT releases — correct fail-closed).

Honest scope / known limitations

  • Snapshot reconstruction is forward-only; no history before the first collected snapshot.
  • EPS year-keys inside a snapshot are not tied to calendar years, so EPS growth is latest-vs-prior, not fiscal-year-pinned.

Exact next action

Implement a real per-stock dividend-history collector (siamchart /stock-info/<SYMBOL>/) that appends dated payments into dividend_ledger so symbols upgrade from dps_annual_proxy to dated_ledger. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — Dated dividend cash-flow ledger (verified + pushed)

Branch: main · HEAD: 5b8b6d1 (pushed 068dff2 + 5b8b6d1; backend 068dff2, frontend 5b8b6d1)

Completed this session (verified + pushed)

  • Dated dividend ledger backend/app/dividend_ledger.py: per-symbol dated store with ex_date / record_date / pay_date / per_share / source / estimate flag, validation and persistence. credit_dividends credits per_share × qty once a payment is due (on/after ex-date and pay date); no ledger entry → no credit (fail closed). build_dps_ledger builds per-share estimate rows from the siamchart ratios.DPS (price-independent, a step up from the yield % × market-value proxy).
  • Backtest integration backend/app/backtest.py: run_backtest accepts dividend_ledger; dividend_income comes from the ledger when set and dividend_method reports dated_ledger (real rows) | dps_annual_proxy (estimate) | final_holdings_yield_proxy (legacy, kept when no ledger). BacktestResult gained a dividend_method field surfaced in to_dict.
  • API backend/app/__init__.py: POST /api/v1/backtest accepts use_ledger, wiring the DPS-built ledger (works independently and alongside use_pit).
  • Frontend frontend/src/App.vue: maps dividend_method to an honest Thai label (ปันผล (ตามวันจริง) / ประมาณการปันผล/หุ้น (DPS) / ประมาณการปันผล (Proxy)) and shows a Proxy badge whenever the dividend is an estimate, on both the result card and saved-run history rows.

Verified

  • Backend full suite 266 passed (was 255); new: ledger store/credit (9) + backtest ledger integration (2).
  • compileall, git diff --check, static secret/dangerous scan clean.
  • Live route probe: use_ledger:true → dividend_method=dps_annual_proxy, dividend_income=4151.0 (per-share) vs default proxy 5041.96 (yield×value); price P&L unchanged (5802.89).
  • Frontend build passes (bundle index-DDD-F_cZ.js).

Honest scope / known limitations

  • build_dps_ledger rows are estimates (no ex-date history in the snapshot) — dividend_method=dps_annual_proxy, flagged Proxy in UI. A symbol only reports dated_ledger once real ex-date/per-share history is collected for it.
  • Ledger credit is end-of-run on final holdings for this milestone (not mid-window on holdings-at-ex-date); a future pass can accrue per rebalance window as ex-date history arrives.

Exact next action

Add a real per-stock dividend-history collector (siamchart /stock-info/<SYMBOL>/) that appends dated payments into the ledger so symbols can upgrade from dps_annual_proxy to dated_ledger. In parallel, add the siamchart vintage chain to un-partial the fundamental dimension. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-27 — PIT factor store + partial PIT score provider (verified + pushed)

Branch: main · HEAD: 1f630be (pushed 1b971ac..1f630be)

Completed this session (verified + pushed)

  • PIT factor store backend/app/factor_vintages.py: append-only, provenance-complete store with observed_at / released_at / retrieved_at and a SHA-256 canonical hash chain (tamper/reorder detectable). value_at(as_of) returns only rows whose released_at <= as_of — a real testable anti-look-ahead guarantee; no value released by as_of fails closed (None / have_pit_value=False). Provenance invariants enforced: released >= observed, retrieved >= released, non-finite / naive-timestamp rejected.
  • Partial PIT score provider backend/app/pit_scorer.py: PitScoreProvider.score_board(as_of) rebuilds the per-symbol board from PIT factor values (theme 60% dimension) plus a partial siamchart view (EPS growth from the 5-year series; current ratios flagged partial_pit). make_pit_score_fn adapts it to the engine's ScoreFn; date-only as_of normalized to midnight +07:00.
  • Honest leakage gating backend/app/backtest.py: _resolve_scores now sets leakage_guard=True only when the supplied score_fn's per-symbol meta asserts pit_meta.pit=True. An arbitrary callable with no PIT proof is no longer treated as PIT — closes the "supplied fn ⇒ PIT" hole.
  • API backend/app/__init__.py: POST /api/v1/backtest accepts use_pit, wiring the PIT provider; _load_siamchart_snapshot loads the SET50 fundamental snapshot. Empty factor store fail-closes (live probe: leakage_guard=false).

Verified

  • Backend full suite 255 passed (was 239); new tests 14 (factor store 9 + pit scorer 5) + 2 backtest leakage-gating + 1 corrected.
  • compileall, git diff --check, static secret/dangerous scan all clean.
  • Live route probe: use_pit with empty store → HTTP 200, leakage_guard=false, real 2-rebalance run.

Honest scope / known limitations

  • Theme dimension is PIT from this store forward; no factor history exists before 2026-08-27, so any pre-today backtest remains non-PIT by construction.
  • Siamchart fundamental is partial: EPS 5-year growth is PIT-grade, but ratios (Yield %, PE, P/BV, ROE, DPS) are current-snapshot only and the snapshot has no dated vintage chain yet. Results are flagged partial_pit, never fully-PIT.
  • dividend_income still uses the final-holdings yield proxy (not a dated dividend ledger); that is the next milestone.

Exact next action

Implement the dated dividend cash-flow ledger (ex-date / record date / per-share / qty held) replacing the final-holdings proxy. After that (or in parallel), add a siamchart vintage chain so the fundamental dimension stops being partial. Do not reset or stage unrelated pre-existing working-tree changes.

Session 2026-08-25 — Multi-theme + Simulation (appended)

Branch: main · HEAD: aae1d13 (pushed 8a6991b..aae1d13)

Completed this session (verified + pushed)

  • 3 Thai alternative-factor themes: tourism (BOT), auto_credit (TradingEconomics Thai car sales), refining_energy (Thai Oil TOP quarterly) — all real Thai data, daily cache.
  • Theme registry + scoring 60/40 (theme / Siamchart fundamental); GET /api/v1/themes returns combined board + per-theme frequency (monthly/quarterly).
  • Capital-allocation simulation: POST /api/v1/simulation — 50/20/30 buckets, min 100 shares, bucket3 = highest dividend yield ignoring score (fix aae1d13), cash fallback, paper/backtest non-PIT label.
  • MT5 bridge interface (app/mt5_bridge.py): dry-run default; live dispatch needs MT5_SEND_ORDERS=1 AND approval (Windows-only MetaTrader5).
  • Thai frontend: theme panel, stock board (Siamchart fundamentals + signal, dividend filter, sortable), simulation tab (capital input → 3 buckets), removed legacy Paper ledger section.
  • Dashboards fully localized to Thai (technical terms kept EN per user).

Verified

  • Backend suite 185 tests OK; compileall ok; static secret scan clean; git diff --check clean.
  • Independent reviews: deleg_1682c760 (blocker: bucket3 ranked by score) → fixed → deleg_43b165e0 passed=true.
  • Live: /api/v1/themes 200; /api/v1/simulation 200 (1M → b1 AOT/MINT/AWC, b3 CRC).

Blockers / pending

  • Price universe = 8 symbols (AOT,AWC,BEM,CPALL,CPN,CRC,MINT,PTT,SET50) from Yahoo snapshot; simulation can only allocate within these. Next action: extend price collector to full SET50.
  • bot_regional only north enabled (NE=955 is 0.0 BE-2570 placeholder); other regions pending.
  • M2.9 legacy files (prices.py, research.py, event_study.py, vintages.py, etc.) still uncommitted/dirty — unrelated, untouched this session.

Exact next action

Extend price-history collection to full SET50 universe so simulation allocates across all names (currently 8). Follow engineering-handoff at session end.

Reference in New Issue View Git Blame Copy Permalink