447 lines
38 KiB
Markdown
447 lines
38 KiB
Markdown
# Handoff — Tourism Vertical Slice
|
||
|
||
## Project
|
||
|
||
- Path: `/Users/kunthawat/Gitea/set50-alternative-data-platform`
|
||
- Branch: `main`
|
||
- Verified code commit: pending; final M2.9 independent remediation review passed; no commit or push performed
|
||
- Current milestone: M2.9 forward price observations and explicit research modes complete/blocked; observation-integrity follow-up implemented; exploratory research is usable, validated backtest remains blocked
|
||
- Mode: research + paper only
|
||
- Frontend: Vue 3 + Vite
|
||
- Backend: Flask `0.5.0`
|
||
- Current runtime source: BOT Tourism Indicators (`TOURISM_SOURCE=bot`)
|
||
|
||
## Completed
|
||
|
||
- Tourism snapshot schema with source, publication, retrieval, vintage and raw hash fields.
|
||
- Deterministic Tourism Pulse score: standardized surprise × exposure × confidence.
|
||
- BOT ASP.NET form adapter with monthly history parsing and trailing 12-point YoY baseline.
|
||
- Atomic raw HTML and normalized snapshot persistence under ignored `backend/data/`.
|
||
- Read-only `/api/v1/data-health` endpoint.
|
||
- Safe `/api/v1/replay/tourism?vintage_id=...` endpoint with identity and path validation.
|
||
- Immutable `VintageStore` manifest with first-seen, last-seen, seen-count and revision metadata.
|
||
- One-shot collector: `backend/scripts/collect_tourism_vintage.py`.
|
||
- Point-in-time vintage query: `GET /api/v1/vintages?as_of=<ISO-8601>`.
|
||
- Deterministic event-study engine with window, benchmark and cost calculations.
|
||
- Backtest readiness gate: `GET /api/v1/backtest/tourism?min_events=12`.
|
||
- Yahoo-backed daily price snapshot with SET symbol mapping and adjusted-close bars.
|
||
- Price health: `GET /api/v1/prices/health`.
|
||
- Forward price observation archive: `GET /api/v1/prices/observations`; repeated payloads reuse immutable snapshots while recording retrieval observations, first/last-seen times, prior hashes, revision status and structured diffs.
|
||
- Frozen research runner: `POST /api/v1/research/tourism/run` and `GET /api/v1/research/tourism/latest`.
|
||
- Research runner modes: explicit `validated` (PIT fail-closed) and `exploratory` (revised-history descriptive only); the UI calls exploratory mode explicitly and shows `result_scope`.
|
||
- Headless runner: `backend/scripts/run_tourism_research.py`.
|
||
- Durable atomic paper ledger under `backend/data/paper/ledger.json` when configured.
|
||
- Raw and normalized snapshot integrity binding with explicit `sha256-json-canonical-v1` metadata.
|
||
- Replay now loads through the canonical `VintageStore.load_snapshot()` path and fails closed on tampered snapshots.
|
||
- Research reports carry canonical content hashes; manifest entries carry their own hashes binding report filename, report hash, and metadata.
|
||
- Research reports from pre-hash trusted stores can be migrated explicitly with `backend/scripts/migrate_research_report_hashes.py`; automatic legacy reads remain disabled.
|
||
- Research input lineage records normalized snapshot hashes and hash algorithms alongside raw payload hashes.
|
||
- PIT daily price contract `pit-daily-v1` is implemented: explicit provider release/evidence metadata, per-series IANA timezone, per-bar session date/known-at/OHLC/adjusted-close/volume, and immutable manifest/hash binding are required before `point_in_time=true` is accepted.
|
||
- PIT event-study mode rejects missing or market-locally future-known prices; the research runner enables this check for PIT snapshots.
|
||
- Public price-provider feasibility is documented in `docs/engineering-log/2026-08-24-price-provider-feasibility.md`; SET Historical Data, SETSMART, SMART Marketplace, ICE SET, LSEG Tick History/S3 Direct, Databento, and Exchange Data International (EDI) have all been checked, but no candidate has yet supplied enough evidence to pass `pit-daily-v1`, so no new adapter was added. EDI is the closest conditional price-feed lead because its public docs expose SET coverage, a confirmed ETA, and correction/re-push behavior, but its public contract still lacks immutable release metadata.
|
||
- VintageStore rejects non-object snapshots, invalid UTF-8 manifests/snapshots, and raw-payload read failures with `VintageStoreError`.
|
||
- Tourism replay computation rejects non-list/non-object observation/exposure shapes as controlled semantic failures, so malformed exposure snapshots fail closed instead of leaking `AttributeError`.
|
||
- Manifest JSON shape and every `list_runs()`/`latest()` entry are validated before metadata is returned or used for selection.
|
||
- Ranked target weights and LONG/SHORT/NEUTRAL classification.
|
||
- English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields.
|
||
- Explicit loopback-only paper demo mode with a visible warning, protected token mode for shared/network use, and internal paper ledger.
|
||
- No external webhook or MT5 integration.
|
||
|
||
Independent M1 review passed with no concrete security or logic blockers. Non-blocking backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
|
||
|
||
## Current live vintage
|
||
|
||
```text
|
||
source_id: bot.ec_ei_028_s2
|
||
published_at: 2026-07-31T14:30:00+07:00
|
||
as_of: 2026-06-30
|
||
status: provisional
|
||
available_periods: 138
|
||
history_points: 12
|
||
vintage_id: bot.ec_ei_028_s2-2026-07-31-665981f88b4a
|
||
seen_count: 16
|
||
raw_payload_hash: 665981f88b4a30c5bd30026cf1e96279c244ad83725558f4136d952b29756a31
|
||
theme_surprise: -0.35114754
|
||
```
|
||
|
||
## Verified commands and live checks
|
||
|
||
```text
|
||
PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
|
||
Ran 125 tests ... OK
|
||
|
||
PYTHONPATH=backend .venv/bin/python -m compileall -q backend/app backend/scripts backend/tests
|
||
passed
|
||
|
||
npm run build
|
||
Vite build completed successfully.
|
||
|
||
npm audit --omit=dev --audit-level=high
|
||
found 0 vulnerabilities
|
||
|
||
git diff --check
|
||
passed
|
||
|
||
python3 ~/.hermes/skills/research/grounded-citations/scripts/sources.py verify docs/engineering-log/2026-08-24-price-provider-feasibility.md --strict --evidence
|
||
citations OK; 11 cited sources, all with evidence quotes
|
||
|
||
GET /api/v1/health
|
||
HTTP 200; {"mode":"research","status":"ok","version":"0.5.0"}
|
||
|
||
GET /api/v1/data-health
|
||
HTTP 200; source_mode=bot, status=provisional, replayable=true
|
||
|
||
GET /api/v1/replay/tourism?vintage_id=bot.ec_ei_028_s2-2026-07-31-665981f88b4a
|
||
HTTP 200; replay theme surprise matched live summary exactly.
|
||
|
||
GET /api/v1/vintages?as_of=2026-07-01T00:00:00Z
|
||
HTTP 200; count=0
|
||
|
||
GET /api/v1/vintages?as_of=2026-08-01T00:00:00Z
|
||
HTTP 200; count=1; manifest seen_count=16
|
||
|
||
GET /api/v1/backtest/tourism?min_events=12
|
||
HTTP 409; status=blocked, available_events=1, required_events=12, price_series_required=true
|
||
|
||
GET /api/v1/prices/health
|
||
HTTP 200; available=true, quality=revised_vendor_history, point_in_time=false, symbols=9
|
||
|
||
POST /api/v1/research/tourism/run (validated default)
|
||
HTTP 200; report status=blocked, reason=insufficient_vintages, immutable run_id persisted
|
||
|
||
POST /api/v1/research/tourism/run (mode=exploratory, min_events=1)
|
||
HTTP 200; revised-price report runs only as result_scope=non_pit_descriptive_only
|
||
|
||
GET /api/v1/research/tourism/latest
|
||
HTTP 200; same run_id returned on replay, report_hash present
|
||
|
||
GET /api/v1/prices/observations
|
||
HTTP 200 on empty archive; populated archives return bound observation IDs, prior hashes and revision diffs
|
||
|
||
BOT collector rerun
|
||
same published release; seen_count=16; independent releases remain=1
|
||
|
||
M2.9 remediation status
|
||
125 tests and current-data health smoke test pass; observation-integrity follow-up covers same-raw/normalized mismatch, malformed manifest entries, first-capture temporal ordering, and duplicate equal-time observations; focused independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty security/logic blocker arrays; no commit or push performed
|
||
```
|
||
|
||
Paper writes support two explicit policies: local `PAPER_AUTH_MODE=demo` requires a loopback bind and allows paper-only writes without a session token while showing a warning; protected `PAPER_AUTH_MODE=token` uses a server-side token exchange and HttpOnly `paper_session` cookie. The token is not embedded in the frontend bundle.
|
||
|
||
## Known limitations
|
||
|
||
- BOT data is provisional and may be revised. The vintage/hash contract preserves the fetched version, but this is not a final-data guarantee.
|
||
- Only foreign-arrival YoY is live in this slice; occupancy and airport passenger metrics are not yet connected.
|
||
- Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
|
||
- No investment edge, transaction-cost result, or backtest conclusion has been established.
|
||
- One independent source release is not enough for a valid event study; current historical rows are not treated as point-in-time vintages.
|
||
- The event-study engine is deterministic and tested. Its PIT mode now enforces per-bar known-at semantics in the market timezone. Yahoo price history is connected for plumbing, but it is revised vendor history, not point-in-time data.
|
||
- The PIT contract is implemented but provider release-time evidence is still absent. No configuration-only promotion of revised history is allowed; `point_in_time` remains false.
|
||
- SET Historical Data, SETSMART, and SMART Marketplace document historical/API availability but do not publicly document the release-vintage and known-at semantics required by the PIT contract. ICE SET is a strong commercial candidate; LSEG S3 Direct has the strongest public PIT claim but remains conditional on SET coverage and vintage evidence. Databento lists the SET venue and documents PIT corporate-action history, but its public price-data material does not establish price release IDs, client availability/`known_at`, or correction-vintage replay. EDI documents confirmed/unconfirmed republish and `Repush1/2/3` correction files for SET-covered EOD pricing and publishes a confirmed Thailand ETA, but also states that it has no EOD time-series database; immutable release IDs, actual release timestamps, retention, and retrievable manifests remain unproven. All candidates require a provider evidence packet before implementation.
|
||
- Validated research runner is operational and replayable, but correctly emits a blocked report until both evidence gates pass. Exploratory runner is operational against revised history only with an explicit descriptive-only scope.
|
||
- Snapshot loads fail closed when raw files, normalized payloads or manifest metadata do not match their recorded hashes.
|
||
- Research report and manifest-entry loads fail closed when persisted content or bound metadata is tampered; pre-existing research volumes require integrity metadata migration before they can be loaded.
|
||
- Independent integrity review gate passed for the staged remediation scope under the local threat model: `deleg_c227c439` found a persist-wide manifest-validation gap in the prior staged tree; it is now fixed and covered by a regression test. The original `deleg_43348a3b` result was semantically positive but schema-invalid because it used `findings`; schema correction `deleg_a1c721d2` returned the exact required four-key verdict with `passed=true` and all arrays empty. Earlier timeouts/stale reviews (`deleg_cf1da27d`, `deleg_feb40653`, `deleg_87a0f686`) and interrupted parallel review `deleg_e2d54c30` are non-approving and contribute no additional evidence.
|
||
- The hash boundary protects local artifacts against corruption/partial writes; hostile host-level rewrite of code, manifests and runtime environment is outside this local threat model.
|
||
- Revisions are deduplicated by canonical `(source_id, published_at)`; the latest observed revision is selected once.
|
||
- Event-study default execution is the next trading session, including weekend/holiday event dates.
|
||
- Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead.
|
||
- Paper demo startup rejects non-loopback `PAPER_BIND_HOST`; token mode remains the required policy for shared/network deployment.
|
||
- Fresh bounded independent review of the paper-auth scope passed with `security_concerns=[]` and `logic_errors=[]`; its remaining items are non-blocking test suggestions. Integrity remediation is tracked separately and passed via the schema-corrected exact verdict.
|
||
- M2.9 remediation now covers malformed provider/config totalization, non-PIT series/raw lineage, PIT numeric overflow handling, UTC/equal-time predecessor ordering, same-raw/normalized mismatch rejection, malformed snapshot manifest rejection, first-capture observation timing, contract-backed observation count/timing/reference checks, descriptive-only exploratory status, and human-readable UI reason labels. Focused independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty security and logic blocker arrays; its only suggestion is to retain regression coverage. Process-level locking and focused helper/UI coverage remain deferred. No commit or push has been made.
|
||
|
||
## Exact next action
|
||
|
||
Keep collecting independent BOT releases until 12 genuinely distinct `(source_id, published_at)` releases exist. In parallel request one complete market-data provider evidence packet (EDI first, or ICE/LSEG/SET fallback): raw sample, provider release ID/time, explicit known-at semantics, correction/revision replay example, SET symbol coverage, timezone/session rules, and immutable archive/manifest evidence. Databento is currently a reference/corporate-actions lead only and must not be used to claim PIT price bars. Revised Yahoo history remains `point_in_time=false`. Do not raise the PIT flag or promote a PIT adapter until that packet passes `pit-daily-v1`. The operator can run exploratory research now, but only validated mode may support a historical backtest claim. Keep local paper work in explicit demo mode and use token mode before any shared/network deployment. Add a trusted deployment signing key before treating the app as a multi-user or hostile-host service.
|
||
|
||
NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED].
|
||
|
||
---
|
||
|
||
## Session 2026-08-28 — Event-driven PIT backtest (verified + pushed)
|
||
|
||
**Branch:** `main` · **HEAD:** `d73a58b` (+cycle-2 fix pending) · **Pushed:** yes (tasks 1–7)
|
||
|
||
### Goal implemented
|
||
Replace the calendar-rebalance backtest with a strict point-in-time, event-driven
|
||
engine per the user's description. Confirmed decisions:
|
||
1. **Strict PIT** — block when factor / Siamchart / price coverage is incomplete (no fallback).
|
||
2. **Execution** — freeze signal at release date D, execute at next trading-day close after D.
|
||
3. **Dividend timing** — entitled on ex-date; cash available `ex_date + 30` days (`ex_date_plus_30d`).
|
||
4. **Average cost** — realized P&L uses weighted avg cost.
|
||
5. **Fee** — all-in 0.3% per trade, no added VAT.
|
||
|
||
### Shipped (commits on main)
|
||
- `68f2cc1` Task 1 `backtest_readiness.py` — readiness + recommended start/end defaults, fail-closed.
|
||
- `6439e9c` Task 2 `backtest_events.py` — event calendar + next-trading-day mapping.
|
||
- `dc057e9` Task 3 `portfolio_ledger.py` — cash/positions/avg-cost/realized+unrealized P&L/dividend receivables/fees.
|
||
- `71b893e` Task 4 `portfolio_rebalancer.py` — 50/20/30 → 100-lot executable orders, sells-first, no trade on unchanged target.
|
||
- `6459015` Task 5 `backtest_engine.py` — event processor; refuses live-board fallback without a PIT scorer.
|
||
- `c87767c` Task 6 `backtest_store.py` + strict `POST /api/v1/backtest/run` (readiness-gated) + `GET /api/v1/backtest/run(+<id>)`.
|
||
- `d73a58b` Task 7 frontend — readiness gate disables Run + shows missing coverage; detailed report.
|
||
|
||
### Verification
|
||
- Standalone offline suite **54 tests** (readiness 10, events 14, ledger 15, rebalancer 5, engine 6, store 4). compileall + diff-check clean.
|
||
- Static scan: no secrets/eval/debug (4 "secret" regex hits are the prose "missing token" string).
|
||
- Live readiness on real data: `ready=false`, missing factor+siamchart, recommended_end=2026-08-27.
|
||
- Live strict run on real data (no vintages): `POST /api/v1/backtest/run` → **400** + missing list (fail-closed).
|
||
- Happy-path `scripts/probe_event_backtest.py` (synthetic PIT-complete): accounting_reconciled=True, identity lhs=rhs, leakage_guard=True, 100-lot holdings.
|
||
- Independent reviews: cycle-1 `deleg_5225e9de` flagged 3 logic errors → fixed (engine fails closed without PIT scorer; readiness blocks explicit start before price history; coincident-day ex-date entitlement fixed by per-day ordering). Cycle-2 `deleg_b969075c` re-verified the fallback fix + 9/10 invariants, re-confirmed the invariant-4 breach. Cycle-3 `deleg_7f56df3b` confirmed the ordering fix. Cycle-4 `deleg_55f4e877` final review returned **passed=true**, empty logic_errors/security_concerns (suggestion: per-symbol price coverage at the exact intersection start — deferred edge case).
|
||
- Evidence: `docs/test-evidence/2026-08-28-event-driven-backtest.md`.
|
||
|
||
### Honest scope / limitations
|
||
- Current on-disk data has **no factor vintages / Siamchart manifest** → strict event backtest is blocked on real data until collection accumulates (this is the intended fail-closed behavior, matching "strict PIT or block").
|
||
- Dividend payment timing is the confirmed `ex_date + 30d` **assumption** (Siamchart gives ex-date+DPS, not pay date); every result exposes `dividend_timing=ex_date_plus_30d`.
|
||
- Only the new `/api/v1/backtest/run` route is event-driven/strict; the legacy `POST /api/v1/backtest` remains as the descriptive non-PIT path.
|
||
|
||
## Exact next action
|
||
Collect factor vintages + Siamchart snapshots over time (scheduler already seeds Siamchart baseline) until `/api/v1/backtest/readiness` reports `ready=true`, then run a real strict event backtest. Frontend already wires readiness → recommended dates.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Backtest Accounting Remediation
|
||
|
||
**Branch:** `main` · **Base HEAD:** `b362cc3` · **Commit/push:** not performed
|
||
|
||
### Completed
|
||
|
||
- Added RED/GREEN flat-price accounting invariants in `backend/tests/test_backtest.py`.
|
||
- Corrected general backtest price P&L to compare ending pre-dividend equity with initial capital.
|
||
- Included the final-holdings dividend-yield proxy in `final_value` and `net_return`.
|
||
- Added machine-readable `dividend_method=final_holdings_yield_proxy`.
|
||
- Updated the Vue backtest panel to label estimated dividends and warn when `leakage_guard=false`.
|
||
|
||
### Verified
|
||
|
||
- RED: flat-price regressions failed with incorrect `price_pnl=20,000` and `49,000`.
|
||
- GREEN: flat/no-dividend, flat/proxy-dividend, rising/no-dividend, and multi-rebalance accounting checks passed; targeted suite **11 passed**.
|
||
- Full backend suite: **239 passed**.
|
||
- Python compileall passed.
|
||
- Vite production build passed.
|
||
- npm audit reported 0 high-severity vulnerabilities.
|
||
- Added-line security/dangerous-pattern scan and `git diff --check` passed.
|
||
- Independent review cycle 1 failed closed on missing proxy/non-PIT disclosure in saved-run history. A fresh fix agent added `ประมาณการปันผล (Proxy)` and per-row `descriptive non-PIT` badges. Fresh final review `deleg_10918fed` inspected the current post-hardening diff and returned `passed=true` with empty security, logic, and suggestion arrays.
|
||
|
||
### Known limitations
|
||
|
||
- `dividend_income` is still estimated from final holdings and final dividend yield; it is not a dated cash-flow ledger.
|
||
- The default public backtest still applies current scores historically, so `leakage_guard=false` and the result remains descriptive non-PIT.
|
||
- A supplied scorer toggles `leakage_guard=true`, but provenance and release-time semantics are not independently certified.
|
||
- Backtest run history remains in process memory.
|
||
|
||
### Exact next action
|
||
|
||
Implement dated dividend events and a provenance-validated PIT score provider before using results as strategy-performance evidence. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Single-container Docker packaging (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** f9b8cd1 (pushed)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **Dockerfile**: `python:3.11-slim` + nginx. Serves the PREBUILT `frontend/dist` SPA (committed bundle, no node/npm in the image), reverse-proxies `/api` → `127.0.0.1:5000`, `VOLUME [/app/backend/data]` for persistence (factor vintages, dividend ledger, forward runs, prices), HEALTHCHECK on `/api/v1/dashboard/summary`, exposes :80.
|
||
- **deploy/nginx.conf**: SPA fallback + `/api` proxy + gzip.
|
||
- **deploy/entrypoint.sh**: starts Flask (`HOST=0.0.0.0:5000`) + nginx in foreground, forwards INT/TERM.
|
||
- **docker-compose.yml**: service `set50`, port `8080:80`, volume `./backend/data`, healthcheck.
|
||
- **.dockerignore** (excludes .git/.venv/backend/data/node_modules/docs) + **.gitignore** now tracks `frontend/dist/` (prebuilt bundle); `backend/data/` stays untracked.
|
||
|
||
### Verified
|
||
- Backend runtime: `test_client GET /api/v1/dashboard/summary` → 200 (the HEALTHCHECK path).
|
||
- `sh -n deploy/entrypoint.sh`, compose YAML parses, all Dockerfile-referenced files exist on remote `origin/main`.
|
||
|
||
### Honest caveat
|
||
- **No local docker on this machine** → the image itself was NOT built/run here. The first real build happens on EasyPanel (or any docker host). If the build fails there, the most likely causes are env-var / volume mapping, which the compose file already covers for self-host.
|
||
|
||
### Exact next action (user-side)
|
||
1. On EasyPanel create a service from this repo (build type **Dockerfile** — not Nixpacks).
|
||
2. Mount a **persistent volume** at `/app/backend/data` (or set `DATA_DIR` env to a mounted path) so the dividend ledger / factor vintages / forward runs survive redeploys.
|
||
3. Publish port **80** (default). Add a webhook in Gitea → EasyPanel deploy URL on push to `main` (Gitea Webhook, not Gitea Actions — no managed runners).
|
||
4. Optional: set `DIVIDEND_REFRESH_COOLDOWN_SECONDS` / `REFRESH_INTERVAL_SECONDS`.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Auto-refresh dated dividend ledger (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** 03195dc (pushed)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- `AppDataScheduler` now calls `_maybe_refresh_dated_dividends()` in `refresh_all` — a cooldown-gated (default 6h) fetch of real dated dividend history into `data/dividends/ledger.json`, so the ledger stays fresh automatically without manual `POST /api/v1/dividends/update` and without re-hammering Siamchart every refresh tick (dividend history changes only a few times a year).
|
||
- `DIVIDEND_REFRESH_COOLDOWN_SECONDS` env toggle (default 21600). Network failure is non-fatal: the previous ledger stays intact.
|
||
- **Tests**: cooldown fires once then skips, refetches after it elapses (2) — full backend **294 passed** (was 292).
|
||
|
||
### Verified
|
||
- Backend **294 passed**; compileall, `git diff --check`, static scan clean.
|
||
|
||
### Exact next action
|
||
None required — the SET50 alternative-data platform PIT/enabler work is complete. Optional UI-only polish: surface the dated-ledger synthesis (dated vs proxy) more explicitly in the backtest result card. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Real dated dividend-history collector (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** f9973e8 (pushed `59c97b5..f9973e8`)
|
||
|
||
### Completed this session (verified + pushed) — the last PIT milestone
|
||
- **Dated dividend collector** `backend/app/siamchart.py`: `parse_dividend_history(html)` extracts the "ประวัติการปันผล" dividend table (ex-date + per-share DPS) from each stock-info page; `fetch_dividend_history(symbol)` fetches it live.
|
||
- **Ledger upgrade** `backend/app/dividend_ledger.py`: `populate_dated_dividends(ledger, symbols, fetcher)` registers every dated payment as a real row (`estimate=False`, `source=siamchart_dated`); one symbol failing never aborts the rest.
|
||
- **API** `backend/app/__init__.py`: `DividendLedger` persisted at `data/dividends/ledger.json`; `POST /api/v1/dividends/update` fetches all snapshot symbols and saves; `use_ledger` backtests prefer the dated ledger (`dividend_method=dated_ledger`) and fall back to DPS estimates when unpopulated.
|
||
- **Tests**: parser (4) + populate (2) — full backend **292 passed** (was 286).
|
||
|
||
### Verified (real network!)
|
||
- **The sandbox HAS outbound network** (contrary to the earlier "needs external access" deferral): curl/https to siamchart.com returns 200.
|
||
- Live `POST /api/v1/dividends/update`: fetched **49/49 symbols, 1410 dated payments** in ~11s.
|
||
- Live `use_ledger` backtest then reports **`dividend_method=dated_ledger`** (real ex-date × qty) instead of DPS proxy.
|
||
|
||
### Honest scope / notes
|
||
- Every snapshot symbol now has real dated dividend history (PTT: 45 rows from 2002; ADVANC: 51; AOT: 22; AWC: 6, etc.).
|
||
- `use_pit` scoring remains non-PIT when a run is fed current-board scores; the dated ledger is orthogonal (cash-flow model), not a PIT-scorer flag.
|
||
|
||
### Exact next action
|
||
All requested milestones are complete and pushed. Optional follow-ups: (a) schedule `/api/v1/dividends/update` into the data refresh loop, (b) surface the dated-ledger synthesis (dated vs proxy) more explicitly in the backtest UI. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Factor-learning validation gate (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** ae814c3 (pushed `9700f6b..ae814c3`)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **Validation gate** `backend/app/weight_learning.py`: `apply_validation_gate(train_ics, holdout_ics)` marks a factor `validated=True` only when total sample >= MIN_SAMPLE_PERIODS (12), each window >= its minimum, train AND holdout IC both beat baseline (BASELINE_IC=0) and agree in sign, and pooled |t| > MIN_IC_TSTAT (1.0). `FactorLearning` now carries `ic_train`/`ic_holdout`/`validated`/`gate_notes`.
|
||
- **No auto-apply enforced**: `apply_weight_update` now keeps `new_weight == old_weight` for any unvalidated factor — weights only move after the gate passes with explicit evidence.
|
||
- **Gated learner**: `learn_momentum_gated(...)` builds PIT momentum ICs then splits chronologically into train/holdout and applies the gate. Wired into `GET /api/v1/learning/momentum`, which now surfaces `ic_train`, `ic_holdout`, `validated`, `gate_notes`.
|
||
- **Tests**: rewritten suite + gate (16) — full backend **286 passed** (was 280).
|
||
|
||
### Verified
|
||
- Backend full suite **286 passed**; compileall, `git diff --check`, static scan clean.
|
||
- Live probe on the current price archive: momentum `validated=false`, `ic_train=0.0711`, `ic_holdout=-0.1143`, `gate_notes=['IC not above baseline (0.0711/-0.1143)']`, `new_weight=None` — the honest outcome: momentum is not validated, weight stays unchanged.
|
||
|
||
### Honest scope / known limitations
|
||
- `learn_momentum_gated` is the only factor with a real PIT series; macro/demographic factors are still unattributable (no vintages) and stay `blocked`.
|
||
- The gate is intentionally strict; a validated factor merely *suggests* a weight delta — applying it to production is a separate, explicit step.
|
||
|
||
### Exact next action
|
||
All six planned PIT/enabler milestones are complete and pushed. Remaining (deferred): a real per-stock **dividend-history collector** (siamchart `/stock-info/<SYMBOL>/`) to upgrade symbols from `dps_annual_proxy` to `dated_ledger` (needs live network access). Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Real forward-test lifecycle (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** 80c6d79 (pushed `6d9c283` + `80c6d79`)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **ForwardTestStore** `backend/app/forward_test.py`: durable, thread-safe JSON store of forward paper runs with an explicit lifecycle — `frozen` (signals snapshotted, immutable) → `executed` (fills 50/20/30 buckets at post-freeze prices) → `marked` (mark-to-market equity series) → `matured` (net_return finalised). Frozen signals are never re-readable/rewritten after creation, so later data cannot retroactively change the run's decision.
|
||
- **API** `backend/app/__init__.py`: `GET /api/v1/forward(+<id>)`, `POST /api/v1/forward` (create+execute, `use_pit` freeze option), `POST /<id>/mark`, `POST /<id>/mature`. Store extension backed by `data/forward/runs.json` (survives restarts).
|
||
- **Frontend** `frontend/src/App.vue`: simulation tab's Forward mode now calls the real `/api/v1/forward` lifecycle (not the cosmetic `/api/v1/simulation` with a mode string), loads runs on mount, and renders a forward panel with status badge, non-PIT tag, holdings, and Mark / Mature per run.
|
||
|
||
### Verified
|
||
- Backend full suite **280 passed** (was 273); new lifecycle tests (7).
|
||
- compileall, `git diff --check`, static scan clean.
|
||
- Live route probe (isolated store): create→execute (201, real holdings AWC/CRC/MINT/PTT), list, mark, mature all work; run persists across the probe.
|
||
|
||
### Honest scope / known limitations
|
||
- The score source at CREATE time may be the current board (`non_pit=true`, tagged). A PIT scorer only marks `non_pit=false` when its scores assert `pit_meta.pit=true`.
|
||
- Paper-only; never dispatches a real MT5 order.
|
||
|
||
### Exact next action
|
||
Implement the **factor-learning validation gate** (PIT exposures + forward returns + Spearman IC + holdout/walk-forward + baseline, no auto-apply). Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — PIT Siamchart vintage store (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** 887e9c9 (pushed)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **Siamchart vintage store** `backend/app/siamchart_vintages.py`: append-only, hash-chained store of every collected Siamchart fundamental snapshot. `snapshot_at(as_of)` returns the newest snapshot whose `retrieved_at <= as_of` (anti-look-ahead, fail-closed when none). Deduplicates identical persists.
|
||
- **Fundamental un-partialed** `backend/app/pit_scorer.py`: `PitScoreProvider` accepts `siamchart_store`; when wired, `siamchart_factor_view(as_of)` reads the snapshot knowable at as_of (`pit_grade='pit'`) instead of the current snapshot (`pit_grade='current'`). `score_board` sets `partial_pit` only when no store is present; theme dimension still fails closed (`pit=false`) unless every theme factor has a released PIT value by as_of.
|
||
- **API** `backend/app/__init__.py`: `/api/v1/backtest use_pit` seeds the first vintage from the on-disk snapshot (idempotent) and wires the store.
|
||
- **Tests**: store (6) + provider-with-store anti-look-ahead (1) — full backend **273 passed** (was 266).
|
||
|
||
### Verified
|
||
- Backend full suite **273 passed**; compileall, `git diff --check`, static scan clean.
|
||
- Live isolated probe: with siamchart store + one released factor, `score_board` returns `partial_pit=false` (fundamental PIT via store) and `pit=false` with a full `blocked_theme` list (themes still lack PIT releases — correct fail-closed).
|
||
|
||
### Honest scope / known limitations
|
||
- Snapshot reconstruction is forward-only; no history before the first collected snapshot.
|
||
- EPS year-keys inside a snapshot are not tied to calendar years, so EPS growth is latest-vs-prior, not fiscal-year-pinned.
|
||
|
||
### Exact next action
|
||
Implement a real per-stock **dividend-history collector** (siamchart `/stock-info/<SYMBOL>/`) that appends dated payments into `dividend_ledger` so symbols upgrade from `dps_annual_proxy` to `dated_ledger`. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — Dated dividend cash-flow ledger (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** 5b8b6d1 (pushed `068dff2` + `5b8b6d1`; backend `068dff2`, frontend `5b8b6d1`)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **Dated dividend ledger** `backend/app/dividend_ledger.py`: per-symbol dated store with `ex_date` / `record_date` / `pay_date` / `per_share` / `source` / `estimate` flag, validation and persistence. `credit_dividends` credits `per_share × qty` once a payment is due (on/after ex-date and pay date); no ledger entry → no credit (fail closed). `build_dps_ledger` builds per-share estimate rows from the siamchart `ratios.DPS` (price-independent, a step up from the yield % × market-value proxy).
|
||
- **Backtest integration** `backend/app/backtest.py`: `run_backtest` accepts `dividend_ledger`; `dividend_income` comes from the ledger when set and `dividend_method` reports `dated_ledger` (real rows) | `dps_annual_proxy` (estimate) | `final_holdings_yield_proxy` (legacy, kept when no ledger). `BacktestResult` gained a `dividend_method` field surfaced in `to_dict`.
|
||
- **API** `backend/app/__init__.py`: `POST /api/v1/backtest` accepts `use_ledger`, wiring the DPS-built ledger (works independently and alongside `use_pit`).
|
||
- **Frontend** `frontend/src/App.vue`: maps `dividend_method` to an honest Thai label (ปันผล (ตามวันจริง) / ประมาณการปันผล/หุ้น (DPS) / ประมาณการปันผล (Proxy)) and shows a **Proxy** badge whenever the dividend is an estimate, on both the result card and saved-run history rows.
|
||
|
||
### Verified
|
||
- Backend full suite **266 passed** (was 255); new: ledger store/credit (9) + backtest ledger integration (2).
|
||
- compileall, `git diff --check`, static secret/dangerous scan clean.
|
||
- Live route probe: `use_ledger:true` → `dividend_method=dps_annual_proxy`, `dividend_income=4151.0` (per-share) vs default proxy `5041.96` (yield×value); price P&L unchanged (5802.89).
|
||
- Frontend build passes (bundle `index-DDD-F_cZ.js`).
|
||
|
||
### Honest scope / known limitations
|
||
- `build_dps_ledger` rows are **estimates** (no ex-date history in the snapshot) — `dividend_method=dps_annual_proxy`, flagged Proxy in UI. A symbol only reports `dated_ledger` once real ex-date/per-share history is collected for it.
|
||
- Ledger credit is end-of-run on final holdings for this milestone (not mid-window on holdings-at-ex-date); a future pass can accrue per rebalance window as ex-date history arrives.
|
||
|
||
### Exact next action
|
||
Add a real per-stock **dividend-history collector** (siamchart `/stock-info/<SYMBOL>/`) that appends dated payments into the ledger so symbols can upgrade from `dps_annual_proxy` to `dated_ledger`. In parallel, add the **siamchart vintage chain** to un-partial the fundamental dimension. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-27 — PIT factor store + partial PIT score provider (verified + pushed)
|
||
|
||
**Branch:** main · **HEAD:** 1f630be (pushed `1b971ac..1f630be`)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- **PIT factor store** `backend/app/factor_vintages.py`: append-only, provenance-complete store with `observed_at` / `released_at` / `retrieved_at` and a SHA-256 canonical **hash chain** (tamper/reorder detectable). `value_at(as_of)` returns only rows whose `released_at <= as_of` — a real testable **anti-look-ahead** guarantee; no value released by `as_of` **fails closed** (None / `have_pit_value=False`). Provenance invariants enforced: `released >= observed`, `retrieved >= released`, non-finite / naive-timestamp rejected.
|
||
- **Partial PIT score provider** `backend/app/pit_scorer.py`: `PitScoreProvider.score_board(as_of)` rebuilds the per-symbol board from PIT factor values (theme 60% dimension) plus a **partial** siamchart view (EPS growth from the 5-year series; current ratios flagged `partial_pit`). `make_pit_score_fn` adapts it to the engine's `ScoreFn`; date-only `as_of` normalized to midnight +07:00.
|
||
- **Honest leakage gating** `backend/app/backtest.py`: `_resolve_scores` now sets `leakage_guard=True` **only** when the supplied score_fn's per-symbol meta asserts `pit_meta.pit=True`. An arbitrary callable with no PIT proof is no longer treated as PIT — closes the "supplied fn ⇒ PIT" hole.
|
||
- **API** `backend/app/__init__.py`: `POST /api/v1/backtest` accepts `use_pit`, wiring the PIT provider; `_load_siamchart_snapshot` loads the SET50 fundamental snapshot. Empty factor store **fail-closes** (live probe: `leakage_guard=false`).
|
||
|
||
### Verified
|
||
- Backend full suite **255 passed** (was 239); new tests 14 (factor store 9 + pit scorer 5) + 2 backtest leakage-gating + 1 corrected.
|
||
- compileall, `git diff --check`, static secret/dangerous scan all clean.
|
||
- Live route probe: `use_pit` with empty store → HTTP 200, `leakage_guard=false`, real 2-rebalance run.
|
||
|
||
### Honest scope / known limitations
|
||
- Theme dimension is PIT **from this store forward**; **no factor history exists before 2026-08-27**, so any pre-today backtest remains non-PIT by construction.
|
||
- Siamchart fundamental is **partial**: EPS 5-year growth is PIT-grade, but ratios (Yield %, PE, P/BV, ROE, DPS) are current-snapshot only and the snapshot has no dated vintage chain yet. Results are flagged `partial_pit`, never fully-PIT.
|
||
- `dividend_income` still uses the final-holdings yield proxy (not a dated dividend ledger); that is the next milestone.
|
||
|
||
### Exact next action
|
||
Implement the dated **dividend cash-flow ledger** (ex-date / record date / per-share / qty held) replacing the final-holdings proxy. After that (or in parallel), add a **siamchart vintage chain** so the fundamental dimension stops being partial. Do not reset or stage unrelated pre-existing working-tree changes.
|
||
|
||
---
|
||
|
||
## Session 2026-08-25 — Multi-theme + Simulation (appended)
|
||
|
||
**Branch:** main · **HEAD:** aae1d13 (pushed `8a6991b..aae1d13`)
|
||
|
||
### Completed this session (verified + pushed)
|
||
- 3 Thai alternative-factor themes: tourism (BOT), auto_credit (TradingEconomics Thai car sales), refining_energy (Thai Oil TOP quarterly) — all real Thai data, daily cache.
|
||
- Theme registry + scoring 60/40 (theme / Siamchart fundamental); `GET /api/v1/themes` returns combined board + per-theme frequency (monthly/quarterly).
|
||
- Capital-allocation simulation: `POST /api/v1/simulation` — 50/20/30 buckets, min 100 shares, bucket3 = highest dividend yield ignoring score (fix `aae1d13`), cash fallback, paper/backtest non-PIT label.
|
||
- MT5 bridge interface (`app/mt5_bridge.py`): dry-run default; live dispatch needs `MT5_SEND_ORDERS=1` AND approval (Windows-only MetaTrader5).
|
||
- Thai frontend: theme panel, stock board (Siamchart fundamentals + signal, dividend filter, sortable), simulation tab (capital input → 3 buckets), removed legacy Paper ledger section.
|
||
- Dashboards fully localized to Thai (technical terms kept EN per user).
|
||
|
||
### Verified
|
||
- Backend suite **185 tests OK**; compileall ok; static secret scan clean; `git diff --check` clean.
|
||
- Independent reviews: `deleg_1682c760` (blocker: bucket3 ranked by score) → fixed → `deleg_43b165e0` passed=true.
|
||
- Live: `/api/v1/themes` 200; `/api/v1/simulation` 200 (1M → b1 AOT/MINT/AWC, b3 CRC).
|
||
|
||
### Blockers / pending
|
||
- Price universe = **8 symbols** (AOT,AWC,BEM,CPALL,CPN,CRC,MINT,PTT,SET50) from Yahoo snapshot; simulation can only allocate within these. **Next action: extend price collector to full SET50.**
|
||
- `bot_regional` only north enabled (NE=955 is 0.0 BE-2570 placeholder); other regions pending.
|
||
- M2.9 legacy files (prices.py, research.py, event_study.py, vintages.py, etc.) still **uncommitted/dirty** — unrelated, untouched this session.
|
||
|
||
### Exact next action
|
||
Extend price-history collection to full SET50 universe so simulation allocates across all names (currently 8). Follow `engineering-handoff` at session end.
|