diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index ebbf6db..cf8e044 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -4,8 +4,8 @@ - Path: `/Users/kunthawat/Gitea/set50-alternative-data-platform` - Branch: `main` -- Verified code commit: `ba9114d` — `[verified] bind snapshots to manifests and raw hashes` -- Current milestone: M2.5 integrity hardening complete; validated backtest blocked +- Verified code commit: pending; final M2.9 independent remediation review passed; no commit or push performed +- Current milestone: M2.9 forward price observations and explicit research modes complete/blocked; observation-integrity follow-up implemented; exploratory research is usable, validated backtest remains blocked - Mode: research + paper only - Frontend: Vue 3 + Vite - Backend: Flask `0.5.0` @@ -26,13 +26,25 @@ - Backtest readiness gate: `GET /api/v1/backtest/tourism?min_events=12`. - Yahoo-backed daily price snapshot with SET symbol mapping and adjusted-close bars. - Price health: `GET /api/v1/prices/health`. +- Forward price observation archive: `GET /api/v1/prices/observations`; repeated payloads reuse immutable snapshots while recording retrieval observations, first/last-seen times, prior hashes, revision status and structured diffs. - Frozen research runner: `POST /api/v1/research/tourism/run` and `GET /api/v1/research/tourism/latest`. +- Research runner modes: explicit `validated` (PIT fail-closed) and `exploratory` (revised-history descriptive only); the UI calls exploratory mode explicitly and shows `result_scope`. - Headless runner: `backend/scripts/run_tourism_research.py`. - Durable atomic paper ledger under `backend/data/paper/ledger.json` when configured. - Raw and normalized snapshot integrity binding with explicit `sha256-json-canonical-v1` metadata. +- Replay now loads through the canonical `VintageStore.load_snapshot()` path and fails closed on tampered snapshots. +- Research reports carry canonical content hashes; manifest entries carry their own hashes binding report filename, report hash, and metadata. +- Research reports from pre-hash trusted stores can be migrated explicitly with `backend/scripts/migrate_research_report_hashes.py`; automatic legacy reads remain disabled. +- Research input lineage records normalized snapshot hashes and hash algorithms alongside raw payload hashes. +- PIT daily price contract `pit-daily-v1` is implemented: explicit provider release/evidence metadata, per-series IANA timezone, per-bar session date/known-at/OHLC/adjusted-close/volume, and immutable manifest/hash binding are required before `point_in_time=true` is accepted. +- PIT event-study mode rejects missing or market-locally future-known prices; the research runner enables this check for PIT snapshots. +- Public price-provider feasibility is documented in `docs/engineering-log/2026-08-24-price-provider-feasibility.md`; SET Historical Data, SETSMART, SMART Marketplace, ICE SET, LSEG Tick History/S3 Direct, Databento, and Exchange Data International (EDI) have all been checked, but no candidate has yet supplied enough evidence to pass `pit-daily-v1`, so no new adapter was added. EDI is the closest conditional price-feed lead because its public docs expose SET coverage, a confirmed ETA, and correction/re-push behavior, but its public contract still lacks immutable release metadata. +- VintageStore rejects non-object snapshots, invalid UTF-8 manifests/snapshots, and raw-payload read failures with `VintageStoreError`. +- Tourism replay computation rejects non-list/non-object observation/exposure shapes as controlled semantic failures, so malformed exposure snapshots fail closed instead of leaking `AttributeError`. +- Manifest JSON shape and every `list_runs()`/`latest()` entry are validated before metadata is returned or used for selection. - Ranked target weights and LONG/SHORT/NEUTRAL classification. - English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields. -- HttpOnly paper session and internal paper ledger. +- Explicit loopback-only paper demo mode with a visible warning, protected token mode for shared/network use, and internal paper ledger. - No external webhook or MT5 integration. Independent M1 review passed with no concrete security or logic blockers. Non-blocking backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures. @@ -47,6 +59,7 @@ status: provisional available_periods: 138 history_points: 12 vintage_id: bot.ec_ei_028_s2-2026-07-31-665981f88b4a +seen_count: 16 raw_payload_hash: 665981f88b4a30c5bd30026cf1e96279c244ad83725558f4136d952b29756a31 theme_surprise: -0.35114754 ``` @@ -55,11 +68,23 @@ theme_surprise: -0.35114754 ```text PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v -Ran 18 tests ... OK +Ran 125 tests ... OK + +PYTHONPATH=backend .venv/bin/python -m compileall -q backend/app backend/scripts backend/tests +passed npm run build Vite build completed successfully. +npm audit --omit=dev --audit-level=high +found 0 vulnerabilities + +git diff --check +passed + +python3 ~/.hermes/skills/research/grounded-citations/scripts/sources.py verify docs/engineering-log/2026-08-24-price-provider-feasibility.md --strict --evidence +citations OK; 11 cited sources, all with evidence quotes + GET /api/v1/health HTTP 200; {"mode":"research","status":"ok","version":"0.5.0"} @@ -73,7 +98,7 @@ GET /api/v1/vintages?as_of=2026-07-01T00:00:00Z HTTP 200; count=0 GET /api/v1/vintages?as_of=2026-08-01T00:00:00Z -HTTP 200; count=1; manifest seen_count=4 +HTTP 200; count=1; manifest seen_count=16 GET /api/v1/backtest/tourism?min_events=12 HTTP 409; status=blocked, available_events=1, required_events=12, price_series_required=true @@ -81,14 +106,26 @@ HTTP 409; status=blocked, available_events=1, required_events=12, price_series_r GET /api/v1/prices/health HTTP 200; available=true, quality=revised_vendor_history, point_in_time=false, symbols=9 -POST /api/v1/research/tourism/run +POST /api/v1/research/tourism/run (validated default) HTTP 200; report status=blocked, reason=insufficient_vintages, immutable run_id persisted +POST /api/v1/research/tourism/run (mode=exploratory, min_events=1) +HTTP 200; revised-price report runs only as result_scope=non_pit_descriptive_only + GET /api/v1/research/tourism/latest -HTTP 200; same run_id returned on replay +HTTP 200; same run_id returned on replay, report_hash present + +GET /api/v1/prices/observations +HTTP 200 on empty archive; populated archives return bound observation IDs, prior hashes and revision diffs + +BOT collector rerun +same published release; seen_count=16; independent releases remain=1 + +M2.9 remediation status +125 tests and current-data health smoke test pass; observation-integrity follow-up covers same-raw/normalized mismatch, malformed manifest entries, first-capture temporal ordering, and duplicate equal-time observations; focused independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty security/logic blocker arrays; no commit or push performed ``` -Paper writes use a server-side token exchange and HttpOnly `paper_session` cookie; the token is not embedded in the frontend bundle. +Paper writes support two explicit policies: local `PAPER_AUTH_MODE=demo` requires a loopback bind and allows paper-only writes without a session token while showing a warning; protected `PAPER_AUTH_MODE=token` uses a server-side token exchange and HttpOnly `paper_session` cookie. The token is not embedded in the frontend bundle. ## Known limitations @@ -97,14 +134,50 @@ Paper writes use a server-side token exchange and HttpOnly `paper_session` cooki - Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment. - No investment edge, transaction-cost result, or backtest conclusion has been established. - One independent source release is not enough for a valid event study; current historical rows are not treated as point-in-time vintages. -- The event-study engine is deterministic and tested. Yahoo price history is connected for plumbing, but it is revised vendor history, not point-in-time data. -- Research runner is operational and replayable, but correctly emits a blocked report until both evidence gates pass. +- The event-study engine is deterministic and tested. Its PIT mode now enforces per-bar known-at semantics in the market timezone. Yahoo price history is connected for plumbing, but it is revised vendor history, not point-in-time data. +- The PIT contract is implemented but provider release-time evidence is still absent. No configuration-only promotion of revised history is allowed; `point_in_time` remains false. +- SET Historical Data, SETSMART, and SMART Marketplace document historical/API availability but do not publicly document the release-vintage and known-at semantics required by the PIT contract. ICE SET is a strong commercial candidate; LSEG S3 Direct has the strongest public PIT claim but remains conditional on SET coverage and vintage evidence. Databento lists the SET venue and documents PIT corporate-action history, but its public price-data material does not establish price release IDs, client availability/`known_at`, or correction-vintage replay. EDI documents confirmed/unconfirmed republish and `Repush1/2/3` correction files for SET-covered EOD pricing and publishes a confirmed Thailand ETA, but also states that it has no EOD time-series database; immutable release IDs, actual release timestamps, retention, and retrievable manifests remain unproven. All candidates require a provider evidence packet before implementation. +- Validated research runner is operational and replayable, but correctly emits a blocked report until both evidence gates pass. Exploratory runner is operational against revised history only with an explicit descriptive-only scope. - Snapshot loads fail closed when raw files, normalized payloads or manifest metadata do not match their recorded hashes. +- Research report and manifest-entry loads fail closed when persisted content or bound metadata is tampered; pre-existing research volumes require integrity metadata migration before they can be loaded. +- Independent integrity review gate passed for the staged remediation scope under the local threat model: `deleg_c227c439` found a persist-wide manifest-validation gap in the prior staged tree; it is now fixed and covered by a regression test. The original `deleg_43348a3b` result was semantically positive but schema-invalid because it used `findings`; schema correction `deleg_a1c721d2` returned the exact required four-key verdict with `passed=true` and all arrays empty. Earlier timeouts/stale reviews (`deleg_cf1da27d`, `deleg_feb40653`, `deleg_87a0f686`) and interrupted parallel review `deleg_e2d54c30` are non-approving and contribute no additional evidence. - The hash boundary protects local artifacts against corruption/partial writes; hostile host-level rewrite of code, manifests and runtime environment is outside this local threat model. - Revisions are deduplicated by canonical `(source_id, published_at)`; the latest observed revision is selected once. - Event-study default execution is the next trading session, including weekend/holiday event dates. - Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead. +- Paper demo startup rejects non-loopback `PAPER_BIND_HOST`; token mode remains the required policy for shared/network deployment. +- Fresh bounded independent review of the paper-auth scope passed with `security_concerns=[]` and `logic_errors=[]`; its remaining items are non-blocking test suggestions. Integrity remediation is tracked separately and passed via the schema-corrected exact verdict. +- M2.9 remediation now covers malformed provider/config totalization, non-PIT series/raw lineage, PIT numeric overflow handling, UTC/equal-time predecessor ordering, same-raw/normalized mismatch rejection, malformed snapshot manifest rejection, first-capture observation timing, contract-backed observation count/timing/reference checks, descriptive-only exploratory status, and human-readable UI reason labels. Focused independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty security and logic blocker arrays; its only suggestion is to retain regression coverage. Process-level locking and focused helper/UI coverage remain deferred. No commit or push has been made. ## Exact next action -Collect independent BOT releases over time and replace/supplement revised vendor history with a point-in-time daily price source. The operator can run the frozen research check now; only raise the gate when both requirements pass. Add a trusted deployment signing key before treating the app as a multi-user or hostile-host service. +Keep collecting independent BOT releases until 12 genuinely distinct `(source_id, published_at)` releases exist. In parallel request one complete market-data provider evidence packet (EDI first, or ICE/LSEG/SET fallback): raw sample, provider release ID/time, explicit known-at semantics, correction/revision replay example, SET symbol coverage, timezone/session rules, and immutable archive/manifest evidence. Databento is currently a reference/corporate-actions lead only and must not be used to claim PIT price bars. Revised Yahoo history remains `point_in_time=false`. Do not raise the PIT flag or promote a PIT adapter until that packet passes `pit-daily-v1`. The operator can run exploratory research now, but only validated mode may support a historical backtest claim. Keep local paper work in explicit demo mode and use token mode before any shared/network deployment. Add a trusted deployment signing key before treating the app as a multi-user or hostile-host service. + +NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED]. + +--- + +## Session 2026-08-25 — Multi-theme + Simulation (appended) + +**Branch:** main · **HEAD:** aae1d13 (pushed `8a6991b..aae1d13`) + +### Completed this session (verified + pushed) +- 3 Thai alternative-factor themes: tourism (BOT), auto_credit (TradingEconomics Thai car sales), refining_energy (Thai Oil TOP quarterly) — all real Thai data, daily cache. +- Theme registry + scoring 60/40 (theme / Siamchart fundamental); `GET /api/v1/themes` returns combined board + per-theme frequency (monthly/quarterly). +- Capital-allocation simulation: `POST /api/v1/simulation` — 50/20/30 buckets, min 100 shares, bucket3 = highest dividend yield ignoring score (fix `aae1d13`), cash fallback, paper/backtest non-PIT label. +- MT5 bridge interface (`app/mt5_bridge.py`): dry-run default; live dispatch needs `MT5_SEND_ORDERS=1` AND approval (Windows-only MetaTrader5). +- Thai frontend: theme panel, stock board (Siamchart fundamentals + signal, dividend filter, sortable), simulation tab (capital input → 3 buckets), removed legacy Paper ledger section. +- Dashboards fully localized to Thai (technical terms kept EN per user). + +### Verified +- Backend suite **185 tests OK**; compileall ok; static secret scan clean; `git diff --check` clean. +- Independent reviews: `deleg_1682c760` (blocker: bucket3 ranked by score) → fixed → `deleg_43b165e0` passed=true. +- Live: `/api/v1/themes` 200; `/api/v1/simulation` 200 (1M → b1 AOT/MINT/AWC, b3 CRC). + +### Blockers / pending +- Price universe = **8 symbols** (AOT,AWC,BEM,CPALL,CPN,CRC,MINT,PTT,SET50) from Yahoo snapshot; simulation can only allocate within these. **Next action: extend price collector to full SET50.** +- `bot_regional` only north enabled (NE=955 is 0.0 BE-2570 placeholder); other regions pending. +- M2.9 legacy files (prices.py, research.py, event_study.py, vintages.py, etc.) still **uncommitted/dirty** — unrelated, untouched this session. + +### Exact next action +Extend price-history collection to full SET50 universe so simulation allocates across all names (currently 8). Follow `engineering-handoff` at session end. diff --git a/docs/engineering-log/2026-08-25-multitheme-and-simulation.md b/docs/engineering-log/2026-08-25-multitheme-and-simulation.md new file mode 100644 index 0000000..f79f710 --- /dev/null +++ b/docs/engineering-log/2026-08-25-multitheme-and-simulation.md @@ -0,0 +1,55 @@ +# 2026-08-25 — Multi-theme platform + Capital simulation + +## Scope +Took the single-theme tourism demo and built a full multi-theme SET50 research +platform with capital-allocation simulation, all in Thai UI, on real Thai data. + +## Milestones (all verified + pushed) +| # | Commit | Milestone | Status | Last verified | +|---|--------|-----------|--------|---------------| +| 1 | `f6dd630` | Auto Credit (TradingEconomics) + EIA collectors | complete | live fetch | +| 2 | `2fa8055`,`fdfd832` | BOT regional consumption (north; NE = 0.0 placeholder disabled) | complete | live (idx 100.3) | +| 3 | `674dcfd` | Scope: national-only Thai factors; cancel DLT province scraping | complete | docs | +| 4 | `3f7fccd` | **Thai Energy (TOP) collector** replacing EIA US proxy | complete | live (Q2/2026 net 19,481) | +| 5 | `affc29a` | theme registry + scoring engine (60 theme / 40 Siamchart) | complete | tests | +| 6 | `63058ec` | daily cache (fetch-once/day, stale fallback, atomic) | complete | tests | +| 7 | `7643764` | GET /api/v1/themes combined board (3 themes + frequency) | complete | live HTTP 200 | +| 8 | `90e63e3` | frontend Thai theme panel (3 cards + signals) | complete | browser | +| 9 | `2caf814` | Localize dashboard to Thai; fix fixture-badge honesty | complete | browser | +| 10 | `c3a1461` | **capital allocation engine + MT5 bridge** | complete | tests + live | +| 11 | `aae1d13` | **Fix bucket3 rank by dividend yield (ignore score)** | complete | review passed | +| 12 | `e24023c` | Thai Simulation UI; remove legacy Paper ledger section | complete | browser | + +## Key decisions +- Factors must be **Thailand-specific** (auto = Thai car sales, energy = TOP Thai + refinery, NOT US EIA/Gulf crack spread). +- Regional = **national ภาพรวมไทย only** — no province/region-level scraping. +- **Frequency matters**: tourism/auto = monthly, energy = quarterly, Siamchart + = annual. Don't mix different-cadence factors as same-timestamp. +- Simulation buckets (user-confirmed): b1 50% score+dividend · b2 20% score no + dividend · b3 30% **highest dividend yield** (ignore score, exclude b1) · + min 100 shares · cash fallback. +- Paper ledger replaced by simulation (backtest + forward test share one engine; + forward may optionally dispatch MT5, live gated). + +## Verified (evidence) +- Full backend suite: **185 tests OK** +- Live HTTP: `/api/v1/themes` 200 (3 Thai themes), `/api/v1/simulation` 200 + (1,000,000 → buckets; b1 AOT/MINT/AWC, b3 CRC after yield fix) +- Independent reviews: `deleg_1682c760` (found bucket3-by-score blocker) → + fixed → `deleg_43b165e0` **passed=true** (no security/logic errors) +- `git diff --check` clean, static secret scan clean, compileall ok + +## Risks / remaining work +- Price universe = 8 symbols (Yahoo snapshot); full SET50 needs extending the + price collector. +- `bot_regional` only north enabled (NE=955 is a 0.0 BE-2570 placeholder). +- BOT portal API (`portal.api.bot.or.th`) requires auth — JSON alternative future. +- MT5 live dispatch is interface-only (dry-run); needs MT5_SEND_ORDERS=1 + config. +- M2.9 legacy work (prices.py, research.py, event_study.py, vintages.py, etc.) + is still **uncommitted/dirty** — separate, unrelated to this session, untouched. + +## Next action +Extend price-history collection to the full SET50 universe so simulation can +allocate across all names (currently limited to the 8 symbols that have Yahoo +history in `backend/data/prices/snapshots/`).