kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[verified] add event-study readiness gate

Browse Source
This commit is contained in:
Kunthawat Greethong
2026-08-23 12:02:22 +07:00
parent 91eb83631e
commit 0b47a06238
10 changed files with 396 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/HANDOFF.md
View File

@@ -5,10 +5,10 @@
- Path: `/Users/kunthawat/Gitea/set50-alternative-data-platform`
- Branch: `main`
- Verified code commit: `d1ba6ef` — `[verified] add vintage collector and point-in-time API`
- Current milestone: M2 vintage foundation complete; event study deferred
- Current milestone: M2.3 event-study foundation complete; real backtest blocked
- Mode: research + paper only
- Frontend: Vue 3 + Vite
- Backend: Flask `0.3.0`
- Backend: Flask `0.4.0`
- Current runtime source: BOT Tourism Indicators (`TOURISM_SOURCE=bot`)
## Completed
@@ -22,6 +22,8 @@
- Immutable `VintageStore` manifest with first-seen, last-seen, seen-count and revision metadata.
- One-shot collector: `backend/scripts/collect_tourism_vintage.py`.
- Point-in-time vintage query: `GET /api/v1/vintages?as_of=<ISO-8601>`.
- Deterministic event-study engine with window, benchmark and cost calculations.
- Backtest readiness gate: `GET /api/v1/backtest/tourism?min_events=12`.
- Ranked target weights and LONG/SHORT/NEUTRAL classification.
- English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields.
- HttpOnly paper session and internal paper ledger.
@@ -53,7 +55,7 @@ npm run build
Vite build completed successfully.
GET /api/v1/health
HTTP 200; {"mode":"research","status":"ok","version":"0.3.0"}
HTTP 200; {"mode":"research","status":"ok","version":"0.4.0"}
GET /api/v1/data-health
HTTP 200; source_mode=bot, status=provisional, replayable=true
@@ -66,6 +68,9 @@ HTTP 200; count=0
GET /api/v1/vintages?as_of=2026-08-01T00:00:00Z
HTTP 200; count=1; manifest seen_count=4
GET /api/v1/backtest/tourism?min_events=12
HTTP 409; status=blocked, available_events=1, required_events=12, price_series_required=true
```
Paper writes use a server-side token exchange and HttpOnly `paper_session` cookie; the token is not embedded in the frontend bundle.
@@ -77,8 +82,9 @@ Paper writes use a server-side token exchange and HttpOnly `paper_session` cooki
- Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
- No investment edge, transaction-cost result, or backtest conclusion has been established.
- One independent source release is not enough for a valid event study; current historical rows are not treated as point-in-time vintages.
- The event-study engine is deterministic and tested, but no real price provider is connected yet.
- Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead.
## Exact next action
Collect independent BOT releases over time, then implement event-study/backtest checks using only vintages whose `published_at` is known at each test date. Add another metric only when its historical release coverage is real.
Collect independent BOT releases over time and add a point-in-time daily price provider. Only then raise the readiness gate and run the event study. Add another metric only when its historical release coverage is real.

5
docs/engineering-log.md
View File

@@ -7,6 +7,7 @@
| M0 repo foundation | complete | Flask API, Vue/Vite shell | keep research/paper guardrails |
| M1 BOT Tourism adapter | complete | 18 tests, live BOT fetch, raw/snapshot persistence | validate multiple vintages |
| M2 vintage collector | complete | 25 tests, manifest idempotency, live collector and point-in-time API | collect independent releases |
| M2.3 event-study gate | complete/blocked | 30 tests, pure engine and truthful 409 readiness API | add point-in-time price provider |
| Tourism deterministic signal | complete | live foreign-arrivals YoY surprise | add occupancy/airport metric |
| Internal paper ledger | complete | POST/readback through live API | persist in PostgreSQL later |
| Dashboard | complete | Vite build + served source check with live-sign copy | visual browser capture after permission is available |
@@ -24,7 +25,7 @@
## Verification
- Backend: 25 unittest tests pass.
- Backend: 30 unittest tests pass.
- Independent M1 review: **PASSED**; no concrete security or logic blockers.
- Reviewer suggestions: set `PAPER_COOKIE_SECURE=1` outside local HTTP; replace in-memory sessions before multi-worker deployment.
- M1 reviewer backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
@@ -37,4 +38,6 @@
- Live vintage replay returned the same theme surprise as the current dashboard summary.
- Vintage collector preserved one live `vintage_id` with `seen_count=4` and point-in-time API excluded it before `published_at`.
- Independent M2 review: **PASSED**; no concrete security or logic blockers.
- Event-study readiness gate correctly returns HTTP 409 with 1/12 independent vintages; no backtest result is fabricated.
- Independent M2.3 review: **PASSED**; no concrete security or logic blockers.
- Browser visual capture was blocked by Chrome remote-debugging permission; no permission dialog was clicked.

68
docs/engineering-log/2026-08-23-event-study-gate.md Normal file
View File

@@ -0,0 +1,68 @@
# 2026-08-23 — event-study engine and backtest readiness gate
## Plan status
- Deterministic event-study engine: complete.
- Cost-aware window calculations: complete.
- Point-in-time vintage sufficiency gate: complete.
- Market-price provider and real backtest: blocked by design until independent releases and a point-in-time daily price series exist.
## Changed files
- `backend/app/event_study.py` — readiness gate, event-window returns, weighted portfolio returns, benchmark comparison, turnover cost and hit-rate calculations.
- `backend/app/__init__.py` — `/api/v1/backtest/tourism` readiness endpoint and API version `0.4.0`.
- `backend/tests/test_event_study.py` — sufficiency, window/cost and missing-price tests.
- `backend/tests/test_api.py` — blocked and invalid-query endpoint tests.
- `frontend/src/App.vue` — Backtest gate KPI that accepts HTTP 409 as a truthful blocked state instead of treating it as a dashboard failure.
- `README.md` — engine contract and readiness behavior.
## Contract
The engine accepts point-in-time events, daily symbol prices, an optional benchmark, event windows, and cost assumptions. It does not fetch prices, infer missing closes, or convert current revised historical data into past knowledge.
When fewer than the configured minimum events exist, the API returns:
```json
{
"status": "blocked",
"reason": "insufficient_vintages",
"available_events": 1,
"required_events": 12,
"price_series_required": true
}
```
## Live evidence
```text
GET /api/v1/backtest/tourism?min_events=12
HTTP 409
status: blocked
available_events: 1
required_events: 12
next_action: collect independent published vintages before running event study
```
The dashboard served source includes `fetchBacktestReadiness`, `Backtest gate`, and the blocked-vintage count. HTTP 409 is handled as an expected research state, not a page error.
## Verification
- `PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v` — **30 tests passed**.
- `npm run build` — passed.
- Live API readiness gate — passed with truthful 409 blocked result.
- Invalid `min_events` — HTTP 400.
- Pure event-study fixture — window return, benchmark and cost calculation passed.
## Independent review
```text
passed: true
security_concerns: []
logic_errors: []
```
Non-blocking backlog: add API boundary cases for zero/negative `min_events` and multi-vintage benchmark/window gaps.
## Risks and exact next action
There is only one independent BOT release. No investment backtest result exists. Continue collecting releases and add a point-in-time daily price adapter before raising the readiness threshold or interpreting event-study output.

42
docs/test-evidence/2026-08-23-event-study-gate.md Normal file
View File

@@ -0,0 +1,42 @@
# Test evidence — 2026-08-23 event-study gate
## Automated
```text
PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
Ran 30 tests ... OK
npm run build
Vite build completed successfully.
```
## Live readiness
```text
GET /api/v1/backtest/tourism?min_events=12
HTTP 409
status=blocked
available_events=1
required_events=12
price_series_required=true
```
The 409 is intentional: the API refuses to produce an event study with one independent vintage.
## Engine fixture
The pure event-study fixture verified one-session window returns, weighted long/short portfolio aggregation, benchmark comparison, turnover cost deduction, hit rate, and missing-price rejection.
## Frontend served verification
The frontend build and served `/src/App.vue` contained the readiness loader and `Backtest gate` KPI. Browser screenshot capture remains unavailable because the Chrome remote-debugging permission prompt has not been approved.
## Independent review
```text
passed: true
security_concerns: []
logic_errors: []
```
Non-blocking backlog: add API boundary cases for zero/negative `min_events` and multi-vintage benchmark/window gaps.