kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[verified] add BOT tourism source adapter

Browse Source
This commit is contained in:
Kunthawat Greethong
2026-08-23 10:19:40 +07:00
parent 3e978c5948
commit a113a51589
15 changed files with 800 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
docs/HANDOFF.md
View File

@@ -3,45 +3,71 @@
## Project
- Path: `/Users/kunthawat/Gitea/set50-alternative-data-platform`
- Branch: `main`
- Current milestone: M1 complete and independently reviewed
- Mode: research + paper only
- Frontend: Vue 3 + Vite
- Backend: Flask
- Current data: deterministic fixture with lineage metadata
- Backend: Flask `0.2.0`
- Current runtime source: BOT Tourism Indicators (`TOURISM_SOURCE=bot`)
## Completed
- Tourism snapshot schema with source, publication, retrieval and vintage fields.
- Tourism snapshot schema with source, publication, retrieval, vintage and raw hash fields.
- Deterministic Tourism Pulse score: standardized surprise × exposure × confidence.
- BOT ASP.NET form adapter with monthly history parsing and trailing 12-point YoY baseline.
- Atomic raw HTML and normalized snapshot persistence under ignored `backend/data/`.
- Read-only `/api/v1/data-health` endpoint.
- Safe `/api/v1/replay/tourism?vintage_id=...` endpoint with identity and path validation.
- Ranked target weights and LONG/SHORT/NEUTRAL classification.
- Flask endpoints for health, summary, observations, signals and paper ledger.
- English dashboard with live API data, lineage panel, signal table and paper-entry form.
- English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields.
- HttpOnly paper session and internal paper ledger.
- No external webhook or MT5 integration.
## Verified commands
Independent M1 review passed with no concrete security or logic blockers. Non-blocking backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
## Current live vintage
```text
PYTHONPATH=backend .venv/bin/python -m unittest discover -s backend/tests -v
Ran 10 tests ... OK
source_id: bot.ec_ei_028_s2
published_at: 2026-07-31T14:30:00+07:00
as_of: 2026-06-30
status: provisional
available_periods: 138
history_points: 12
vintage_id: bot.ec_ei_028_s2-2026-07-31-665981f88b4a
raw_payload_hash: 665981f88b4a30c5bd30026cf1e96279c244ad83725558f4136d952b29756a31
theme_surprise: -0.35114754
```
Paper writes use a server-side token exchange and HttpOnly `paper_session` cookie; the token is not embedded in the frontend bundle.
## Verified commands and live checks
```text
PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
Ran 18 tests ... OK
npm run build
Vite build completed successfully.
GET /api/v1/health
{"mode":"research","status":"ok","version":"0.1.0"}
HTTP 200; {"mode":"research","status":"ok","version":"0.2.0"}
POST /api/v1/paper/ledger + GET /api/v1/paper/ledger
PAPER_RECORDED and readback verified.
GET /api/v1/data-health
HTTP 200; source_mode=bot, status=provisional, replayable=true
Independent review
PASSED — no concrete security or logic blockers.
GET /api/v1/replay/tourism?vintage_id=bot.ec_ei_028_s2-2026-07-31-665981f88b4a
HTTP 200; replay theme surprise matched live summary exactly.
```
## Known limitation
Paper writes use a server-side token exchange and HttpOnly `paper_session` cookie; the token is not embedded in the frontend bundle.
The data adapter is a fixture. It demonstrates the contract and calculation, not live tourism data quality. Browser visual screenshot verification was blocked by a Chrome remote-debugging permission prompt; served HTML, API response and frontend build were verified instead.
## Known limitations
## Next action
- BOT data is provisional and may be revised. The vintage/hash contract preserves the fetched version, but this is not a final-data guarantee.
- Only foreign-arrival YoY is live in this slice; occupancy and airport passenger metrics are not yet connected.
- Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
- No investment edge, transaction-cost result, or backtest conclusion has been established.
- Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead.
Implement one replayable Tourism source adapter without changing the snapshot contract. Add parser fixture tests, publication timestamp handling, raw snapshot hash, and a data-health failure state before adding LLM analysis.
## Exact next action
Validate multiple BOT vintages and add the next real Tourism metric without changing the snapshot contract. Run event-study/backtest checks before adding LLM analysis.

17
docs/engineering-log.md
View File

@@ -4,10 +4,11 @@
| Milestone | Status | Evidence | Next action |
|---|---|---|---|
| M0 repo foundation | complete | Flask API, Vue/Vite shell | replace fixture with source adapter |
| Tourism deterministic signal | complete | 7 backend tests pass | add replayable real source |
| M0 repo foundation | complete | Flask API, Vue/Vite shell | keep research/paper guardrails |
| M1 BOT Tourism adapter | complete | 18 tests, live BOT fetch, raw/snapshot persistence | validate multiple vintages |
| Tourism deterministic signal | complete | live foreign-arrivals YoY surprise | add occupancy/airport metric |
| Internal paper ledger | complete | POST/readback through live API | persist in PostgreSQL later |
| Dashboard | complete | Vite build + live HTML/API checks | visual browser capture after permission is available |
| Dashboard | complete | Vite build + served source check with live-sign copy | visual browser capture after permission is available |
| LLM analysis | deferred | intentionally no LLM dependency in M0 | add after signal lineage is stable |
| Webhook receiver | deferred | contract only, no external receiver | choose after core app is usable |
| MT5 bridge | deferred | not started | paper bridge after webhook decision |
@@ -17,16 +18,20 @@
- Research and paper modes only.
- No live orders, external webhook receiver, broker credentials, or MT5 connection.
- Deterministic signal is authoritative; LLM will remain downstream.
- Fixture source is clearly marked and must be replaced before investment use.
- Fixture and provisional BOT sources are explicitly labelled; neither is investment-ready without validation.
- `target_weight` is recorded in the internal paper ledger; it is not an order.
## Verification
- Backend: 10 unittest tests pass.
- Independent review: **PASSED**; no concrete security or logic blockers.
- Backend: 18 unittest tests pass.
- Independent M1 review: **PASSED**; no concrete security or logic blockers.
- Reviewer suggestions: set `PAPER_COOKIE_SECURE=1` outside local HTTP; replace in-memory sessions before multi-worker deployment.
- M1 reviewer backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
- Frontend: `npm run build` passes with Vite.
- Backend health endpoint returns HTTP 200 JSON.
- Dashboard served HTML contains the current title, Vue mount point and Vite entry.
- Paper ledger POST and readback work through the live API.
- BOT source live fetch parsed 138 monthly periods and persisted raw HTML plus normalized snapshot.
- Data-health reports `source_mode=bot`, `status=provisional`, and `replayable=true`.
- Live vintage replay returned the same theme surprise as the current dashboard summary.
- Browser visual capture was blocked by Chrome remote-debugging permission; no permission dialog was clicked.

55
docs/engineering-log/2026-08-23-tourism-bot-adapter.md Normal file
View File

@@ -0,0 +1,55 @@
# 2026-08-23 — BOT Tourism source adapter
## Plan status
- M1 Tourism source adapter: complete.
- API data-health and replay endpoints: complete.
- Dashboard lineage wiring: complete.
- LLM, webhook receiver, and MT5 bridge: deferred by design.
## Changed files
- `backend/app/bot_tourism.py` — BOT ASP.NET form fetcher, table parser, YoY baseline, raw hash, atomic raw/snapshot persistence.
- `backend/app/__init__.py` — `TOURISM_SOURCE=bot`, data-health endpoint, safe vintage replay endpoint, expanded summary lineage.
- `backend/app/tourism.py` — preserves provisional source quality.
- `backend/tests/test_bot_tourism.py` — parser, default opener, fetch form, persistence and failure tests.
- `backend/tests/test_api.py` — source-mode, data-health, replay and path-safety tests.
- `frontend/src/App.vue` — live-vintage lineage fields and sign-aware surprise copy.
- `frontend/src/style.css` — provisional-quality warning badge.
- `README.md` — fixture/BOT run modes and endpoint documentation.
## Source and live evidence
- Source: `https://app.bot.or.th/BTWS_STAT/statistics/ReportPage.aspx?reportID=875&language=eng`
- Source ID: `bot.ec_ei_028_s2`
- Published: `2026-07-31T14:30:00+07:00`
- Retrieved: `2026-08-23T03:09:09.490017+00:00`
- As of: `2026-06-30`
- Release status: `provisional`
- Available monthly periods: `138`
- Baseline points: `12`
- Raw payload SHA-256: `665981f88b4a30c5bd30026cf1e96279c244ad83725558f4136d952b29756a31`
- Raw HTML and normalized snapshot were persisted locally under ignored `backend/data/`.
The latest observation is `foreign_arrivals_yoy` for `2026-06`: value `-9.8495%`, trailing-baseline expectation `-7.2791%`, robust scale `7.32`, resulting in theme surprise `-0.35114754σ`. The dashboard now says **below seasonal expectation** for this live result; it no longer assumes a bullish tourism direction.
## Verification evidence
- `PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v` — **18 tests passed**.
- `npm run build` — passed.
- BOT adapter live fetch — passed; `138` periods parsed and files persisted.
- `GET /api/v1/health` — HTTP 200, API version `0.2.0`.
- `GET /api/v1/data-health` — HTTP 200, `source_mode=bot`, `status=provisional`, `replayable=true`.
- `GET /api/v1/replay/tourism?vintage_id=bot.ec_ei_028_s2-2026-07-31-665981f88b4a` — HTTP 200; replay surprise matched live summary exactly.
- Served Vite source after restart contains `BOT live vintage`, sign-aware below copy, `Raw hash`, and `warning-tag`.
## Risks and remaining work
- BOT report is provisional and may revise. The vintage/hash contract preserves the fetched version, but the strategy must not silently treat provisional data as final.
- Only foreign-arrival YoY is live in this slice; occupancy and airport passenger metrics are still absent.
- Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
- No investment edge, transaction-cost result, or backtest conclusion has been established.
## Exact next action
Validate multiple BOT vintages and add the next real Tourism metric (occupancy or airport passenger proxy) without changing the snapshot contract. Run an event-study/backtest before introducing LLM analysis.

41
docs/test-evidence/2026-08-23-tourism-bot-adapter.md Normal file
View File

@@ -0,0 +1,41 @@
# Test evidence — 2026-08-23 BOT Tourism adapter
## Automated
```text
PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
Ran 18 tests ... OK
npm run build
Vite build completed successfully.
```
## Live source
```text
Source: https://app.bot.or.th/BTWS_STAT/statistics/ReportPage.aspx?reportID=875&language=eng
source_id: bot.ec_ei_028_s2
as_of: 2026-06-30
status: provisional
available_periods: 138
history_points: 12
replayable: true
```
## Replay integrity
The live API summary theme surprise and replay endpoint theme surprise both returned `-0.35114754`. The replay used the persisted vintage identified by `bot.ec_ei_028_s2-2026-07-31-665981f88b4a`.
## Frontend served verification
After rebuilding and restarting Vite, the served `/src/App.vue` contained the live-vintage label, sign-aware `below` copy, raw-hash lineage field, and provisional warning class. Browser screenshot capture remains unavailable because the Chrome remote-debugging permission prompt has not been approved.
## Independent review
```text
passed: true
security_concerns: []
logic_errors: []
```
Non-blocking backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.