kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[verified] add SET price snapshot adapter

Browse Source
This commit is contained in:
Kunthawat Greethong
2026-08-23 12:38:17 +07:00
parent 9301c02306
commit 7f7a6145cd
12 changed files with 705 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
README.md
View File

@@ -62,6 +62,7 @@ GET /api/v1/data-health
GET /api/v1/vintages?as_of=<ISO-8601 timestamp>
GET /api/v1/replay/tourism?vintage_id=<vintage_id>
GET /api/v1/backtest/tourism?min_events=12
GET /api/v1/prices/health
```
Source: `https://app.bot.or.th/BTWS_STAT/statistics/ReportPage.aspx?reportID=875&language=eng`
@@ -92,6 +93,7 @@ cd frontend && npm run build
- Raw response hash and normalized snapshot persistence
- Immutable vintage manifest with first-seen/revision metadata
- Read-only data-health, vintage timeline and vintage replay endpoints
- Yahoo-backed daily price snapshot contract with SET symbol mapping
- Deterministic event-study engine with benchmark and cost inputs
- Backtest readiness gate that blocks without independent vintages and prices
- Deterministic surprise × exposure × confidence score
@@ -100,6 +102,23 @@ cd frontend && npm run build
- No webhook receiver yet
- No MT5 bridge yet
The next implementation step is the event-study/backtest layer using only vintages whose `published_at` is known at each test date.
The next implementation step is replacing or supplementing revised vendor history with a point-in-time daily price source, while continuing to collect independent BOT releases.
The event-study gate is now exposed through `/api/v1/backtest/tourism`. It returns HTTP `409` with `status=blocked` when the independent-vintage minimum is not met, and it explicitly reports that a point-in-time daily price series is still required. The pure engine accepts events, daily prices, benchmark prices, event windows, and cost assumptions; it does not fetch or invent market prices.
## Collect daily price snapshots
The initial research price provider uses Yahoo Finance Chart API with SET ticker mappings. It stores OHLCV plus adjusted close for the eight exposure names and `^SET.BK` as `SET50`. This is **revised vendor history**, not point-in-time market data, so the snapshot is visible for data plumbing but cannot unlock the backtest gate:
```bash
PYTHONPATH=backend .venv/bin/python backend/scripts/collect_prices.py \
--root backend/data/prices \
--start 2024-01-01 \
--end 2026-08-24
```
Inspect the latest price snapshot:
```text
GET /api/v1/prices/health
```

40
backend/app/__init__.py
View File

@@ -16,6 +16,7 @@ from flask import Flask, jsonify, request
from .bot_tourism import BotTourismSource, TourismSourceError
from .event_study import EventStudyError, assess_backtest_readiness
from .paper import PaperLedger
from .prices import PriceSnapshotStore, PriceSourceError
from .tourism import compute_tourism_signal
from .vintages import VintageStore, VintageStoreError
@@ -52,6 +53,8 @@ def create_app(config: dict[str, Any] | None = None) -> Flask:
TOURISM_RAW_DIR=Path(os.getenv("TOURISM_RAW_DIR", str(data_root / "raw" / "tourism"))),
SNAPSHOT_DIR=Path(os.getenv("TOURISM_SNAPSHOT_DIR", str(data_root / "snapshots"))),
VINTAGE_STORE=None,
PRICE_DATA_ROOT=Path(os.getenv("PRICE_DATA_ROOT", str(data_root / "prices"))),
PRICE_STORE=None,
SNAPSHOT=_load_default_snapshot(),
)
if config:
@@ -59,6 +62,8 @@ def create_app(config: dict[str, Any] | None = None) -> Flask:
vintage_store = app.config.get("VINTAGE_STORE") or VintageStore(app.config["TOURISM_DATA_ROOT"])
app.extensions["vintage_store"] = vintage_store
price_store = app.config.get("PRICE_STORE") or PriceSnapshotStore(app.config["PRICE_DATA_ROOT"])
app.extensions["price_store"] = price_store
source_snapshot = app.config["SNAPSHOT"]
source_mode = str(app.config["TOURISM_SOURCE"]).lower()
if source_mode == "bot":
@@ -146,6 +151,31 @@ def create_app(config: dict[str, Any] | None = None) -> Flask:
snapshot_path = _snapshot_file_for(vintage_id) if vintage_id else None
return snapshot_path is not None and snapshot_path.is_file()
def _price_health_payload() -> dict[str, Any]:
try:
snapshots = list(app.extensions["price_store"].load_manifest().get("snapshots", {}).values())
except PriceSourceError as exc:
return {"available": False, "status": "error", "error": str(exc), "snapshot_count": 0}
if not snapshots:
return {"available": False, "status": "missing", "point_in_time": False, "snapshot_count": 0}
latest = max(snapshots, key=lambda item: str(item.get("retrieved_at", "")))
return {
"available": True,
"status": "available",
"snapshot_count": len(snapshots),
"snapshot_id": latest.get("snapshot_id"),
"retrieved_at": latest.get("retrieved_at"),
"period_start": latest.get("period_start"),
"period_end": latest.get("period_end"),
"quality": latest.get("quality"),
"point_in_time": bool(latest.get("point_in_time")),
"symbols": latest.get("symbols", []),
}
@app.get("/api/v1/prices/health")
def prices_health():
return jsonify(_price_health_payload())
@app.get("/api/v1/data-health")
def data_health():
current = app.extensions["tourism_result"]
@@ -184,11 +214,19 @@ def create_app(config: dict[str, Any] | None = None) -> Flask:
readiness = assess_backtest_readiness(app.extensions["vintage_store"].list_vintages(), min_events)
except (ValueError, TypeError, EventStudyError) as exc:
return jsonify({"error": str(exc)}), 400
price_snapshot = _price_health_payload()
if readiness["status"] == "ready" and (not price_snapshot.get("available") or not price_snapshot.get("point_in_time")):
readiness = {
**readiness,
"status": "blocked",
"reason": "price_series_not_point_in_time",
}
body = {
**readiness,
"theme": "tourism",
"price_series_required": True,
"next_action": "collect independent published vintages before running event study" if readiness["status"] == "blocked" else "provide point-in-time daily price series",
"price_snapshot": price_snapshot,
"next_action": "collect independent published vintages before running event study" if readiness["reason"] == "insufficient_vintages" else "provide point-in-time daily price series" if readiness["reason"] == "price_series_not_point_in_time" else "run event study",
}
return jsonify(body), 409 if readiness["status"] == "blocked" else 200

306
backend/app/prices.py Normal file
View File

@@ -0,0 +1,306 @@
"""Daily SET price snapshots from Yahoo Finance Chart API.
This provider is for research plumbing only. Historical vendor responses are
revision-prone and are deliberately marked ``point_in_time=false``.
"""
from __future__ import annotations
import copy
import hashlib
import json
import math
import re
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Mapping
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import Request, build_opener
from zoneinfo import ZoneInfo
DEFAULT_SYMBOL_MAP = {
"AOT": "AOT.BK",
"MINT": "MINT.BK",
"AWC": "AWC.BK",
"CPN": "CPN.BK",
"CPALL": "CPALL.BK",
"CRC": "CRC.BK",
"BEM": "BEM.BK",
"PTT": "PTT.BK",
"SET50": "^SET.BK",
}
PRICE_SCHEMA_VERSION = 1
PRICE_PARSER_VERSION = "yahoo-chart-v1"
MAX_PAYLOAD_BYTES = 5_000_000
_PROVIDER_SYMBOL_RE = re.compile(r"^[A-Za-z0-9^._-]{1,64}$")
class PriceSourceError(RuntimeError):
"""Raised when a price payload cannot be trusted or normalized."""
def _parse_date(value: str) -> date:
try:
return date.fromisoformat(value)
except (TypeError, ValueError) as exc:
raise PriceSourceError("price period must be YYYY-MM-DD") from exc
def _combined_hash(raw_payloads: Mapping[str, bytes]) -> str:
digest = hashlib.sha256()
for symbol in sorted(raw_payloads):
digest.update(symbol.encode("utf-8"))
digest.update(b"\0")
digest.update(raw_payloads[symbol])
digest.update(b"\0")
return digest.hexdigest()
def _raw_filename(provider_symbol: str) -> str:
if not _PROVIDER_SYMBOL_RE.fullmatch(provider_symbol):
raise PriceSourceError("invalid provider symbol")
safe = provider_symbol.replace("^", "index_").replace(".", "_")
return f"{safe}.json"
def _atomic_write(path: Path, content: bytes) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_name(f".{path.name}.tmp")
temporary.write_bytes(content)
temporary.replace(path)
def normalize_yahoo_chart(
payload: dict[str, Any],
*,
canonical_symbol: str,
provider_symbol: str,
retrieved_at: str,
raw_payload_hash: str,
) -> dict[str, Any]:
try:
chart = payload["chart"]
result = chart["result"][0]
meta = result["meta"]
timestamps = result["timestamp"]
quote = result["indicators"]["quote"][0]
adjusted = result["indicators"]["adjclose"][0]["adjclose"]
except (KeyError, IndexError, TypeError) as exc:
raise PriceSourceError("Yahoo chart result is missing required fields") from exc
if not timestamps or not isinstance(adjusted, list):
raise PriceSourceError("Yahoo chart result has no price rows")
if meta.get("exchangeName") != "SET":
raise PriceSourceError(f"unexpected exchange for {provider_symbol}")
timezone_name = str(meta.get("exchangeTimezoneName") or "Asia/Bangkok")
try:
market_zone = ZoneInfo(timezone_name)
except Exception as exc:
raise PriceSourceError("Yahoo chart returned an unknown exchange timezone") from exc
bars: list[dict[str, Any]] = []
seen_dates: set[str] = set()
def at(values: Any, index: int) -> Any:
return values[index] if isinstance(values, list) and index < len(values) else None
opens = quote.get("open", [])
highs = quote.get("high", [])
lows = quote.get("low", [])
closes = quote.get("close", [])
volumes = quote.get("volume", [])
for index, timestamp in enumerate(timestamps):
raw_close = at(closes, index)
raw_adjusted_close = at(adjusted, index)
if timestamp is None or raw_close is None or raw_adjusted_close is None:
continue
try:
timestamp_value = float(timestamp)
close = float(raw_close)
adjusted_close = float(raw_adjusted_close)
except (KeyError, IndexError, TypeError, ValueError) as exc:
raise PriceSourceError(f"invalid Yahoo price row for {provider_symbol}") from exc
if not math.isfinite(timestamp_value) or not math.isfinite(close) or not math.isfinite(adjusted_close):
continue
if close <= 0 or adjusted_close <= 0:
raise PriceSourceError(f"non-positive Yahoo close for {provider_symbol}")
trading_date = datetime.fromtimestamp(timestamp_value, tz=timezone.utc).astimezone(market_zone).date().isoformat()
if trading_date in seen_dates:
raise PriceSourceError(f"duplicate Yahoo trading date for {provider_symbol}")
seen_dates.add(trading_date)
bar: dict[str, Any] = {
"date": trading_date,
"open": at(opens, index),
"high": at(highs, index),
"low": at(lows, index),
"close": round(close, 8),
"adjusted_close": round(adjusted_close, 8),
"volume": at(volumes, index),
}
for key in ("open", "high", "low"):
if bar[key] is not None:
value = float(bar[key])
if not math.isfinite(value) or value <= 0:
raise PriceSourceError(f"invalid Yahoo {key} for {provider_symbol}")
bar[key] = round(value, 8)
if bar["volume"] is not None:
volume = float(bar["volume"])
if not math.isfinite(volume) or volume < 0:
raise PriceSourceError(f"invalid Yahoo volume for {provider_symbol}")
bar["volume"] = int(volume)
bars.append(bar)
if not bars:
raise PriceSourceError(f"Yahoo chart has no usable rows for {provider_symbol}")
bars.sort(key=lambda bar: bar["date"])
return {
"canonical_symbol": canonical_symbol,
"provider_symbol": provider_symbol,
"exchange": meta["exchangeName"],
"currency": meta.get("currency"),
"timezone": timezone_name,
"retrieved_at": retrieved_at,
"raw_payload_hash": raw_payload_hash,
"bars": bars,
}
def trading_dates(series: Mapping[str, Any]) -> list[str]:
return [bar["date"] for bar in series.get("bars", [])]
class YahooChartProvider:
"""Minimal public Yahoo chart API client with bounded response reads."""
def __init__(self, *, opener: Any | None = None, timeout: float = 30.0) -> None:
self.opener = opener or build_opener()
self.timeout = timeout
def fetch_series(self, canonical_symbol: str, provider_symbol: str, start: str, end: str) -> tuple[dict[str, Any], bytes]:
start_date = _parse_date(start)
end_date = _parse_date(end)
if end_date <= start_date:
raise PriceSourceError("price end must be after price start")
period1 = int(datetime.combine(start_date, datetime.min.time(), tzinfo=timezone.utc).timestamp())
period2 = int(datetime.combine(end_date + timedelta(days=1), datetime.min.time(), tzinfo=timezone.utc).timestamp())
query = urlencode({"period1": period1, "period2": period2, "interval": "1d", "events": "div,splits", "includeAdjustedClose": "true"})
url = f"https://query1.finance.yahoo.com/v8/finance/chart/{provider_symbol}?{query}"
request = Request(url, headers={"User-Agent": "SET50-Alternative-Data-Platform/0.4"})
try:
with self.opener.open(request, timeout=self.timeout) as response:
if getattr(response, "status", 200) >= 400:
raise PriceSourceError(f"Yahoo price source returned HTTP {response.status}")
raw_bytes = response.read(MAX_PAYLOAD_BYTES + 1)
except PriceSourceError:
raise
except (HTTPError, URLError, TimeoutError, OSError) as exc:
raise PriceSourceError(f"Yahoo price source request failed: {exc.__class__.__name__}") from exc
if len(raw_bytes) > MAX_PAYLOAD_BYTES:
raise PriceSourceError("Yahoo price payload exceeds size limit")
try:
payload = json.loads(raw_bytes)
except json.JSONDecodeError as exc:
raise PriceSourceError("Yahoo price payload is not valid JSON") from exc
retrieved_at = datetime.now(timezone.utc).isoformat()
normalized = normalize_yahoo_chart(
payload,
canonical_symbol=canonical_symbol,
provider_symbol=provider_symbol,
retrieved_at=retrieved_at,
raw_payload_hash=hashlib.sha256(raw_bytes).hexdigest(),
)
return normalized, raw_bytes
class PriceSnapshotStore:
"""Filesystem store for normalized price snapshots and raw provider payloads."""
def __init__(self, root: Path | str) -> None:
self.root = Path(root).resolve()
self.raw_dir = self.root / "raw"
self.snapshot_dir = self.root / "snapshots"
self.manifest_path = self.root / "manifest.json"
def load_manifest(self) -> dict[str, Any]:
if not self.manifest_path.is_file():
return {"schema_version": PRICE_SCHEMA_VERSION, "snapshots": {}}
try:
manifest = json.loads(self.manifest_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise PriceSourceError("price manifest is unreadable") from exc
if manifest.get("schema_version") != PRICE_SCHEMA_VERSION or not isinstance(manifest.get("snapshots"), dict):
raise PriceSourceError("unsupported price manifest schema")
return manifest
def persist(self, snapshot: dict[str, Any], raw_payloads: Mapping[str, bytes]) -> dict[str, Any]:
source = snapshot.get("source")
if not isinstance(source, dict) or not source.get("snapshot_id"):
raise PriceSourceError("price snapshot source metadata is required")
if not raw_payloads or any(not isinstance(value, bytes) or not value for value in raw_payloads.values()):
raise PriceSourceError("price raw payloads must be non-empty bytes")
raw_hash = _combined_hash(raw_payloads)
if source.get("raw_payload_hash") and source["raw_payload_hash"] != raw_hash:
raise PriceSourceError("raw payload hash does not match price snapshot")
stored = copy.deepcopy(snapshot)
stored_source = stored["source"]
snapshot_id = str(stored_source["snapshot_id"])
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}", snapshot_id):
raise PriceSourceError("invalid price snapshot id")
stored_source["raw_payload_hash"] = raw_hash
stored_source["raw_payload_files"] = {symbol: _raw_filename(symbol) for symbol in raw_payloads}
stored_source["snapshot_file"] = f"{snapshot_id}.json"
for provider_symbol, payload in raw_payloads.items():
_atomic_write(self.raw_dir / snapshot_id / _raw_filename(provider_symbol), payload)
_atomic_write(self.snapshot_dir / stored_source["snapshot_file"], (json.dumps(stored, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode("utf-8"))
manifest = self.load_manifest()
manifest["snapshots"][snapshot_id] = {
"snapshot_id": snapshot_id,
"source_id": stored_source.get("source_id"),
"retrieved_at": stored_source.get("retrieved_at"),
"period_start": stored_source.get("period_start"),
"period_end": stored_source.get("period_end"),
"raw_payload_hash": raw_hash,
"quality": stored_source.get("quality"),
"point_in_time": stored_source.get("point_in_time"),
"symbols": sorted(stored.get("series", {}).keys()),
"snapshot_file": stored_source["snapshot_file"],
}
_atomic_write(self.manifest_path, (json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode("utf-8"))
return stored
def collect_price_snapshot(
root: Path | str,
*,
start: str,
end: str,
symbol_map: Mapping[str, str] = DEFAULT_SYMBOL_MAP,
provider: YahooChartProvider | Any | None = None,
retrieved_at: str | None = None,
) -> dict[str, Any]:
if not symbol_map:
raise PriceSourceError("symbol_map must not be empty")
provider = provider or YahooChartProvider()
normalized: dict[str, Any] = {}
raw_payloads: dict[str, bytes] = {}
for canonical_symbol, provider_symbol in symbol_map.items():
series, raw_bytes = provider.fetch_series(canonical_symbol, provider_symbol, start, end)
normalized[canonical_symbol] = series
raw_payloads[provider_symbol] = raw_bytes
raw_hash = _combined_hash(raw_payloads)
retrieved = retrieved_at or datetime.now(timezone.utc).isoformat()
snapshot_id = f"prices-yahoo-chart-{start}-{end}-{raw_hash[:12]}"
source = {
"source_id": "yahoo.chart",
"source_url": "https://query1.finance.yahoo.com/v8/finance/chart/{provider_symbol}",
"retrieved_at": retrieved,
"period_start": start,
"period_end": end,
"raw_payload_hash": raw_hash,
"parser_version": PRICE_PARSER_VERSION,
"quality": "revised_vendor_history",
"point_in_time": False,
"adjusted_prices": True,
"snapshot_id": snapshot_id,
"bar_counts": {symbol: len(series["bars"]) for symbol, series in normalized.items()},
}
snapshot = {"schema_version": PRICE_SCHEMA_VERSION, "source": source, "series": normalized, "benchmark_symbol": "SET50" if "SET50" in normalized else None}
return PriceSnapshotStore(root).persist(snapshot, raw_payloads)

42
backend/scripts/collect_prices.py Normal file
View File

@@ -0,0 +1,42 @@
"""Collect one daily SET price snapshot from Yahoo Finance Chart API."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from app.prices import YahooChartProvider, collect_price_snapshot
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1] / "data" / "prices")
parser.add_argument("--start", required=True, help="inclusive YYYY-MM-DD")
parser.add_argument("--end", required=True, help="inclusive YYYY-MM-DD")
parser.add_argument("--timeout", type=float, default=30.0)
args = parser.parse_args()
snapshot = collect_price_snapshot(args.root, start=args.start, end=args.end, provider=YahooChartProvider(timeout=args.timeout))
source = snapshot["source"]
print(
json.dumps(
{
"snapshot_id": source["snapshot_id"],
"source_id": source["source_id"],
"period_start": source["period_start"],
"period_end": source["period_end"],
"symbols": sorted(snapshot["series"]),
"bar_counts": source["bar_counts"],
"quality": source["quality"],
"point_in_time": source["point_in_time"],
"snapshot_root": str(args.root),
},
ensure_ascii=False,
sort_keys=True,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

28
backend/tests/fixtures/yahoo_chart_aot.json vendored Normal file
View File

@@ -0,0 +1,28 @@
{
"chart": {
"result": [
{
"meta": {
"currency": "THB",
"symbol": "AOT.BK",
"exchangeName": "SET",
"exchangeTimezoneName": "Asia/Bangkok"
},
"timestamp": [1767322800, 1767582000],
"indicators": {
"quote": [
{
"open": [60.0, 61.0],
"high": [62.0, 63.0],
"low": [59.5, 60.5],
"close": [61.5, 62.5],
"volume": [1000000, 1100000]
}
],
"adjclose": [{"adjclose": [60.0, 61.25]}]
}
}
],
"error": null
}
}

32
backend/tests/test_api.py
View File

@@ -4,6 +4,7 @@ import unittest
from pathlib import Path
from app import create_app
from app.prices import PriceSnapshotStore
from app.vintages import VintageStore
@@ -99,13 +100,42 @@ class ApiTests(unittest.TestCase):
with tempfile.TemporaryDirectory() as temp_dir:
store = VintageStore(Path(temp_dir))
store.persist(b"fixture raw", self.snapshot)
app = create_app({"TESTING": True, "SNAPSHOT": self.snapshot, "VINTAGE_STORE": store})
app = create_app({"TESTING": True, "SNAPSHOT": self.snapshot, "VINTAGE_STORE": store, "PRICE_STORE": PriceSnapshotStore(Path(temp_dir) / "prices")})
response = app.test_client().get("/api/v1/backtest/tourism?min_events=12")
body = response.get_json()
self.assertEqual(response.status_code, 409)
self.assertEqual(body["status"], "blocked")
self.assertEqual(body["reason"], "insufficient_vintages")
self.assertEqual(body["available_events"], 1)
self.assertEqual(body["price_snapshot"]["status"], "missing")
def test_backtest_readiness_blocks_revised_or_missing_prices(self):
class FakeVintageStore:
def list_vintages(self, _as_of=None):
return [{"vintage_id": f"vintage-{index}"} for index in range(12)]
with tempfile.TemporaryDirectory() as temp_dir:
app = create_app(
{
"TESTING": True,
"VINTAGE_STORE": FakeVintageStore(),
"PRICE_STORE": PriceSnapshotStore(Path(temp_dir)),
}
)
response = app.test_client().get("/api/v1/backtest/tourism?min_events=12")
body = response.get_json()
self.assertEqual(response.status_code, 409)
self.assertEqual(body["status"], "blocked")
self.assertEqual(body["reason"], "price_series_not_point_in_time")
def test_prices_health_reports_missing_snapshot(self):
with tempfile.TemporaryDirectory() as temp_dir:
app = create_app({"TESTING": True, "PRICE_STORE": PriceSnapshotStore(Path(temp_dir))})
response = app.test_client().get("/api/v1/prices/health")
body = response.get_json()
self.assertEqual(response.status_code, 200)
self.assertFalse(body["available"])
self.assertEqual(body["status"], "missing")
def test_backtest_readiness_rejects_invalid_min_events(self):
response = self.client.get("/api/v1/backtest/tourism?min_events=bad")

105
backend/tests/test_prices.py Normal file
View File

@@ -0,0 +1,105 @@
import copy
import hashlib
import json
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
from app.prices import (
DEFAULT_SYMBOL_MAP,
PriceSnapshotStore,
PriceSourceError,
collect_price_snapshot,
normalize_yahoo_chart,
trading_dates,
)
FIXTURE = Path(__file__).parent / "fixtures" / "yahoo_chart_aot.json"
PAYLOAD = json.loads(FIXTURE.read_text(encoding="utf-8"))
class FakePriceProvider:
def __init__(self, payload):
self.payload = payload
self.calls = []
def fetch_series(self, canonical_symbol, provider_symbol, start, end):
self.calls.append((canonical_symbol, provider_symbol, start, end))
body = json.dumps(self.payload, sort_keys=True).encode("utf-8")
return normalize_yahoo_chart(
self.payload,
canonical_symbol=canonical_symbol,
provider_symbol=provider_symbol,
retrieved_at="2026-08-23T06:00:00+00:00",
raw_payload_hash=hashlib.sha256(body).hexdigest(),
), body
class PriceSnapshotTests(unittest.TestCase):
def test_default_symbol_map_uses_set_provider_symbols(self):
self.assertEqual(DEFAULT_SYMBOL_MAP["AOT"], "AOT.BK")
self.assertEqual(DEFAULT_SYMBOL_MAP["SET50"], "^SET.BK")
self.assertEqual(DEFAULT_SYMBOL_MAP["PTT"], "PTT.BK")
def test_normalize_yahoo_chart_preserves_adjusted_close_and_trading_dates(self):
series = normalize_yahoo_chart(
PAYLOAD,
canonical_symbol="AOT",
provider_symbol="AOT.BK",
retrieved_at="2026-08-23T06:00:00+00:00",
raw_payload_hash="a" * 64,
)
self.assertEqual(series["exchange"], "SET")
self.assertEqual(series["currency"], "THB")
self.assertEqual([bar["date"] for bar in series["bars"]], ["2026-01-02", "2026-01-05"])
self.assertEqual(series["bars"][0]["adjusted_close"], 60.0)
self.assertEqual(trading_dates(series), ["2026-01-02", "2026-01-05"])
def test_normalize_yahoo_chart_skips_incomplete_rows(self):
payload = copy.deepcopy(PAYLOAD)
payload["chart"]["result"][0]["indicators"]["quote"][0]["close"][0] = None
payload["chart"]["result"][0]["indicators"]["adjclose"][0]["adjclose"][0] = None
series = normalize_yahoo_chart(
payload,
canonical_symbol="AOT",
provider_symbol="AOT.BK",
retrieved_at="2026-08-23T06:00:00+00:00",
raw_payload_hash="a" * 64,
)
self.assertEqual(len(series["bars"]), 1)
def test_normalize_yahoo_chart_rejects_missing_result(self):
payload = copy.deepcopy(PAYLOAD)
payload["chart"]["result"] = []
with self.assertRaisesRegex(PriceSourceError, "result"):
normalize_yahoo_chart(payload, canonical_symbol="AOT", provider_symbol="AOT.BK", retrieved_at="2026-08-23T06:00:00+00:00", raw_payload_hash="a" * 64)
def test_collector_persists_revised_vendor_history_snapshot(self):
provider = FakePriceProvider(PAYLOAD)
with tempfile.TemporaryDirectory() as temp_dir:
snapshot = collect_price_snapshot(
Path(temp_dir),
start="2026-01-01",
end="2026-01-06",
symbol_map={"AOT": "AOT.BK", "SET50": "^SET.BK"},
provider=provider,
retrieved_at="2026-08-23T06:00:00+00:00",
)
self.assertEqual(snapshot["source"]["quality"], "revised_vendor_history")
self.assertFalse(snapshot["source"]["point_in_time"])
self.assertEqual(snapshot["source"]["bar_counts"]["AOT"], 2)
manifest = PriceSnapshotStore(Path(temp_dir)).load_manifest()
self.assertEqual(len(manifest["snapshots"]), 1)
self.assertEqual(len(provider.calls), 2)
def test_price_store_rejects_snapshot_with_mismatched_raw_hash(self):
with tempfile.TemporaryDirectory() as temp_dir:
store = PriceSnapshotStore(Path(temp_dir))
snapshot = {"source": {"snapshot_id": "prices-test", "raw_payload_hash": "a" * 64}}
with self.assertRaisesRegex(PriceSourceError, "raw payload hash"):
store.persist(snapshot, {"AOT.BK": b"different"})
if __name__ == "__main__":
unittest.main()

13
docs/HANDOFF.md
View File

@@ -4,8 +4,8 @@
- Path: `/Users/kunthawat/Gitea/set50-alternative-data-platform`
- Branch: `main`
- Verified code commit: `0b47a06` — `[verified] add event-study readiness gate`
- Current milestone: M2.3 event-study foundation complete; real backtest blocked
- Verified code commit: `0b47a06` — `[verified] add event-study readiness gate` (price adapter pending commit)
- Current milestone: M2.4 price snapshot foundation complete; real backtest blocked
- Mode: research + paper only
- Frontend: Vue 3 + Vite
- Backend: Flask `0.4.0`
@@ -24,6 +24,8 @@
- Point-in-time vintage query: `GET /api/v1/vintages?as_of=<ISO-8601>`.
- Deterministic event-study engine with window, benchmark and cost calculations.
- Backtest readiness gate: `GET /api/v1/backtest/tourism?min_events=12`.
- Yahoo-backed daily price snapshot with SET symbol mapping and adjusted-close bars.
- Price health: `GET /api/v1/prices/health`.
- Ranked target weights and LONG/SHORT/NEUTRAL classification.
- English dashboard with live/provisional source label, sign-aware surprise copy and lineage fields.
- HttpOnly paper session and internal paper ledger.
@@ -71,6 +73,9 @@ HTTP 200; count=1; manifest seen_count=4
GET /api/v1/backtest/tourism?min_events=12
HTTP 409; status=blocked, available_events=1, required_events=12, price_series_required=true
GET /api/v1/prices/health
HTTP 200; available=true, quality=revised_vendor_history, point_in_time=false, symbols=9
```
Paper writes use a server-side token exchange and HttpOnly `paper_session` cookie; the token is not embedded in the frontend bundle.
@@ -82,9 +87,9 @@ Paper writes use a server-side token exchange and HttpOnly `paper_session` cooki
- Snapshot storage is local filesystem and single-process; shared persistence is required before multi-worker deployment.
- No investment edge, transaction-cost result, or backtest conclusion has been established.
- One independent source release is not enough for a valid event study; current historical rows are not treated as point-in-time vintages.
- The event-study engine is deterministic and tested, but no real price provider is connected yet.
- The event-study engine is deterministic and tested. Yahoo price history is connected for plumbing, but it is revised vendor history, not point-in-time data.
- Browser screenshot verification remains blocked by the Chrome remote-debugging permission prompt; served HTML/source, live API, fresh Vite build and replay integrity were verified instead.
## Exact next action
Collect independent BOT releases over time and add a point-in-time daily price provider. Only then raise the readiness gate and run the event study. Add another metric only when its historical release coverage is real.
Collect independent BOT releases over time and replace/supplement revised vendor history with a point-in-time daily price source. Only then raise the readiness gate and run the event study. Add another metric only when its historical release coverage is real.

6
docs/engineering-log.md
View File

@@ -8,6 +8,7 @@
| M1 BOT Tourism adapter | complete | 18 tests, live BOT fetch, raw/snapshot persistence | validate multiple vintages |
| M2 vintage collector | complete | 25 tests, manifest idempotency, live collector and point-in-time API | collect independent releases |
| M2.3 event-study gate | complete/blocked | 30 tests, pure engine and truthful 409 readiness API | add point-in-time price provider |
| M2.4 price snapshot adapter | complete/blocked | 38 tests, live 9-symbol Yahoo snapshot, revised-history gate | evaluate point-in-time price source |
| Tourism deterministic signal | complete | live foreign-arrivals YoY surprise | add occupancy/airport metric |
| Internal paper ledger | complete | POST/readback through live API | persist in PostgreSQL later |
| Dashboard | complete | Vite build + served source check with live-sign copy | visual browser capture after permission is available |
@@ -25,7 +26,7 @@
## Verification
- Backend: 30 unittest tests pass.
- Backend: 38 unittest tests pass.
- Independent M1 review: **PASSED**; no concrete security or logic blockers.
- Reviewer suggestions: set `PAPER_COOKIE_SECURE=1` outside local HTTP; replace in-memory sessions before multi-worker deployment.
- M1 reviewer backlog: add schema-drift, duplicate/reordered-row, and malformed-vintage regression fixtures.
@@ -40,4 +41,7 @@
- Independent M2 review: **PASSED**; no concrete security or logic blockers.
- Event-study readiness gate correctly returns HTTP 409 with 1/12 independent vintages; no backtest result is fabricated.
- Independent M2.3 review: **PASSED**; no concrete security or logic blockers.
- Price snapshot normalized 9 symbols with adjusted close and provider-derived trading dates; quality is explicitly `revised_vendor_history` and `point_in_time=false`.
- Backtest gate requires both independent vintages and point-in-time prices.
- Independent M2.4 review: **PASSED**; no concrete security or logic blockers.
- Browser visual capture was blocked by Chrome remote-debugging permission; no permission dialog was clicked.

72
docs/engineering-log/2026-08-23-price-snapshot.md Normal file
View File

@@ -0,0 +1,72 @@
# 2026-08-23 — SET price snapshot adapter
## Plan status
- SET symbol mapping: complete.
- Daily OHLCV/adjusted-close snapshot contract: complete.
- Yahoo Chart API provider: complete and live-tested.
- Trading-date extraction from provider bars: complete.
- Point-in-time price quality: intentionally blocked; Yahoo history is labelled revised vendor history.
## Changed files
- `backend/app/prices.py` — Yahoo Chart API client, SET symbol map, payload normalization, adjusted-close handling, trading dates, raw/snapshot store and quality metadata.
- `backend/scripts/collect_prices.py` — one-shot daily price collector.
- `backend/app/__init__.py` — price store wiring, `/api/v1/prices/health`, and backtest gate price-quality check.
- `backend/tests/test_prices.py` — mapping, normalized bars, adjusted prices, incomplete-row handling, storage/hash tests.
- `backend/tests/test_api.py` — missing/revised price gate tests.
- `frontend/src/App.vue` — Backtest gate shows price quality beside vintage count.
- `README.md` — price collector and revised-vendor-history boundary.
## Live source evidence
Provider: Yahoo Finance Chart API, used as a research plumbing source only.
```text
snapshot_id: prices-yahoo-chart-2024-01-01-2026-08-24-a8b5c7abae7c
source_id: yahoo.chart
period_start: 2024-01-01
period_end: 2026-08-24
symbols: 9 (AOT, AWC, BEM, CPALL, CPN, CRC, MINT, PTT, SET50)
quality: revised_vendor_history
point_in_time: false
```
The provider returned 645 bars for each exposure symbol and 614 bars for the SET50 index series after incomplete rows were skipped. The snapshot is available to inspect, but it cannot unlock a real backtest because its historical values were retrieved/revised as a current vendor response.
## Readiness behavior
With the live price snapshot and one BOT vintage:
```text
GET /api/v1/prices/health
HTTP 200; available=true, quality=revised_vendor_history, point_in_time=false
GET /api/v1/backtest/tourism?min_events=12
HTTP 409; status=blocked, reason=insufficient_vintages
```
If the vintage threshold were met while prices remained non-point-in-time, the gate would still return `reason=price_series_not_point_in_time`.
## Verification
- `PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v` — **38 tests passed**.
- `npm run build` — passed.
- `npm audit --omit=dev --audit-level=high` — 0 vulnerabilities.
- Yahoo live fetch — 9 symbols normalized and persisted.
- API price health and readiness gate — passed.
- Served Vue source contains the Backtest gate and price-quality copy.
## Independent review
```text
passed: true
security_concerns: []
logic_errors: []
```
Non-blocking backlog: add explicit revised/non-point-in-time gate coverage and document adjusted-close semantics for downstream calculations.
## Risks and exact next action
Yahoo historical data is not a point-in-time price source. Do not interpret the snapshot as a valid historical execution series. Next, evaluate a price source with release/availability timestamps or explicitly maintain a separate research-only revised-history mode.

47
docs/test-evidence/2026-08-23-price-snapshot.md Normal file
View File

@@ -0,0 +1,47 @@
# Test evidence — 2026-08-23 SET price snapshot
## Automated
```text
PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
Ran 38 tests ... OK
npm run build
Vite build completed successfully.
npm audit --omit=dev --audit-level=high
found 0 vulnerabilities
```
## Live Yahoo snapshot
```text
snapshot_id: prices-yahoo-chart-2024-01-01-2026-08-24-a8b5c7abae7c
symbols: 9
quality: revised_vendor_history
point_in_time: false
```
## Live API
```text
GET /api/v1/prices/health
HTTP 200; available=true
GET /api/v1/backtest/tourism?min_events=12
HTTP 409; blocked because only 1 independent BOT vintage exists
```
## Frontend served verification
After rebuild/restart, the served Vue source contains the `Backtest gate` KPI and price-quality value path. Screenshot capture remains unavailable because the Chrome remote-debugging permission prompt has not been approved.
## Independent review
```text
passed: true
security_concerns: []
logic_errors: []
```
Non-blocking backlog: add explicit revised/non-point-in-time gate coverage and document adjusted-close semantics for downstream calculations.

2
frontend/src/App.vue
View File

@@ -220,7 +220,7 @@ onMounted(loadDashboard)
<article class="kpi-card">
<div class="kpi-label">Backtest gate</div>
<div class="kpi-value quality-value">{{ backtest?.status || '—' }}</div>
<div class="kpi-foot">{{ backtest?.available_events || 0 }} / {{ backtest?.required_events || 0 }} vintages</div>
<div class="kpi-foot">{{ backtest?.available_events || 0 }} / {{ backtest?.required_events || 0 }} vintages · {{ backtest?.price_snapshot?.quality || 'price snapshot missing' }}</div>
</article>
</section>