kunthawat/set50-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(backtest): runnable with partial factor coverage + oldest-history default (owner rule)

Browse Source 
Owner's rule: backtest must run as soon as there's enough data to estimate an
investment — it must NOT be blocked just because some sources lack deep PIT
history. Scoring is deliberately flexible (a theme uses whatever subset of
factors was knowable that day).

- backtest_readiness: readiness = usable window (price + Siamchart + >=1 factor),
  not all-factors-present. Missing factors still reported (transparency) but no
  longer block the run. recommended_start = oldest executable price (oldest
  history the system holds); recommended_end = last complete trading day.
- pit_scorer.theme_surprise_report: flexible — skips factors not released by
  as_of; blocked only when NO factor has a value. pit_meta.partial_pit reflects
  themes scored from a partial factor subset.
- Verified end-to-end: readiness ready=true (recommended 2024-01-03 -> 2026-08-29);
  POST /api/v1/backtest/run default window returns 201 full result (1M -> final
  equity 1,117,243.95), no 400 from missing factors.
- test_backtest_readiness updated to earliest-runnable semantics; full suite 369 green.
This commit is contained in:
Kunthawat Greethong
2026-08-30 20:27:22 +07:00
parent 053b94b45f
commit be6a92540c
4 changed files with 123 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

78
backend/app/backtest_readiness.py
View File

@@ -196,11 +196,18 @@ def evaluate_readiness(
start/end: optional explicit window; if given, readiness is evaluated
only against ``[start, end]``.
"Ready" requires, within the window:
1. every registry factor has a released value by the start date;
"Ready" requires only a *usable* window — NOT that every factor is present
(the owner's rule: scoring is deliberately flexible, a theme scores from
whatever subset of factors was knowable that day, so a backtest may run as
soon as there is enough data to estimate, even if some sources are missing
or don't provide history):
1. at least one registry factor has a released value by the start date;
2. a Siamchart snapshot retrieved no later than the start date;
3. at least one executable price in the window for every required symbol.
Missing inputs are reported explicitly rather than silently skipped.
Missing inputs are still reported for transparency, but a partial window is
runnable rather than rejected. ``recommended_start`` is the EARLIEST date at
which usable data exists (the oldest history the system holds), and
``recommended_end`` defaults to the last complete trading day (yesterday).
"""
miss: list[str] = []
cov: list[DataCoverage] = []
@@ -252,46 +259,77 @@ def evaluate_readiness(
if not scv.available:
miss.append("siamchart")
# derive recommended start = the latest earliest-available date among the
# inputs (the point at which *all* of them are simultaneously available).
candidates = [
d for d in (fac_earliest, sc_earliest, pc.earliest)
if d is not None
]
# recommended start = the EARLIEST date a run is genuinely executable from —
# i.e. the oldest price data (you must be able to value the holdings). Factors
# and Siamchart are flexible (they may start later; scoring just uses whatever
# subset was known by each day), but price is the hard floor. So the default
# start is the oldest available executable price, per the owner's wish to
# backtest over the oldest history the system holds.
recommended_end = _date_from_ts(end) or today
if start is not None:
recommended_start = _parse_date(start)
elif candidates:
recommended_start = max(candidates)
elif pc.available and pc.earliest is not None:
recommended_start = pc.earliest
else:
recommended_start = None
candidates = [d for d in (fac_earliest, sc_earliest) if d is not None]
recommended_start = min(candidates) if candidates else None
# cap recommended start so it never exceeds the end
if recommended_start and recommended_end and recommended_start > recommended_end:
recommended_start = recommended_end
# explicit window requested: readiness is whether the window is covered
# ---- usable-window gate (NOT all-factors-present) ---------------------
# Scoring is flexible: a theme scores from whatever factors were known by
# as_of, so the backtest is runnable once we have a price series, a Siamchart
# snapshot, and *at least one* released factor. Missing individual factors
# are still reported in `missing` for transparency but do NOT block the run.
hard_block = (
not pc.available
or not scv.available
or fac_earliest is None # no factor has any released value at all
)
if start is not None:
s = _parse_date(start)
e_ = _parse_date(end) if end else today
# price must be executable AT the start: an explicit start that predates
# all usable price history would otherwise manufacture a false PIT
# window (factor + Siamchart are already checked at start above).
# all usable price history would otherwise manufacture a false PIT window.
if pc.available and pc.earliest is not None and s < pc.earliest:
hard_block = True
miss.append("price")
ready = (not miss) and s <= e_
# an explicit start with no factor released by it is unusable
if fac_earliest is None or s < fac_earliest:
hard_block = True
if "factor" not in miss:
miss.append("factor")
ready = (not hard_block) and s <= e_
else:
ready = (not miss) and bool(recommended_start)
ready = (not hard_block) and bool(recommended_start)
if not scv.available and "siamchart" not in miss:
miss.append("siamchart")
if not pc.available and "price" not in miss:
miss.append("price")
if fac_earliest is None and "factor" not in miss:
miss.append("factor")
reason = ""
rs = recommended_start.isoformat() if recommended_start else None
re_iso = recommended_end.isoformat()
if ready:
fac_missing_n = fac_cov[0].missing_count if (fac_cov and fac_cov[0].source == "factor") else 0
reason = (
f"strict PIT coverage from {rs} to {re_iso}"
f"usable PIT coverage from {rs} to {re_iso}"
+ (f" ({fac_missing_n} factors without history)"
if fac_missing_n else "")
)
elif miss:
reason = "missing inputs: " + ", ".join(sorted(set(miss)))
elif hard_block:
blockers = []
if not pc.available:
blockers.append("price")
if not scv.available:
blockers.append("siamchart")
if fac_earliest is None:
blockers.append("no factor has any released value")
reason = "missing hard inputs: " + ", ".join(blockers)
else:
reason = "no usable PIT-ready window"

54
backend/app/pit_scorer.py
View File

@@ -123,19 +123,29 @@ class PitScoreProvider:
# -- theme PIT score -------------------------------------------------
def theme_surprise_report(self, theme_key: str, as_of: str) -> dict[str, Any]:
"""Weighted-average theme surprise from PIT factor values only.
"""Weighted-average theme surprise from the factors that ARE released by
``as_of`` — flexible, not all-or-nothing.
Returns {theme, surprise, blocked, partial_pit}. ``blocked`` is True
when at least one of the theme's factors had no value released by
``as_of`` — a caller must never treat a blocked theme as PIT.
``theme_surprise_report`` intentionally matches the live dashboard's
``compute_theme_surprises`` semantics: any factor with no PIT value by
``as_of`` is simply omitted from the weighted average, so a theme scores
from whatever subset of its factors was genuinely knowable that day and
is never blocked by a single missing source. ``blocked`` is only True
when NO factor in the theme had a released value by ``as_of`` (nothing
to compute). ``used``/``total`` report how many factors contributed, so
callers can judge how thin the estimate is.
PIT integrity is preserved: we only ever use values released <= ``as_of``
(``factor_at``), never future knowledge.
"""
factors = self.theme_factor_map.get(theme_key, [])
if not factors:
return {"theme": theme_key, "surprise": None, "blocked": True, "partial_pit": True}
return {"theme": theme_key, "surprise": None, "blocked": True,
"partial_pit": True, "used": 0, "total": 0}
from . import factors as factors_mod
weighted = 0.0
w_sum = 0.0
blocked = False
used = 0
for spec in factors:
fkey = spec.get("key")
if not isinstance(fkey, str) or not fkey:
@@ -145,8 +155,7 @@ class PitScoreProvider:
continue
value = self.factor_at(fkey, as_of)
if value is None:
blocked = True
continue
continue # not released yet -> flexibly skip (no blanket block)
norm = factors_mod.normalize(
value, sign=fact.get("sign", 1),
center=fact.get("center", 0.0), span=fact.get("span", 10.0),
@@ -155,17 +164,21 @@ class PitScoreProvider:
continue
weighted += spec.get("weight", 1.0) * norm
w_sum += abs(spec.get("weight", 1.0))
surprise: Optional[float]
used += 1
total = len([f for f in factors if isinstance(f.get("key"), str) and f.get("key")])
if w_sum == 0:
surprise = None
blocked = True
blocked = True # nothing had a released value -> cannot score
else:
surprise = round(min(1.0, max(-1.0, weighted / w_sum)), 3)
blocked = False
return {
"theme": theme_key,
"surprise": surprise,
"blocked": blocked,
"partial_pit": blocked,
"partial_pit": used < total,
"used": used,
"total": total,
}
# -- per-symbol combined board as of a date ----------------------------
@@ -189,6 +202,7 @@ class PitScoreProvider:
theme_scores: dict[str, dict[str, float]] = {}
blocked_themes: list[str] = []
partial_themes: list[str] = []
fv = self.siamchart_factor_view(as_of)
# reverse-map symbol -> fundamental for quality_within_theme's factor view
fv_for_quality = {
@@ -203,6 +217,8 @@ class PitScoreProvider:
blocked_themes.append(tid)
theme_scores[tid] = {}
continue
if rep["partial_pit"]:
partial_themes.append(tid)
q = {}
for sym in themes_mod.THEME_SYMBOLS.get(tid, set()):
if sym not in fv:
@@ -212,7 +228,7 @@ class PitScoreProvider:
theme_scores[tid] = q
combined = themes_mod.combine_score(list(theme_scores.values()), siamchart_score)
is_full_pit = not blocked_themes
is_full_pit = not blocked_themes and not partial_themes
# fundamental is PIT only when a siamchart vintage store is wired;
# otherwise the current snapshot makes the overall result partial.
fundamental_pit = self.siamchart_store is not None
@@ -227,11 +243,17 @@ class PitScoreProvider:
"dividend_yield": fm.get("dividend_yield") or 0.0,
"pit_meta": {
"pit": is_full_pit and fundamental_pit,
"partial_pit": not fundamental_pit,
"partial_pit": (not fundamental_pit) or bool(partial_themes),
"blocked_theme": blocked_themes,
"note": ("theme surprises PIT + siamchart fundamental PIT (store)"
if fundamental_pit else
"theme surprises PIT; siamchart fundamental partial (current snapshot)"),
"partial_themes": partial_themes,
"note": (
"theme surprises PIT + siamchart fundamental PIT (store)"
if (fundamental_pit and not partial_themes) else
("theme surprises PIT; some themes partial (missing factors); "
"siamchart fundamental PIT (store)"
if fundamental_pit else
"theme surprises PIT; siamchart fundamental partial (current snapshot)")
),
},
}
# fall back to the current board for any symbol the PIT path could not

8
backend/tests/test_backtest_readiness.py
View File

@@ -123,7 +123,9 @@ class NoSiamchartSnapshotTest(unittest.TestCase):
class RecommendedStartTest(unittest.TestCase):
def test_start_is_latest_of_first_ready_dates(self):
def test_start_is_oldest_runnable_price(self):
# Owner rule: backtest default start = oldest history the system holds
# (the oldest executable price), NOT the date every input co-exists.
# factors ready 2025-01-01, siamchart ready 2025-06-01, price from 2024
store = full_factor_store("2025-01-01")
sc = FakeSiamchartStore(["2025-06-01T09:00:00+07:00"])
@@ -132,7 +134,9 @@ class RecommendedStartTest(unittest.TestCase):
factor_store=store, siamchart_store=sc, price_series=series
)
self.assertTrue(res.ready)
self.assertEqual(res.recommended_start, "2025-06-01")
# oldest runnable history = oldest price (2024-01-01), even though
# factors/siamchart start later (flexible scoring uses what's known).
self.assertEqual(res.recommended_start, "2024-01-01")
self.assertEqual(res.recommended_end, today_iso())
def test_start_limited_by_price_when_price_latest(self):

21
docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md
View File

@@ -88,3 +88,24 @@ needs-deeper-capture and left as a decision for the owner.
- Note: a **concurrent process** also landed `thai_trade.py` (external-sector
exports/imports/current-account) and external_* factors mid-session; its 3
initially-broken tests were fixed to reach the 360-green baseline here.
## Flexible PIT backtest (owner rule — 2026-08-30)
Owner clarified the backtest contract: it must run as soon as there is enough
data to estimate an investment, NOT only when EVERY factor has PIT history.
Some sources may not provide deep history — scoring is deliberately flexible
(a theme scores from whatever subset of factors was knowable that day).
Implemented (commit 2026-08-30):
- `backtest_readiness.evaluate_readiness`: readiness is now a *usable* window
(price + Siamchart + >=1 released factor), not all-factors-present. Missing
factors are still reported in `missing` for transparency but no longer block.
`recommended_start` = oldest executable price (the oldest history held);
`recommended_end` = last complete trading day (yesterday Bangkok).
- `pit_scorer.theme_surprise_report`: flexible — skips any factor with no release
by `as_of` (no blanket theme block); `blocked` only when NO factor has a value.
`pit_meta.partial_pit` reflects themes that scored from a partial subset.
- Verified: `/api/v1/backtest/readiness` → ready=true, recommended 2024-01-03 →
2026-08-29. `POST /api/v1/backtest/run` (default window) → 201 full result
(final_equity 1,117,243.95 on 1M), no 400 from missing factors.
- Tests: test_backtest_readiness updated to earliest-runnable semantics; full
suite 369 green.