feat(backtest): runnable with partial factor coverage + oldest-history default (owner rule)

Owner's rule: backtest must run as soon as there's enough data to estimate an
investment — it must NOT be blocked just because some sources lack deep PIT
history. Scoring is deliberately flexible (a theme uses whatever subset of
factors was knowable that day).

- backtest_readiness: readiness = usable window (price + Siamchart + >=1 factor),
  not all-factors-present. Missing factors still reported (transparency) but no
  longer block the run. recommended_start = oldest executable price (oldest
  history the system holds); recommended_end = last complete trading day.
- pit_scorer.theme_surprise_report: flexible — skips factors not released by
  as_of; blocked only when NO factor has a value. pit_meta.partial_pit reflects
  themes scored from a partial factor subset.
- Verified end-to-end: readiness ready=true (recommended 2024-01-03 -> 2026-08-29);
  POST /api/v1/backtest/run default window returns 201 full result (1M -> final
  equity 1,117,243.95), no 400 from missing factors.
- test_backtest_readiness updated to earliest-runnable semantics; full suite 369 green.

backend/app/backtest_readiness.py

@@ -196,11 +196,18 @@ def evaluate_readiness(
         start/end: optional explicit window; if given, readiness is evaluated
             only against ``[start, end]``.
 
-    "Ready" requires, within the window:
+    "Ready" requires only a *usable* window — NOT that every factor is present
-      1. every registry factor has a released value by the start date;
+    (the owner's rule: scoring is deliberately flexible, a theme scores from
+    whatever subset of factors was knowable that day, so a backtest may run as
+    soon as there is enough data to estimate, even if some sources are missing
+    or don't provide history):
+      1. at least one registry factor has a released value by the start date;
       2. a Siamchart snapshot retrieved no later than the start date;
       3. at least one executable price in the window for every required symbol.
-    Missing inputs are reported explicitly rather than silently skipped.
+    Missing inputs are still reported for transparency, but a partial window is
+    runnable rather than rejected. ``recommended_start`` is the EARLIEST date at
+    which usable data exists (the oldest history the system holds), and
+    ``recommended_end`` defaults to the last complete trading day (yesterday).
     """
     miss: list[str] = []
     cov: list[DataCoverage] = []
@@ -252,46 +259,77 @@ def evaluate_readiness(
     if not scv.available:
         miss.append("siamchart")
 
-    # derive recommended start = the latest earliest-available date among the
+    # recommended start = the EARLIEST date a run is genuinely executable from —
-    # inputs (the point at which *all* of them are simultaneously available).
+    # i.e. the oldest price data (you must be able to value the holdings). Factors
-    candidates = [
+    # and Siamchart are flexible (they may start later; scoring just uses whatever
-        d for d in (fac_earliest, sc_earliest, pc.earliest)
+    # subset was known by each day), but price is the hard floor. So the default
-        if d is not None
+    # start is the oldest available executable price, per the owner's wish to
-    ]
+    # backtest over the oldest history the system holds.
     recommended_end = _date_from_ts(end) or today
     if start is not None:
         recommended_start = _parse_date(start)
-    elif candidates:
+    elif pc.available and pc.earliest is not None:
-        recommended_start = max(candidates)
+        recommended_start = pc.earliest
     else:
-        recommended_start = None
+        candidates = [d for d in (fac_earliest, sc_earliest) if d is not None]
+        recommended_start = min(candidates) if candidates else None
 
     # cap recommended start so it never exceeds the end
     if recommended_start and recommended_end and recommended_start > recommended_end:
         recommended_start = recommended_end
 
-    # explicit window requested: readiness is whether the window is covered
+    # ---- usable-window gate (NOT all-factors-present) ---------------------
+    # Scoring is flexible: a theme scores from whatever factors were known by
+    # as_of, so the backtest is runnable once we have a price series, a Siamchart
+    # snapshot, and *at least one* released factor. Missing individual factors
+    # are still reported in `missing` for transparency but do NOT block the run.
+    hard_block = (
+        not pc.available
+        or not scv.available
+        or fac_earliest is None  # no factor has any released value at all
+    )
     if start is not None:
         s = _parse_date(start)
         e_ = _parse_date(end) if end else today
         # price must be executable AT the start: an explicit start that predates
-        # all usable price history would otherwise manufacture a false PIT
+        # all usable price history would otherwise manufacture a false PIT window.
-        # window (factor + Siamchart are already checked at start above).
         if pc.available and pc.earliest is not None and s < pc.earliest:
+            hard_block = True
             miss.append("price")
-        ready = (not miss) and s <= e_
+        # an explicit start with no factor released by it is unusable
+        if fac_earliest is None or s < fac_earliest:
+            hard_block = True
+            if "factor" not in miss:
+                miss.append("factor")
+        ready = (not hard_block) and s <= e_
     else:
-        ready = (not miss) and bool(recommended_start)
+        ready = (not hard_block) and bool(recommended_start)
+    if not scv.available and "siamchart" not in miss:
+        miss.append("siamchart")
+    if not pc.available and "price" not in miss:
+        miss.append("price")
+    if fac_earliest is None and "factor" not in miss:
+        miss.append("factor")
 
     reason = ""
     rs = recommended_start.isoformat() if recommended_start else None
     re_iso = recommended_end.isoformat()
     if ready:
+        fac_missing_n = fac_cov[0].missing_count if (fac_cov and fac_cov[0].source == "factor") else 0
         reason = (
-            f"strict PIT coverage from {rs} to {re_iso}"
+            f"usable PIT coverage from {rs} to {re_iso}"
+            + (f" ({fac_missing_n} factors without history)"
+               if fac_missing_n else "")
         )
-    elif miss:
+    elif hard_block:
-        reason = "missing inputs: " + ", ".join(sorted(set(miss)))
+        blockers = []
+        if not pc.available:
+            blockers.append("price")
+        if not scv.available:
+            blockers.append("siamchart")
+        if fac_earliest is None:
+            blockers.append("no factor has any released value")
+        reason = "missing hard inputs: " + ", ".join(blockers)
     else:
         reason = "no usable PIT-ready window"
 

backend/app/pit_scorer.py

@@ -123,19 +123,29 @@ class PitScoreProvider:
 
     # -- theme PIT score -------------------------------------------------
     def theme_surprise_report(self, theme_key: str, as_of: str) -> dict[str, Any]:
-        """Weighted-average theme surprise from PIT factor values only.
+        """Weighted-average theme surprise from the factors that ARE released by
+        ``as_of`` — flexible, not all-or-nothing.
 
-        Returns {theme, surprise, blocked, partial_pit}. ``blocked`` is True
+        ``theme_surprise_report`` intentionally matches the live dashboard's
-        when at least one of the theme's factors had no value released by
+        ``compute_theme_surprises`` semantics: any factor with no PIT value by
-        ``as_of`` — a caller must never treat a blocked theme as PIT.
+        ``as_of`` is simply omitted from the weighted average, so a theme scores
+        from whatever subset of its factors was genuinely knowable that day and
+        is never blocked by a single missing source. ``blocked`` is only True
+        when NO factor in the theme had a released value by ``as_of`` (nothing
+        to compute). ``used``/``total`` report how many factors contributed, so
+        callers can judge how thin the estimate is.
+
+        PIT integrity is preserved: we only ever use values released <= ``as_of``
+        (``factor_at``), never future knowledge.
         """
         factors = self.theme_factor_map.get(theme_key, [])
         if not factors:
-            return {"theme": theme_key, "surprise": None, "blocked": True, "partial_pit": True}
+            return {"theme": theme_key, "surprise": None, "blocked": True,
+                    "partial_pit": True, "used": 0, "total": 0}
         from . import factors as factors_mod
         weighted = 0.0
         w_sum = 0.0
-        blocked = False
+        used = 0
         for spec in factors:
             fkey = spec.get("key")
             if not isinstance(fkey, str) or not fkey:
@@ -145,8 +155,7 @@ class PitScoreProvider:
                 continue
             value = self.factor_at(fkey, as_of)
             if value is None:
-                blocked = True
+                continue  # not released yet -> flexibly skip (no blanket block)
-                continue
             norm = factors_mod.normalize(
                 value, sign=fact.get("sign", 1),
                 center=fact.get("center", 0.0), span=fact.get("span", 10.0),
@@ -155,17 +164,21 @@ class PitScoreProvider:
                 continue
             weighted += spec.get("weight", 1.0) * norm
             w_sum += abs(spec.get("weight", 1.0))
-        surprise: Optional[float]
+            used += 1
+        total = len([f for f in factors if isinstance(f.get("key"), str) and f.get("key")])
         if w_sum == 0:
             surprise = None
-            blocked = True
+            blocked = True  # nothing had a released value -> cannot score
         else:
             surprise = round(min(1.0, max(-1.0, weighted / w_sum)), 3)
+            blocked = False
         return {
             "theme": theme_key,
             "surprise": surprise,
             "blocked": blocked,
-            "partial_pit": blocked,
+            "partial_pit": used < total,
+            "used": used,
+            "total": total,
         }
 
     # -- per-symbol combined board as of a date ----------------------------
@@ -189,6 +202,7 @@ class PitScoreProvider:
 
         theme_scores: dict[str, dict[str, float]] = {}
         blocked_themes: list[str] = []
+        partial_themes: list[str] = []
         fv = self.siamchart_factor_view(as_of)
         # reverse-map symbol -> fundamental for quality_within_theme's factor view
         fv_for_quality = {
@@ -203,6 +217,8 @@ class PitScoreProvider:
                 blocked_themes.append(tid)
                 theme_scores[tid] = {}
                 continue
+            if rep["partial_pit"]:
+                partial_themes.append(tid)
             q = {}
             for sym in themes_mod.THEME_SYMBOLS.get(tid, set()):
                 if sym not in fv:
@@ -212,7 +228,7 @@ class PitScoreProvider:
             theme_scores[tid] = q
 
         combined = themes_mod.combine_score(list(theme_scores.values()), siamchart_score)
-        is_full_pit = not blocked_themes
+        is_full_pit = not blocked_themes and not partial_themes
         # fundamental is PIT only when a siamchart vintage store is wired;
         # otherwise the current snapshot makes the overall result partial.
         fundamental_pit = self.siamchart_store is not None
@@ -227,11 +243,17 @@ class PitScoreProvider:
                 "dividend_yield": fm.get("dividend_yield") or 0.0,
                 "pit_meta": {
                     "pit": is_full_pit and fundamental_pit,
-                    "partial_pit": not fundamental_pit,
+                    "partial_pit": (not fundamental_pit) or bool(partial_themes),
                     "blocked_theme": blocked_themes,
-                    "note": ("theme surprises PIT + siamchart fundamental PIT (store)"
+                    "partial_themes": partial_themes,
-                             if fundamental_pit else
+                    "note": (
-                             "theme surprises PIT; siamchart fundamental partial (current snapshot)"),
+                        "theme surprises PIT + siamchart fundamental PIT (store)"
+                        if (fundamental_pit and not partial_themes) else
+                        ("theme surprises PIT; some themes partial (missing factors); "
+                         "siamchart fundamental PIT (store)"
+                         if fundamental_pit else
+                         "theme surprises PIT; siamchart fundamental partial (current snapshot)")
+                    ),
                 },
             }
         # fall back to the current board for any symbol the PIT path could not

backend/tests/test_backtest_readiness.py

@@ -123,7 +123,9 @@ class NoSiamchartSnapshotTest(unittest.TestCase):
 
 
 class RecommendedStartTest(unittest.TestCase):
-    def test_start_is_latest_of_first_ready_dates(self):
+    def test_start_is_oldest_runnable_price(self):
+        # Owner rule: backtest default start = oldest history the system holds
+        # (the oldest executable price), NOT the date every input co-exists.
         # factors ready 2025-01-01, siamchart ready 2025-06-01, price from 2024
         store = full_factor_store("2025-01-01")
         sc = FakeSiamchartStore(["2025-06-01T09:00:00+07:00"])
@@ -132,7 +134,9 @@ class RecommendedStartTest(unittest.TestCase):
             factor_store=store, siamchart_store=sc, price_series=series
         )
         self.assertTrue(res.ready)
-        self.assertEqual(res.recommended_start, "2025-06-01")
+        # oldest runnable history = oldest price (2024-01-01), even though
+        # factors/siamchart start later (flexible scoring uses what's known).
+        self.assertEqual(res.recommended_start, "2024-01-01")
         self.assertEqual(res.recommended_end, today_iso())
 
     def test_start_limited_by_price_when_price_latest(self):

docs/engineering-log/2026-08-29-data-source-expansion-and-ui-fix.md

@@ -88,3 +88,24 @@ needs-deeper-capture and left as a decision for the owner.
 - Note: a **concurrent process** also landed `thai_trade.py` (external-sector
   exports/imports/current-account) and external_* factors mid-session; its 3
   initially-broken tests were fixed to reach the 360-green baseline here.
+
+## Flexible PIT backtest (owner rule — 2026-08-30)
+Owner clarified the backtest contract: it must run as soon as there is enough
+data to estimate an investment, NOT only when EVERY factor has PIT history.
+Some sources may not provide deep history — scoring is deliberately flexible
+(a theme scores from whatever subset of factors was knowable that day).
+
+Implemented (commit 2026-08-30):
+- `backtest_readiness.evaluate_readiness`: readiness is now a *usable* window
+  (price + Siamchart + >=1 released factor), not all-factors-present. Missing
+  factors are still reported in `missing` for transparency but no longer block.
+  `recommended_start` = oldest executable price (the oldest history held);
+  `recommended_end` = last complete trading day (yesterday Bangkok).
+- `pit_scorer.theme_surprise_report`: flexible — skips any factor with no release
+  by `as_of` (no blanket theme block); `blocked` only when NO factor has a value.
+  `pit_meta.partial_pit` reflects themes that scored from a partial subset.
+- Verified: `/api/v1/backtest/readiness` → ready=true, recommended 2024-01-03 →
+  2026-08-29. `POST /api/v1/backtest/run` (default window) → 201 full result
+  (final_equity 1,117,243.95 on 1M), no 400 from missing factors.
+- Tests: test_backtest_readiness updated to earliest-runnable semantics; full
+  suite 369 green.