[verified] Cross-theme surprise normalization + historical factor store (P4 enabler)

A. Cross-theme comparability:
- compute_theme_surprises now weight-normalizes by total |weight| (weighted
  average), so every theme surprise on same [-1,1] scale regardless of factor
  count/weight (retail 0.189->0.145; auto_credit 1.0->0.64).

B. Historical factor store (enables learning macro/demographic factors):
- New factor_history.py: append-only per-factor JSONL, dedupes unchanged
  values, rejects non-finite, records every FACTORS value each scheduler run.
- scheduler.py: jobs carry fetch_module; refresh_all records factor history
  (non-fatal); added bank_npl job.
- GET /api/v1/learning/factors?min_points= reports n_points/learnable per
  factor so users see when P4 learning unlocks (validated query parsing).
- weight_learning: generic learn_factor_series() aggregator (momentum reuses).

Independent review deleg_5dd358e3 passed=true (empty security/logic arrays);
its two robustness suggestions applied (finite guard in record(), clean 400 on
bad min_points). 234 tests pass; Vite build passes.

backend/app/__init__.py

@@ -780,6 +780,37 @@ def create_app(config: dict[str, Any] | None = None) -> Flask:
             return jsonify({"error": str(exc)}), 400
         return jsonify({"factor": learning.to_dict(), "window": {"start": start, "end": end}})
 
+    @app.get("/api/v1/learning/factors")
+    def learning_factors():
+        """P4 dataset-readiness: how many historical points each factor has.
+
+        Factors with >= `min_points` historical observations are learnable
+        (they accumulate automatically each scheduler run, starting now).
+        Macro/demographic factors start at 0 and become learnable over time.
+        """
+        from app import factors as factors_mod
+        from app.factor_history import FactorHistory
+        fh = FactorHistory(Path(__file__).resolve().parents[1] / "data" / "factor_history")
+        try:
+            min_points = max(int(request.args.get("min_points", "12")), 0)
+        except (TypeError, ValueError):
+            return jsonify({"error": "min_points must be a non-negative integer"}), 400
+        rows = []
+        for fkey, fact in factors_mod.FACTORS.items():
+            series = fh.series(fkey)
+            rows.append({
+                "factor_key": fkey,
+                "name_th": fact.get("name_th", fkey),
+                "source": fact.get("source"),
+                "frequency": fact.get("frequency"),
+                "n_points": len(series),
+                "learnable": len(series) >= min_points,
+                "last_value": series[-1]["value"] if series else None,
+                "last_as_of": series[-1].get("as_of", "") if series else "",
+            })
+        rows.sort(key=lambda r: (-r["n_points"], r["factor_key"]))
+        return jsonify({"min_points": min_points, "factors": rows})
+
     @app.route("/api/v1/paper/ledger", methods=["GET", "POST"])
     def paper_ledger():
         current_ledger = app.extensions["paper_ledger"]

backend/app/factor_history.py

@@ -0,0 +1,127 @@
+"""Append-only historical store for FACTORS registry values (P4 enabler).
+
+Background: the dashboard shows each alternative factor's *current* value, but
+factor-weight learning (P4) needs each factor's value *as seen at many past
+points in time* so we can compute forward-return attribution (IC). Retroactive
+reconstruction from a single snapshot is impossible, so this store starts
+recording every factor value from now on, once per scheduled run.
+
+Each factor gets its own JSONL file under
+`backend/data/factor_history/<factor_key>.jsonl`, one JSON object per line:
+    {"ts": "2026-08-27T07:00:00+07:00", "value": 16.2, "as_of": "..."}
+
+Only appends when the value actually changed since the last recorded point, so
+the series stays compact and `series()` returns distinct observations (useful
+for a proper time-series / IC analysis rather than N duplicate rows).
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import os
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Optional
+
+from . import factors as factors_mod
+
+_DEFAULT_DIR = Path(__file__).resolve().parent.parent / "data" / "factor_history"
+
+
+class FactorHistoryError(ValueError):
+    pass
+
+
+class FactorHistory:
+    """Append-only time-series store for factor values (per-factor JSONL)."""
+
+    def __init__(self, history_dir: Path = _DEFAULT_DIR) -> None:
+        self.history_dir = Path(history_dir)
+        self.history_dir.mkdir(parents=True, exist_ok=True)
+
+    def _path(self, factor_key: str) -> Path:
+        safe = "".join(c if (c.isalnum() or c in "._-") else "_" for c in factor_key)
+        return self.history_dir / f"{safe}.jsonl"
+
+    def last_value(self, factor_key: str) -> Optional[float]:
+        """Most recent recorded value, or None if the factor has no history yet."""
+        path = self._path(factor_key)
+        if not path.exists():
+            return None
+        try:
+            with path.open(encoding="utf-8") as fh:
+                last = None
+                for line in fh:
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        rec = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    last = rec
+                return float(last["value"]) if last and last.get("value") is not None else None
+        except OSError:
+            return None
+
+    def record(self, factor_key: str, value: Optional[float],
+               ts: Optional[str] = None, as_of: str = "") -> bool:
+        """Append a point only if it differs from the last recorded value.
+        Returns True if a point was written."""
+        if value is None:
+            return False
+        try:
+            value = float(value)
+        except (TypeError, ValueError):
+            return False
+        if not math.isfinite(value):
+            return False
+        last = self.last_value(factor_key)
+        if last is not None and abs(last - value) < 1e-12:
+            return False
+        ts = ts or datetime.now(timezone.utc).isoformat(timespec="seconds")
+        path = self._path(factor_key)
+        try:
+            with path.open("a", encoding="utf-8") as fh:
+                fh.write(json.dumps({"ts": ts, "value": value, "as_of": as_of},
+                                    ensure_ascii=False) + "\n")
+        except OSError as exc:
+            raise FactorHistoryError(f"cannot write factor history: {exc}") from exc
+        return True
+
+    def series(self, factor_key: str) -> list[dict]:
+        """Full chronological series [{ts, value, as_of}] for a factor."""
+        path = self._path(factor_key)
+        out: list[dict] = []
+        if not path.exists():
+            return out
+        try:
+            with path.open(encoding="utf-8") as fh:
+                for line in fh:
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        rec = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    if isinstance(rec, dict) and "value" in rec:
+                        out.append(rec)
+        except OSError as exc:
+            raise FactorHistoryError(f"cannot read factor history: {exc}") from exc
+        return out
+
+    def record_all(self, fetched: dict, ts: Optional[str] = None) -> dict[str, bool]:
+        """Record the current value of every FACTORS entry from a `fetched`
+        dict (fetch-module -> collector dict). Returns {factor_key: wrote_bool}."""
+        writes: dict[str, bool] = {}
+        for fkey, fact in factors_mod.FACTORS.items():
+            if not fact.get("fetch"):
+                continue
+            val = factors_mod.factor_value(fact, fetched.get(fact.get("fetch")))
+            # best-effort as-of: pull the collector's own period/as_of if present
+            collector = fetched.get(fact.get("fetch")) or {}
+            as_of = str(collector.get("as_of") or collector.get("period") or "")
+            writes[fkey] = self.record(fkey, val, ts=ts, as_of=as_of)
+        return writes

backend/app/scheduler.py

@@ -27,12 +27,14 @@ log = logging.getLogger("set50.scheduler")
 
 # collectors returning a .to_dict()/dict, keyed by cache key
 # (imported lazily to avoid import cycles at module load)
+# `fetch_module` = the FACTORS.fetch module name this job feeds (for history).
 _REFRESH_JOBS: List[dict] = [
-    {"key": "bot_tourism", "label": "ท่องเที่ยว (BOT)", "module": "bot_tourism", "fn": "BotTourismSource().fetch"},
+    {"key": "bot_tourism", "label": "ท่องเที่ยว (BOT)", "module": "bot_tourism", "fn": "BotTourismSource().fetch", "fetch_module": "macro_thai"},
-    {"key": "auto_credit/tourism", "label": "ยอดขายรถ (TradingEconomics)", "module": "auto_credit", "fn": "fetch_auto_credit"},
+    {"key": "auto_credit/tourism", "label": "ยอดขายรถ (TradingEconomics)", "module": "auto_credit", "fn": "fetch_auto_credit", "fetch_module": "auto_credit"},
-    {"key": "auto_npl", "label": "NPL รถยนต์ (BOT)", "module": "auto_npl", "fn": "fetch_auto_npl"},
+    {"key": "auto_npl", "label": "NPL รถยนต์ (BOT)", "module": "auto_npl", "fn": "fetch_auto_npl", "fetch_module": "auto_npl"},
-    {"key": "energy_thai", "label": "โรงกลั่น TOP", "module": "energy_thai", "fn": "fetch_energy_thai"},
+    {"key": "energy_thai", "label": "โรงกลั่น TOP", "module": "energy_thai", "fn": "fetch_energy_thai", "fetch_module": "energy_thai"},
-    {"key": "macro_thai", "label": "ภาพรวมประเทศไทย (BOT)", "module": "macro_thai", "fn": "fetch_macro_thai"},
+    {"key": "macro_thai", "label": "ภาพรวมประเทศไทย (BOT)", "module": "macro_thai", "fn": "fetch_macro_thai", "fetch_module": "macro_thai"},
+    {"key": "bank_npl", "label": "NPL ภาคการเงิน (BOT)", "module": "bank_npl", "fn": "fetch_bank_npl", "fetch_module": "bank_npl"},
 ]
 
 
@@ -71,19 +73,41 @@ class AppDataScheduler:
                 log.exception("set50 scheduler refresh_all failed (will retry)")
 
     def refresh_all(self) -> list[dict]:
-        """Run every collector, warm the daily cache, and snapshot the state."""
+        """Run every collector, warm the daily cache, snapshot the state, and
+        append each factor's value to the historical store (P4 enabler)."""
         results: list[dict] = []
+        fetched_by_module: dict[str, dict] = {}
         for job in _REFRESH_JOBS:
-            results.append(self._run_job(job))
+            res = self._run_job(job)
+            results.append(res)
+            fetch_module = job.get("fetch_module")
+            if res.get("ok") and isinstance(res.get("value"), dict) and fetch_module:
+                fetched_by_module[fetch_module] = res["value"]
+        self._record_history(fetched_by_module)
         self._write_marker(results)
         return results
 
+    def _record_history(self, fetched_by_module: dict[str, dict]) -> None:
+        """Append current factor values to the historical store (append-only).
+
+        `fetched_by_module` maps FACTORS.fetch module name -> collector dict.
+        Runs after each refresh so vintages accumulate; macro/demographic
+        factors become learnable (P4) once they have enough history points.
+        """
+        try:
+            from .factor_history import FactorHistory
+            fh = FactorHistory(self.data_root / "factor_history")
+            fh.record_all(fetched_by_module)
+        except Exception:  # noqa: BLE001 — never let history break the refresh loop
+            log.exception("set50 factor-history record failed (non-fatal)")
+
     def _run_job(self, job: dict) -> dict:
         key = job["key"]
         label = job["label"]
         try:
             value = self.cache.fetch_or_stale(key, self._make_fetcher(job))
-            return {"key": key, "label": label, "ok": True, "at": self._now()}
+            return {"key": key, "label": label, "ok": True, "at": self._now(),
+                    "value": value}
         except Exception as exc:  # noqa: BLE001
             log.warning("set50 refresh failed for %s: %s", key, exc)
             return {"key": key, "label": label, "ok": False, "error": str(exc), "at": self._now()}

backend/app/themes.py

@@ -217,10 +217,13 @@ def compute_theme_surprises(fetched: dict, tourism_surprise: Optional[float] = N
     `fetched` maps fetch-module name -> collector dict (e.g.
     {"macro_thai": {...}, "auto_credit": {...}, "energy_thai": {...}}).
 
-    For each theme in `THEMES`, the surprise is the weighted blend of its
+    For each theme in `THEMES`, the surprise is the **weighted average** of its
     declared FACTORS (each normalized by its own center/span/sign and extracted
-    via `factor_value`). This replaces the old hand-written per-theme blocks:
+    via `factor_value`), normalised by the total absolute weight of the factors
-    editing `THEMES`/`FACTORS` weights now actually changes the score.
+    that actually contributed. Normalising by |weight| keeps every theme's
+    surprise on the same [-1, 1] scale regardless of how many (or how heavy)
+    its factors are, so a surprise of +0.2 means the same thing for retail and
+    for banks (cross-theme comparable — important for P4 weight learning).
 
     `tourism_surprise` (optional) overrides the tourism theme so the richer
     bot-tourism observation z-score can win when available; otherwise tourism
@@ -230,8 +233,8 @@ def compute_theme_surprises(fetched: dict, tourism_surprise: Optional[float] = N
 
     out: dict[str, Optional[float]] = {}
     for tid, tdef in THEMES.items():
-        blended = 0.0
+        weighted = 0.0
-        n = 0
+        w_sum = 0.0
         for ref in tdef.get("factors", []):
             fkey = ref.get("key")
             fact = factors_mod.FACTORS.get(fkey)
@@ -245,12 +248,13 @@ def compute_theme_surprises(fetched: dict, tourism_surprise: Optional[float] = N
                                          span=fact.get("span", 10.0))
             if norm is None:
                 continue
-            blended += float(ref.get("weight", 1.0)) * norm
+            w = float(ref.get("weight", 1.0))
-            n += 1
+            weighted += w * norm
-        if n == 0:
+            w_sum += abs(w)
+        if w_sum == 0:
             out[tid] = None
         else:
-            s = max(-1.0, min(1.0, blended))
+            s = max(-1.0, min(1.0, weighted / w_sum))
             out[tid] = round(s, 3)
 
     # tourism override: prefer the observation-derived surprise when provided.

backend/app/weight_learning.py

@@ -146,6 +146,26 @@ def _bar_date(s: Optional[str]) -> dt.date:
     return dt.date.fromisoformat(str(s)[:10])
 
 
+# ---------------------------------------------------------------------------
+# Generic factor learning from a historical cross-sectional series (P4)
+# ---------------------------------------------------------------------------
+def learn_factor_series(period_ics: list[float]) -> FactorLearning:
+    """Aggregate a list of per-period cross-sectional ICs into a report.
+
+    `period_ics` is one IC per period (e.g. one per month). Used by factor
+    learners that already built the per-period IC; `learn_momentum` is a
+    concrete instance that builds period_ics from the price archive.
+    """
+    res = FactorLearning(factor_key="custom")
+    res.n_periods = len(period_ics)
+    if period_ics:
+        res.ic_mean = round(statistics.fmean(period_ics), 4)
+        res.ic_std = round(statistics.pstdev(period_ics), 4)
+        res.ic_tstat = round(res.ic_mean / (res.ic_std / math.sqrt(len(period_ics))), 3) \
+            if res.ic_std else None
+    return res
+
+
 # ---------------------------------------------------------------------------
 # Momentum factor learning (real PIT demo)
 # ---------------------------------------------------------------------------
@@ -170,13 +190,8 @@ def learn_momentum(series: dict, symbols: list[str], start: str, end: str,
             ic_series.append(ic)
         cur += dt.timedelta(days=step_days)
 
-    res = FactorLearning(factor_key="momentum_12_1")
+    res = learn_factor_series(ic_series)
-    res.n_periods = len(ic_series)
+    res.factor_key = "momentum_12_1"
-    if ic_series:
-        res.ic_mean = round(statistics.fmean(ic_series), 4)
-        res.ic_std = round(statistics.pstdev(ic_series), 4)
-        res.ic_tstat = round(res.ic_mean / (res.ic_std / math.sqrt(len(ic_series))), 3) \
-            if res.ic_std else None
     return res
 
 

backend/tests/test_factor_history.py

@@ -0,0 +1,78 @@
+"""Tests for the historical factor store (P4 enabler)."""
+
+from __future__ import annotations
+
+import json
+import tempfile
+import unittest
+from pathlib import Path
+
+from app.factor_history import FactorHistory, FactorHistoryError
+
+
+class FactorHistoryTest(unittest.TestCase):
+    def setUp(self):
+        self._tmp = tempfile.TemporaryDirectory()
+        self.dir = Path(self._tmp.name)
+        self.fh = FactorHistory(self.dir)
+
+    def tearDown(self):
+        self._tmp.cleanup()
+
+    def test_record_and_series_roundtrip(self):
+        self.assertTrue(self.fh.record("macro_consumption", 4.9, ts="2026-01-01T00:00:00+00:00"))
+        self.assertTrue(self.fh.record("macro_consumption", 5.2, ts="2026-02-01T00:00:00+00:00"))
+        series = self.fh.series("macro_consumption")
+        self.assertEqual(len(series), 2)
+        self.assertEqual(series[0]["value"], 4.9)
+        self.assertEqual(series[1]["value"], 5.2)
+
+    def test_duplicate_value_not_rewritten(self):
+        self.assertTrue(self.fh.record("f", 1.0, ts="2026-01-01T00:00:00+00:00"))
+        self.assertFalse(self.fh.record("f", 1.0, ts="2026-02-01T00:00:00+00:00"))
+        self.assertEqual(len(self.fh.series("f")), 1)
+
+    def test_none_value_not_recorded(self):
+        self.assertFalse(self.fh.record("f", None))
+        self.assertEqual(len(self.fh.series("f")), 0)
+
+    def test_non_finite_not_recorded(self):
+        self.assertFalse(self.fh.record("f", float("nan")))
+        self.assertFalse(self.fh.record("f", float("inf")))
+        self.assertEqual(len(self.fh.series("f")), 0)
+
+    def test_last_value(self):
+        self.fh.record("f", 3.0)
+        self.fh.record("f", 4.0)
+        self.assertEqual(self.fh.last_value("f"), 4.0)
+        self.assertIsNone(self.fh.last_value("missing"))
+
+
+class RecordAllTest(unittest.TestCase):
+    def test_record_all_uses_fetched_dicts(self):
+        import tempfile
+        from pathlib import Path
+        with tempfile.TemporaryDirectory() as td:
+            fh = FactorHistory(Path(td))
+            fetched = {
+                "macro_thai": {
+                    "private_consumption_yoy": 4.9,
+                    "private_investment_yoy": 18.1,
+                    "headline_inflation_yoy": 1.95,
+                    "manufacturing_yoy": -3.1,
+                    "tourists_ytd_mn": 16.2,
+                },
+                "auto_credit": {"new_car_sales_yoy": 20.07,
+                                "vehicle_production": 117383.0, "auto_exports": 81526.0},
+                "auto_npl": {"pct_of_npls": 3.95},
+                "bank_npl": {"pct_of_npls": 1.07},
+            }
+            writes = fh.record_all(fetched)
+            # Every registered factor with a fetch module present gets a write.
+            self.assertGreaterEqual(len([w for w in writes.values() if w]), 5)
+            # macro_consumption should have a recorded value 4.9 norm path.
+            self.assertEqual(fh.last_value("macro_consumption"), 4.9)
+
+
+if __name__ == "__main__":
+    unittest.main()

backend/tests/test_themes.py

@@ -143,12 +143,13 @@ class RegistryDrivenSurpriseTest(unittest.TestCase):
     def test_retail_driven_by_registry_weights(self):
         from app import themes
         s = themes.compute_theme_surprises(self._fetched())
-        # registry retail = consumption*1.0 + inflation*(-0.3):
+        # registry retail = weighted average over |weights| of
-        #   cons=4.9 -> (4.9-3)/10=0.19 ; infl=1.95, sign -1 -> -(1.95-2)/10=0.005*0.3? no:
+        #   consumption*1.0 + inflation*(-0.3):
-        #   inflation normalized = -(1.95-2)/10 = +0.005, weight -0.3 -> -0.0015
+        #   cons=4.9 -> (4.9-3)/10=0.19 (w=1.0) ; inflation sign -1 ->
-        #   retail ≈ 0.19 - 0.0015 ≈ 0.189
+        #   -(1.95-2)/10=+0.005 (w=-0.3) -> weighted = 0.19 - 0.0015 = 0.1885
+        #   surprise = 0.1885 / (1.0 + 0.3) = 0.145
         self.assertIsNotNone(s["retail"])
-        self.assertAlmostEqual(s["retail"], 0.189, places=3)
+        self.assertAlmostEqual(s["retail"], 0.145, places=3)
 
     def test_changing_factor_weight_changes_output(self):
         """The defining property of P0-B: the registry is not decorative."""

backend/tests/test_weight_learning.py

@@ -94,5 +94,19 @@ class AddMonthsTest(unittest.TestCase):
         self.assertEqual(wl._add_months(dt.date(2024, 1, 31), 1), dt.date(2024, 2, 29))  # leap
 
 
+class LearnFactorSeriesTest(unittest.TestCase):
+    def test_aggregates_ics(self):
+        res = wl.learn_factor_series([0.1, 0.2, 0.3, 0.4])
+        self.assertEqual(res.n_periods, 4)
+        self.assertIsNotNone(res.ic_mean)
+        self.assertTrue(res.ic_mean is not None and abs(res.ic_mean - 0.25) < 1e-6)
+        self.assertIsNotNone(res.ic_tstat)
+
+    def test_empty_is_blocked_like(self):
+        res = wl.learn_factor_series([])
+        self.assertEqual(res.n_periods, 0)
+        self.assertIsNone(res.ic_mean)
+
+
 if __name__ == "__main__":
     unittest.main()

docs/engineering-log.md

@@ -94,3 +94,5 @@
 - Fresh final M2.9 independent review `deleg_e5407553` returned a schema-valid `passed=true` verdict with empty `security_concerns` and `logic_errors`. It recorded three non-blocking test gaps and three deferred suggestions covering concurrent persistence, same-raw normalized-content mismatch cases, predecessor-reference mismatch cases, malformed manifest fixtures, and focused helper/UI coverage. No commit or push has been made.
 - M2.9 observation-integrity follow-up added RED/GREEN regressions for same-raw/different-normalized content, malformed snapshot manifest entries, observations before immutable first capture, and duplicate equal-time observations for one snapshot. The focused price suite and full backend suite pass at 125 tests; independent review `deleg_0e2282c8` returned schema-valid `passed=true` with empty `security_concerns` and `logic_errors` (suggestion: retain the new regression coverage). Process-level locking and focused helper/UI coverage remain deferred.
 - Audit + fix pass (2026-08-26, plan at `docs/audit-and-plan-2026-08-26.md`): triple-confirmed the declarative `FACTORS`/`THEMES` framework is by-passed by hand-written scoring in `dashboard._theme_surprises`, and that `/api/v1/simulation` recomputed a divergent 3-theme path. Fixed P1 (simulation reuses the canonical board via `default_scores` — live check: sim top pick PTT == top board combined 1.600), P2 (dashboard now emits `source_summary{factor_keys:11, rows:6}`; frontend shows "N ปัจจัย · M แหล่ง"), and P5-partial (removed dead `list_themes`/`Theme`/`build_theme_scores`/`_map_index` + the tests that locked them; added `backend/tests/conftest.py` so pytest needs no `PYTHONPATH`). Deferred P0 (registry-driven re-baseline) and P3/P4 (point-in-time backtest + factor-weight learning) pending explicit scope/baseline sign-off. Full backend suite: **203 tests pass**; Vite build passes. This work is own-engine review gated before commit.
+- P0-B + P3 + P4 (commit `8db3d48`, reviews `deleg_fe6f45cd` + `deleg_718218f8` both `passed=true`): the declarative FACTORS/THEMES registry is now the single source of truth (`compute_theme_surprises` reads registry; hand-written per-theme blocks removed; FACTORS carries center/span normalization spec; bank NPL wired into banks). P3 rebuilt `run_backtest` as a real multi-rebalance PIT engine (`leakage_guard`, `momentum_at` true 12-1). P4 added factor-weight learning (Spearman IC -> `apply_weight_update`) + `/api/v1/learning/momentum`. Live result: momentum IC=0.012, t=0.132 over 22 periods (no reliable predictive power in this SET50 window). 226 tests pass.
+- Follow-up (A+B): theme surprises now weight-normalized by total |weight| so cross-theme magnitudes are comparable (retail 0.189->0.145). Added append-only `FactorHistory` store (`data/factor_history/<key>.jsonl`) that records every FACTORS value each scheduler run, wired into `refresh_all` (non-fatal), plus `/api/v1/learning/factors` readiness endpoint. Macro/demographic factors start at n=1 and become learnable (P4) as history accumulates. 233 tests pass.