set50-system/docs/engineering-log/2026-08-24-forward-price-observation-modes.md

M2.9 — Forward price observations and explicit research modes

Plan status

• Status: complete/blocked
• Complete: the system can record forward observations, detect local revisions, and run an explicitly labelled exploratory descriptive study.
• Blocked: validated historical PIT backtest remains closed until a provider passes pit-daily-v1 and BOT has at least 12 independent releases.

Decision

The strict PIT requirement is a gate for validated historical backtests, not a launch blocker for exploratory or forward paper work. Revised vendor history remains usable only when the result is labelled non_pit_descriptive_only.

Root cause addressed

The price store already preserved raw payloads and immutable normalized snapshots, but repeated retrievals had no durable observation history. The research runner also treated the PIT requirement as a universal gate, which prevented descriptive research from using an available revised history.

Changes

• backend/app/prices.py
  • Added an optional, backward-compatible manifest observations list.
  • Records one observation per distinct retrieval identity with source/period scope, retrieval time, raw hash, prior snapshot ID/hash, revision status, and structured diff paths.
  • Preserves the original raw and normalized snapshot on unchanged retrievals; an actual content change creates a new snapshot/vintage instead of overwriting the prior one.
  • Tracks first_seen_at, last_seen_at, and observation_count on snapshot manifest entries.
  • Binds observation IDs to their content and rejects tampered or unknown-snapshot observation records at load time.
  • Keeps revised Yahoo history at quality=revised_vendor_history and point_in_time=false.
• backend/app/research.py
  • Added explicit mode=validated|exploratory.
  • validated remains fail-closed on missing PIT archive evidence and requires per-bar known-at validation.
  • exploratory may proceed with revised history only as result_scope=non_pit_descriptive_only; it disables PIT known-at enforcement and emits limitations.
  • Runtime observation timestamps are excluded from the research input fingerprint, so an unchanged feed reuses the same frozen research run.
• backend/app/__init__.py
  • Price health now exposes observation count, last observation time, and revision status.
  • Added GET /api/v1/prices/observations for the local audit trail.
  • The research endpoint accepts the explicit mode and defaults exploratory requests to one event unless overridden; validated requests retain the 12-event default.
• frontend/src/App.vue
  • The research action explicitly requests exploratory mode with min_events=1.
  • UI copy and result metadata distinguish a descriptive study from a validated backtest.
• Tests cover unchanged retrievals, revision diffs, immutable prior snapshots, observation tampering, exploratory execution, validated regression behavior, API mode validation, and stable run reuse.

Verification evidence

• PYTHONPATH=backend .venv/bin/python -W error -m unittest discover -s backend/tests -v
  • Baseline before this slice: 93 tests passed.
  • First full run after the slice: 98 tests passed.
  • Focused run after observation-integrity hardening: 67 tests passed.
  • Final full run: 121 tests passed with -W error.
• PYTHONPATH=backend .venv/bin/python -m compileall -q backend passed.
• npm run build from frontend/ passed.
• npm audit --omit=dev --audit-level=high reported 0 vulnerabilities.
• git diff --check passed.
• Static scan over the changed code found zero hardcoded-secret assignments, shell execution, eval/exec, pickle, or SQL-f-string patterns.
• Final bounded independent reviewer deleg_e5407553: schema-valid passed=true; security_concerns=[] and logic_errors=[]. Non-blocking gaps: concurrent persistence/atomic manifest-update coverage, same-raw normalized-content mismatch coverage, and invalid/missing/mismatched predecessor-reference coverage. Suggestions are deferred because the store remains single-process and validated PIT deployment is still blocked.

Risks and limits

• Observation history proves only what this system observed after capture began. It does not prove provider release time, historical knowledge state, or absence of revisions before first capture.
• The local filesystem store remains single-process; concurrent-writer safety and shared database migration are deferred.
• No provider has passed the complete PIT evidence packet. Do not set point_in_time=true for revised vendor history.
• No live execution, webhook, broker, MT5, or external order path was added.

Exact next action

Keep collecting independent BOT releases until 12 genuinely distinct (source_id, published_at) releases exist. In parallel, request one complete market-data provider evidence packet with SET coverage, provider release identity/time, explicit known-at semantics, correction/revision replay, immutable raw/archive/manifest evidence, and timezone/session rules. Only then implement or promote a PIT adapter.

NEVER include API keys, tokens, passwords, secrets, credentials, or connection strings in the summary — replace any that appear with [REDACTED].