ALwrity/docs/flat_file_context/FLAT_FILE_CONTEXT_FRAMEWORK_DESIGN.md

Flat File Context Framework Design (Agent-Optimized)

Purpose

Design a compact, machine-first flat-file framework for ALwrity AI agents.

This framework is optimized for:

• deterministic structure,
• minimal token footprint,
• fast parsing,
• high-signal retrieval,
• robust fallback behavior.

Core Principles

1. Agent-first, not human-first

   • Keys are short and stable.
   • Avoid verbose prose in payloads.
   • Include only fields needed for reasoning and tool actions.

2. Compact + predictable schema

   • Fixed top-level keys in strict order.
   • Canonical value types (no shape drift).
   • Avoid polymorphic fields when possible.

3. Dual-layer context

   • d (full normalized data for deep reasoning).
   • s (summary/high-signal fast path for most agent reads).

4. Fallback-safe design

   • Every context doc includes source + freshness metadata.
   • If missing/stale, consumers fall back to DB then SIF semantic.

5. Multi-tenant isolation

   • Per-user file under workspace/workspace_<safe_user_id>/agent_context/.

---

Canonical Context Envelope (compact)

{
  "v": "1.0",
  "t": "onboarding.step2.website_analysis",
  "u": "<user_id>",
  "ts": "<iso8601>",
  "src": "onboarding_step2",
  "d": {},
  "s": {},
  "m": {
    "db": 0,
    "sb": 0,
    "q": []
  }
}

Field map

• v: schema version
• t: context type
• u: user id
• ts: updated timestamp
• src: source writer
• d: canonical normalized data
• s: high-signal summary for quick agent use
• m: meta (db=data bytes, sb=summary bytes, q=query hints)

---

Agent Readability Best Practices

• Prefer enums/controlled vocab over free text.
• Use compact keys and arrays for repetitive entities.
• Truncate long textual blobs unless explicitly required.
• Keep “quick facts” flattened.
• Separate operational metadata from semantic content.
• Include retrieval hints (q) for consistent query drafting.

---

Write Pipeline Pattern

1. Normalize incoming source payload.
2. Derive compact summary (s) from normalized data.
3. Compute lightweight metadata (m).
4. Atomic write JSON file.
5. Emit writer version + timestamp.

Read Pipeline Pattern

1. Attempt flat-file load.
2. Validate minimum envelope fields (v,t,u,ts,d).
3. Prefer s for quick tasks; use d for deeper reasoning.
4. If invalid/missing/stale: fallback DB -> SIF semantic.

---

Scope Expansion Pattern

Apply same envelope for:

• Step 2: website analysis
• Step 3: research preferences + competitor snapshots
• Step 4: persona profile + platform personas

Only t, d, and s payload contracts should vary.

---

Governance

• Schema changes require version bump (v).
• Backward compatibility policy: readers support N and N-1.
• Drift checks should compare canonical hash/checksum vs DB latest row.

Document Context + End-User Journey Metadata

Each context file should carry explicit machine-oriented document metadata so agents understand what this file is before reading full payloads.

Suggested document_context fields:

• audience: ai_agents
• purpose: fast_context_retrieval
• context_type: step-scoped type identifier
• journey: stage/action/agent expectation
• retrieval_contract: preferred source + fallback order
• context_window_guidance: byte budget and summary-first policy

This block is intentionally compact and deterministic to reduce wasted token usage for agent planning.

Context Window and Length Policy

• Keep combined data + summary under a defined byte budget where practical.
• Enforce summary-first reads in agent consumers.
• Truncate long textual fields in summaries; keep full text only in data when needed.
• Flag oversize docs in metadata so readers can skip low-priority sections.
• Prefer short, stable keys in machine envelopes and avoid natural-language verbosity.

Implemented baseline controls

• Atomic file writes to avoid partial documents.
• Best-effort restricted file permissions (0600).
• Recursive sensitive-key redaction for payload snapshots.
• Payload size budget enforcement with deterministic trimming metadata.
• Internal document linking via related_documents and manifest index.

Security and isolation details: docs/flat_file_context/FLAT_FILE_CONTEXT_SECURITY_AND_ISOLATION.md

Step docs: docs/flat_file_context/STEP2_FLAT_FILE_CONTEXT_DESIGN.md, docs/flat_file_context/STEP3_FLAT_FILE_CONTEXT_DESIGN.md, docs/flat_file_context/STEP4_FLAT_FILE_CONTEXT_DESIGN.md, docs/flat_file_context/STEP5_FLAT_FILE_CONTEXT_DESIGN.md