91beb8c2dc
Production correct-credential login returned auth_unavailable 503 because Flask's SECRET_KEY was unset: wrong-password probes stopped at 401 before CSRF token issuance, while valid credentials reached _csrf_serializer() and crashed. App factory now rejects absent/short (<32 char) SECRET_KEY at startup, and docker_entrypoint.sh fails fast before migration/services. Bootstrap no longer passes ADMIN_PASSWORD in process arguments; env-only. Tests: app-factory + entrypoint regression (5 focused passed), full backend suite 204 passed. Independent review PASS.
94 lines
17 KiB
Markdown
94 lines
17 KiB
Markdown
# HANDOFF — MiroFish SaaS Migration
|
||
|
||
**Repo:** `/Users/kunthawat/Gitea/MiroFish`
|
||
**Branch:** `feat/saas-foundation`
|
||
**Base:** `89d04e7`
|
||
**Status:** SaaS foundation in progress; not production-ready; no commit/push yet
|
||
|
||
## Current outcome
|
||
|
||
The repository now has a tested identity/authentication foundation, tenant/owner guards around the main graph, simulation, report, and task lookup paths, durable operations schema/jobs/idempotency records, and a bounded local graph → profile → simulation → report persistence path. The original Zep runtime remains the default compatibility backend and legacy file-backed product persistence remains, so this is not yet a completed SaaS migration or production release.
|
||
|
||
## Delivered
|
||
|
||
- Thai/English locale policy with Thai default; legacy `zh` values normalize to Thai.
|
||
- Frontend API error sanitization and backend structured error envelopes.
|
||
- No full JSON request-body logging.
|
||
- SQLAlchemy models and Alembic migrations for organizations, users, memberships, and revocable sessions.
|
||
- Argon2 password hashing.
|
||
- HttpOnly session cookie plus signed CSRF cookie/header validation.
|
||
- Explicit credentialed CORS allowlist.
|
||
- Three roles only: `super_admin`, `admin`, `user`.
|
||
- `/api/auth/login`, `/api/auth/me`, `/api/auth/logout`.
|
||
- Production correct-credential login 503 (`auth_unavailable`) was traced to a missing `SECRET_KEY`: wrong-password probes returned 401 before CSRF issuance, while successful credentials reached `_csrf_serializer()` and failed. App factory + Docker entrypoint now fail fast when `SECRET_KEY` is absent or shorter than 32 chars; bootstrap no longer exposes `ADMIN_PASSWORD` in process arguments. Backend **204 passed**; EasyPanel must set a persistent random `SECRET_KEY` and `SESSION_COOKIE_SECURE=true` before restart.
|
||
- `/api/admin/users` list/create with role escalation prevention and response redaction.
|
||
- Project `organization_id`/`owner_user_id` metadata and scoped project reads/lists.
|
||
- Blueprint-wide auth and CSRF checks for simulation/report routes.
|
||
- Fail-closed scope checks for project, graph, simulation, report, and task IDs in path/query/body.
|
||
- Cross-tenant regression tests for project/simulation/report/task access.
|
||
- Frontend auth store using cookie sessions, `/login` view, protected workspace router guard, and localized Thai/English login copy.
|
||
- Production `/login` white screen root cause fixed locally (2026-09-01): raw `@` in `auth.emailPlaceholder` was invalid vue-i18n linked-message syntax and raised compiler code 10 during render. th/en now use `name{'@'}company.com`; an all-message compiler regression passes. Frontend **11 tests** + production build pass; Chrome rendered DOM contains the login UI. Pending: fresh review, push/redeploy, then live verification.
|
||
- Frontend report-status client corrected to POST `{ task_id, simulation_id }`.
|
||
- Durable local memory tables and a tenant-scoped SQLAlchemy repository.
|
||
- Durable `jobs`, idempotency records, and audit-log schema with migrations `0004_operations` through `0006_job_metadata`; `TaskManager` binds to the current app/session factory, fails closed when an app context has no factory, and has SQL-level tenant/owner filters.
|
||
- SQLite engines created by the shared DB helper enable foreign-key enforcement; regression coverage verifies invalid organization/owner references plus `SET NULL`/`CASCADE` delete behavior.
|
||
- GraphBuilder async adapter now carries organization/session context into durable task metadata.
|
||
- Deterministic local graph → profile → simulation-config → report-tools → persisted-report regression.
|
||
- Strict Pydantic extraction schema, bounded prompts, JSON validation, and deterministic LLM-result ingestion service.
|
||
- Golden parity fixtures and local adapters for entity filtering/context, quick search, panorama temporal inventory, and insight entity/relationship output shapes.
|
||
- Explicit `MEMORY_BACKEND=local` path now serves simulation entity routes and preparation; `SimulationManager.create_entity_reader()` fails closed when local mode has no injected scoped factory; default remains Zep during migration. Fresh bounded reviewer `deleg_029828b6` independently passed this exact reader/fallback slice with no security concerns or logic errors.
|
||
- The bounded local panorama/insight/entity-reader parity slice is covered by RED/GREEN tests; conflicting graph IDs fail closed and local simulation activity updater persists tenant-scoped episodes without constructing a Zep client.
|
||
- API safe-error regression gate covers simulation, graph, report, agent-group, and template boundaries; task/project/report/run state no longer persists raw exception text.
|
||
|
||
## Verification evidence
|
||
|
||
- Backend full suite after the latest app/factory isolation fixes, auxiliary/idempotency hardening, simulation/report/graph service import-boundary fixes, durable product-resource schema/repository, tenant-scoped `ArtifactStore`, durable `JobQueue`/`worker.py`, versioned redacted `PlatformSettings`, durable `RateLimiter`, durable LLM `UsageService`, durable redacted `AuditService`, and durable single-use `PasswordResetService`: **193 passed** with `PYTHONPATH= uv run pytest -q` from `backend` (2026-08-24); twenty-four Alembic deprecation warnings remain.
|
||
- Schema regression: **16 passed**, including Alembic upgrade → downgrade `0003_memory` → re-upgrade, operation-table column/index/FK inspection, SQLite PRAGMA enforcement, invalid org/owner rejection, delete actions, cross-app TaskManager binding checks, and background use without Flask context.
|
||
- Auxiliary API security regression: **8 passed**; every template/agent-group route requires auth, LLM mutations require CSRF + idempotency, safe internal errors are redacted, and both LLM mutations replay completed responses.
|
||
- Idempotency API regression: **5 passed**; same-route replay/conflict semantics remain, same-key cross-route/body replay is rejected, and multipart file content/repeated-file coverage is verified.
|
||
- Local consumer-boundary regression: **9 fresh-import tests passed** (`test_local_service_import.py`); fresh local imports of the updater, `SimulationManager`, `SimulationRunner`, `ReportAgent`, and `GraphBuilderService` do not eagerly load `zep_cloud`, `add_activity_from_dict` uses the shared activity contract without importing `zep_cloud` on the local path, and the reviewer-requested regression guards for explicit Zep-path imports, shared `AgentActivity` identity, and the `graph_builder.Zep` seam were added and pass. The activity-dict boundary regression also passes.
|
||
- Durable product-resource persistence: **21 product-persistence tests passed** (`test_product_persistence.py`). Migration `0007_product_resources` adds tenant/owner-scoped `projects`, `simulations`, and `reports` tables with FK cascade/SET-NULL, indexes, and a migration round-trip + `command.check` (no autogenerate drift). `ProductRepository` provides flush-only tenant-scoped create/get/list for all three with cross-tenant isolation verified. `sync_project`/`sync_simulation`/`sync_report` are idempotent dual-write bridges wired into project-creation, simulation-creation, and report-generation paths; `SimulationState` carries tenant scope and `_save_simulation_state` mirrors every status update into the durable table when a session factory is available, fail-closed when unscoped. `scoped_project` reads the durable table first (mapping back to the legacy `Project` shape) when the local backend is active, then falls back to filesystem, keeping existing routes intact. Read cutover/dual-write are gated to the local backend so the legacy Zep filesystem path is undisturbed.
|
||
- Tenant-scoped artifact store: **12 artifact-store tests passed** (`test_artifact_store.py`). `ArtifactStore` resolves tenant-scoped paths under a configured root, rejects traversal/absolute components (fail-closed), and exposes a store/read/exists/delete interface (`default_artifact_store()` is filesystem-backed; swapable for object storage later).
|
||
- Durable job queue + worker core: **10 tests passed** (`test_job_queue.py` 7, `test_worker.py` 3). `JobQueue` claims/executes/completes/fails durable `jobs` with optional organization scoping, a handler-registry dispatch (raises on unhandled operations for the worker to fail), and no double-claim; `worker.py` `run_once`/`run_loop` provide the portable worker entry point (no broker dependency yet).
|
||
- Versioned, redacted LLM settings: **4 tests passed** (`test_settings_service.py`). Migration `0008_platform_settings` + `SettingsService` store versioned global platform settings; the API key is encrypted with a `SECRET_KEY`-derived Fernet key (never stored/logged in plaintext, never in job snapshots), public reads return a masked value, and `snapshot_for_job()` captures the settings version for reproducible work. `cryptography` added as a direct dependency.
|
||
- Durable rate limiting: **6 tests passed** (`test_rate_limiter.py`). Migration `0009_rate_limit` + `RateLimiter` implement a durable sliding-window counter (survives restarts, per-key, no secrets), and are wired into the `/api/auth/login` route so excessive attempts return 429 (`api.tooManyAttempts` added in `th`/`en`).
|
||
- Durable LLM usage accounting: **4 tests passed** (`test_usage_service.py`). Migration `0010_usage_events` + `UsageService` record per-org/user LLM usage (model, input/output tokens, estimated cost) without storing prompt content or secrets, with org-scoped listing and total-cost totals.
|
||
- Durable redacted audit service: **3 tests passed** (`test_audit_service.py`). `AuditService` records tenant-scoped audit events while stripping sensitive keys (secrets, tokens, api keys, prompts) from details, and is wired into the `/api/auth/login` success path (`auth.login` event).
|
||
- Durable single-use password reset: **6 tests passed** (`test_password_reset.py`). Migration `0011_password_reset_tokens` + `PasswordResetService` issue expiring, single-use, hashed reset tokens (plaintext never stored), bump `auth_version` on consume to invalidate old sessions, and are exposed as `/api/auth/password-reset/request` + `/api/auth/password-reset/confirm` (enrollment-agnostic request prevents account enumeration; also covers invite-pending password setup).
|
||
- Frontend contract tests: **10 passed**; `npm run build`: **PASS**; existing chunk-size/dynamic-import warnings remain.
|
||
- Python `compileall`: **PASS**.
|
||
- Diff security scan: no introduced `shell=True`, `eval/exec`, or pickle loading in backend code; `git diff --check`: **PASS**.
|
||
- `ruff` is not installed in the backend environment; lint is skipped, not treated as a pass.
|
||
- Dev server `/login` smoke response: **PASS**. Real 320×568 and 500×768 browser capture is **not verified** because Chrome requested an OS remote-debugging permission and no permission dialog was clicked.
|
||
|
||
## Important incomplete work
|
||
|
||
1. **Zep replacement is not complete.** Local memory storage/extraction contracts and a bounded graph → profile → simulation → report persistence regression exist, but semantic LLM parity and full consumer cutover are not complete; Zep remains the default compatibility backend.
|
||
2. **Product persistence foundation is in place.** Durable tenant/owner-scoped `projects`/`simulations`/`reports` schema (migration `0007_product_resources`), a scoped `ProductRepository`, dual-write for creation and simulation run status, `scoped_project` read-first cutover (local backend), and a tenant-scoped `ArtifactStore` abstraction are added and tested. Legacy `ProjectManager`/`SimulationManager`/`ReportManager` filesystem paths remain authoritative-by-default for the Zep backend and for artifact payloads; full read cutover for simulation/report and migrating artifact payloads to object storage are deferred to the production storage/worker phase.
|
||
3. **Production worker/queue topology core is in place; broker/storage not decided.** A durable `JobQueue` (claim/dispatch/complete/fail, tenant-scoped) and a portable `worker.py` run-loop entry point exist and are tested, alongside a tenant-scoped `ArtifactStore`. Production decisions remain: choose a broker (Redis/RabbitMQ) or keep polling the durable table, choose object storage, and define the WSGI/worker/supervisord deployment topology. Background-thread execution in routes is not yet moved onto the durable queue.
|
||
4. **Admin user-management + super-admin settings UI base in place.** Backend `GET/POST/PATCH /api/admin/users` (role/status policy-guarded, status super-admin only) and `GET/PUT /api/admin/settings` (super-admin only, secret encrypted+masked via `SettingsService`). Frontend `AdminView.vue` (user mgmt) and `SettingsView.vue` (LLM settings form) are routed at `/admin` and `/admin/settings` with admin/super-admin role guards and th/en i18n. Invite self-setup UX, a connection-test endpoint, and full i18n/mobile review remain.
|
||
5. **LLM settings storage base is in place; admin/bootstrap UI not built.** A versioned, encrypted `PlatformSettings` store (`SettingsService` + migration `0008`) exists and is tested. Super-admin settings UI, provider preset validation, and connection-test endpoints are not yet built; active runtime still reads environment variables.
|
||
6. **Frontend active UI i18n gates pass; legacy `Process.vue` retired.** Frontend contract tests **10 passed** and the production build contains no CJK code points. The unused legacy `Process.vue` (Chinese copy) was retired — the router's `name: 'Process'` uses `MainView.vue`. Real mobile 320×568/500×768 visual capture requires a full-stack + authenticated browser session and is not yet verified; a standalone `vite preview` of the SPA renders a blank mount (needs backend bootstrap), which is not treated as a pass.
|
||
7. **Docker deployment drafted as production multi-service; container build not yet proven.** Dockerfile is rebuilt as a production multi-service image (multi-stage frontend build, python + gunicorn `wsgi:app`, nginx serving the built SPA with history fallback and `/api`→gunicorn proxy, supervisord running backend + durable worker). `backend/wsgi.py` and `gunicorn>=21` were added. Local smoke test (gunicorn + built-dist static serving) passed — `/health` OK, assets rebuilt — but the nginx SPA-fallback + proxy + worker-poll path must be verified by building the image in EasyPanel; no Docker runtime is available on this Mac.
|
||
8. **SaaS hardening mostly complete.** Durable rate limiting (wired to login), durable LLM usage/cost accounting, a durable redacted audit service, and durable single-use password reset (which also covers invite-pending setup) are added and tested. Still open: a complete resource authorization matrix across every role × endpoint, a full retryable-mutation inventory, and long-running-job review.
|
||
9. **Auxiliary API hardening is verified.** Fresh reviewer `deleg_d128494e` returned `passed=true` with empty `security_concerns` and `logic_errors`; current auxiliary security tests pass **8/8**. This closes only the template/agent-group slice, not the broader SaaS gate.
|
||
10. **Bounded schema/TaskManager review passed.** Fresh reviewer `deleg_fb0f80b0` returned `passed=true` with empty `security_concerns` and `logic_errors` after current-code runtime app/factory mismatch probes, SQLite pooled/NullPool foreign-key checks, ORM/migration parity, and migration round-trip checks. Its suggestions are non-blocking: expand mismatch coverage across every TaskManager operation, broaden pooled/NullPool assertions in tests, and assert full ORM/migration signatures. This closes only the exact schema/TaskManager slice; broad reviewer `deleg_949ab2f4` timed out and the wider SaaS gate remains open.
|
||
11. **The current bounded local import-boundary review passed.** Fresh reviewer `deleg_65bce3de` returned `passed=true` with empty `security_concerns` and `logic_errors`; it ran the focused boundary command with **6 passed** and isolated probes for five fresh local imports, shared `AgentActivity`, explicit Zep paths, the `graph_builder.Zep` seam, and subprocess isolation. This closes only the bounded import-boundary slice; semantic parity, complete consumer cutover, and the wider SaaS gate remain open.
|
||
|
||
## Next implementation order
|
||
|
||
1. Keep the current staged snapshot and re-run verification after any further edits.
|
||
2. Extend golden parity and tenant-negative coverage to the remaining local consumers and simulation-context/report E2E path.
|
||
3. Switch graph builder, entity reader, tools, profile generator, and report agent to the local memory adapter.
|
||
4. Remove Zep dependency/config only after parity and migration tests pass.
|
||
5. Add admin/super-admin UI and settings policy.
|
||
6. Finish production worker/deployment hardening and run a fresh end-to-end tenant matrix.
|
||
|
||
## Guardrails
|
||
|
||
- Exactly three role identifiers: `super_admin`, `admin`, `user`.
|
||
- Browser never receives LLM/API secrets.
|
||
- Backend authorization is authoritative; frontend guards are convenience only.
|
||
- Do not claim semantic LLM/Zep parity without golden-fixture evidence.
|
||
- Do not commit or deploy while the independent review gate remains unresolved.
|