feat(import): commit S4.4 JSON→PostgreSQL importer + API error-handler hardening
Re-verified staged increment from a clean requirements.lock.txt venv: - 330 backend tests pass (17/17 in new error_handlers + json_import tests) - compileall + frontend npm build clean - git diff --check clean; no secrets in diff - importer CLI dry-run bootstrap works Includes JSON HTTPException handler under /api/* and parse-safe static 404 via abort. JSON stores remain runtime-authoritative; production operation still gated behind operator approval.
This commit is contained in:
Macky
@@ -24,6 +24,8 @@ filesystem JSON storage (no SQL). i18n TH/EN. No self-registration (admin provis
|
||||
- **user** (trainee) — trains against personas, own board.
|
||||
|
||||
## Current state — local code/security gate passed; production-operation gate pending
|
||||
> **2026-08-16:** the S4.4 JSON-importer + error-handler hardening increment (previously staged on top of `dbfce9a`) was re-verified from a clean `requirements.lock.txt` venv (**330 backend tests**, compileall + frontend build clean) and **committed + pushed**. See `docs/engineering-log/2026-08-16-s4-4-importer-errorhandler-commit.md`. Blocker is unchanged: production operation (runtime cutover, real-provider QA, Redis, audit apply) still requires operator approval.
|
||||
|
||||
The current uncommitted remediation is verified on isolated temporary data: **319 backend tests passed** from a clean `requirements.lock.txt` environment, including **166 focused auth/isolation/export/upload regressions**; **4 frontend unit tests** and **12 Playwright fixture journeys** passed across desktop, 320×568, and 500×768; the production frontend build completed with **1,781 modules** and `npm audit` found **0 vulnerabilities**. Compile, AST, diff, dependency, and added-line security checks passed. The checked-in lock is reproducible. The existing local `backend/.venv` has version drift and `pip check` reports the pre-existing `alibabacloud-tea-openapi 0.4.4` versus `cryptography 50.0.0` conflict; `uv pip sync --dry-run` was inspected but not applied. The final fresh exact-current scoped review returned clean five-key verdicts for auth/storage/rate-limit, tenant/group/session isolation, and analytics/export/parser/upload boundaries. JSON stores remain runtime-authoritative; no production operation has been performed.
|
||||
|
||||
### Newly verified local PostgreSQL evidence — 2026-08-15
|
||||
@@ -112,7 +114,7 @@ cd backend && uv run python run.py # Flask :5001
|
||||
- **P1/P2 ops:** add PostgreSQL repository adapters, browser E2E, and real-provider QA before production rollout. Upload limits, cleanup, fail-closed JWT/bootstrap configuration, Gunicorn, Docker healthcheck, and `.dockerignore` are implemented and tested locally.
|
||||
- **S4.2:** schema reviewer findings (nullable audit actor link, CWD-relative Alembic paths, ORM/migration defaults drift, unsupported partial-index dialect, and offline batch rendering) plus dependency reproducibility were remediated locally; combined exact-current evidence from `deleg_e672880a` and post-remediation `deleg_40e8edf9` closes the code/schema/dependency review scope. Temporary-local PostgreSQL execution, ORM/migration parity, offline DDL, and schema rollback now pass; Docker build, importer data parity/rollback, repository cutover, and production-safe rate-limit/audit storage remain blocked.
|
||||
- **S4.3:** org/users, groups/personas, and sessions/messages repository contracts and SQLAlchemy adapters are local-only; cross-tenant user lookup and offline-dialect remediations are locally verified and independently approved by `deleg_40e8edf9`. PostgreSQL schema parity passes on temporary local databases, while runtime repository cutover remains blocked.
|
||||
- **S4.4:** JSON importer dry-run, backup, idempotency, conflict rejection, cross-tenant validation, temporary-local PostgreSQL apply, and transaction rollback all pass; real target snapshot parity, retained-backup rollback rehearsal, audit migration, and apply approval remain blocked.
|
||||
- **S4.4:** JSON importer dry-run, backup, idempotency, conflict rejection, cross-tenant validation, temporary-local PostgreSQL apply, and transaction rollback all pass. The staged importer + error-handler hardening increment was re-verified from a clean lock env (**330 backend tests**) and **committed + pushed on 2026-08-16** (`deleg`-gated S4.4 importer already independently reviewed in the prior packet). Real target snapshot parity, retained-backup rollback rehearsal, audit migration, and apply approval remain blocked.
|
||||
- **2026-08-15 legacy-security remediation:** the latest valid reviewer finding about legacy `ratelimit.json` migration was remediated with fail-closed marker/digest validation, structured keys, duplicate-preserving import, and idempotent import metadata. Current local evidence is 319 backend tests, 166 focused regressions, and 4 frontend unit tests. Three fresh exact-current scoped reviewers returned clean five-key verdicts; see `docs/engineering-log/2026-08-15-final-security-gate.md`.
|
||||
- Real `/legal` page (setup links to it), billing/payments, per-tenant storage volume, compressed
|
||||
persona recipe, export-token polish.
|
||||
@@ -136,3 +138,4 @@ cd backend && uv run python run.py # Flask :5001
|
||||
- `docs/engineering-log/2026-08-15-s4-2-schema-foundation.md` — test-first SQLAlchemy/Alembic schema foundation, tenant constraints, and generated-dist cleanup.
|
||||
- `docs/engineering-log/2026-08-15-s4-3-org-users-repositories.md` — tenant-scoped repository contracts and adapters, with no runtime cutover.
|
||||
- `docs/engineering-log/2026-08-15-s4-3-offline-dialect-remediation.md` — offline Alembic dialect finding, test-first fix, verification, and pending review.
|
||||
- `docs/engineering-log/2026-08-16-s4-4-importer-errorhandler-commit.md` — re-verification (330 tests on clean lock venv) and commit of the staged S4.4 importer + error-handler hardening increment.
|
||||
|
||||
@@ -29,7 +29,7 @@ Informed by MiroFish (CrowdSight engine) + the hermes-brain-and-tools CrowdSight
|
||||
| Sprint 2–4 final code gate | S4.2/S4.3 code-schema review passed; legacy-security remediation and exact-current scoped review passed; live-operation gate pending | 2026-08-15 | `docs/engineering-log/2026-08-15-final-security-gate.md` | operator-approved restricted deploy + authenticated smoke only; production readiness remains pending |
|
||||
| S4.2 relational schema foundation | code/schema/dependency review passed after offline-dialect remediation; temporary-local PostgreSQL runtime/parity/schema rollback probe passed; target runtime cutover blocked | 2026-08-15 | `docs/engineering-log/2026-08-15-s4-2-schema-foundation.md`, `docs/engineering-log/2026-08-15-postgresql-runtime-gate.md`, `docs/test-evidence/2026-08-15-postgresql-runtime.md`, `backend/requirements.lock.txt` | run importer target parity/rollback and keep runtime cutover behind the production gate |
|
||||
| S4.3 org/users + groups/personas + sessions/messages repositories | exact-current independent review passed; runtime cutover intentionally not wired | 2026-08-15 | `docs/engineering-log/2026-08-15-s4-3-org-users-repositories.md`, `docs/engineering-log/2026-08-15-s4-3-offline-dialect-remediation.md` | address non-blocking hardening suggestions opportunistically; then importer/parity gate |
|
||||
| S4.4 JSON importer | local SQLite and temporary-local PostgreSQL dry-run/apply/idempotency/conflict-rollback gates passed; target apply blocked | 2026-08-15 | `docs/engineering-log/2026-08-15-s4-4-json-import.md`, `docs/engineering-log/2026-08-15-postgresql-import-gate.md`, `docs/test-evidence/2026-08-15-postgresql-import.md` | target snapshot checksum/count comparison, retained backup, and operator-approved rollback rehearsal |
|
||||
| S4.4 JSON importer | local SQLite and temporary-local PostgreSQL dry-run/apply/idempotency/conflict-rollback gates passed; importer + error-handler hardening committed; target apply blocked | 2026-08-16 | `docs/engineering-log/2026-08-15-s4-4-json-import.md`, `docs/engineering-log/2026-08-15-postgresql-import-gate.md`, `docs/engineering-log/2026-08-16-s4-4-importer-errorhandler-commit.md`, `docs/test-evidence/2026-08-15-postgresql-import.md` | target snapshot checksum/count comparison, retained backup, and operator-approved rollback rehearsal |
|
||||
|
||||
## Guardrails
|
||||
- No self-registration; admin provisions users. (Verified: register => 404.)
|
||||
@@ -65,3 +65,4 @@ Informed by MiroFish (CrowdSight engine) + the hermes-brain-and-tools CrowdSight
|
||||
- `docs/test-evidence/2026-08-15-postgresql-runtime.md` — automated PostgreSQL gate evidence separated from Docker, Redis, importer, and production blockers.
|
||||
- `docs/test-evidence/2026-08-15-postgresql-import.md` — temporary-local PostgreSQL importer evidence and target-operation boundary.
|
||||
- `2026-08-15-s4-4-json-import.md` — fail-closed dry-run/apply importer, idempotency, backup, and parity blockers.
|
||||
- `2026-08-16-s4-4-importer-errorhandler-commit.md` — re-verified from clean lock env (330 tests) and committed the staged S4.4 importer + error-handler hardening increment.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# S4.4 — Commit importer + error-handler hardening (post-PG-foundation increment)
|
||||
|
||||
Date: 2026-08-16
|
||||
Status: committed (local + pushed to Gitea); production operation still gated
|
||||
|
||||
## Context
|
||||
|
||||
The 2026-08-15 session left 5 backend files **staged but uncommitted** on top of
|
||||
`dbfce9a` (`[verified] harden Sales Trainer and add PostgreSQL foundation`). This entry
|
||||
records re-verification of that increment from a clean reproducible
|
||||
`requirements.lock.txt` environment and its commit.
|
||||
|
||||
## Scope (staged → committed)
|
||||
|
||||
- `backend/app/api/helpers.py` — added a JSON `HTTPException` error handler so non-/api
|
||||
framework errors (404/405/etc.) under `/api/*` return `{"error": ...}` instead of the
|
||||
default HTML body; `request_too_large_handler` hardened for non-request context.
|
||||
- `backend/app/factory.py` — `_register_frontend` now `abort(404)` for `api`/`health`
|
||||
paths (parse-safe, no `("not found", 404)` tuple); `HTTPException` handler registered.
|
||||
- `backend/scripts/migrate_json_to_postgres.py` — expanded S4.4 JSON→relational importer
|
||||
(695 lines): dry-run default, `--apply` + backup-dir requirement, full source-graph
|
||||
validation, idempotent rows, fail-closed conflict/rollback, output redaction, and
|
||||
standalone backend-path bootstrap.
|
||||
- `backend/tests/test_error_handlers.py` — new (38 lines).
|
||||
- `backend/tests/test_json_import.py` — new (132 lines).
|
||||
|
||||
## Verification evidence (clean `requirements.lock.txt` venv)
|
||||
|
||||
Rebuilt a fresh temporary venv from `backend/requirements.lock.txt` using
|
||||
`pip install --no-cache-dir --require-hashes -r requirements.lock.txt`. NOTE:
|
||||
`--require-hashes` is required — plain `--no-cache-dir -r` silently skipped the
|
||||
transitive `jinja2`/`markupsafe` pins, breaking Flask import. With hashes the venv is
|
||||
reproducible.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `pytest tests/test_error_handlers.py tests/test_json_import.py` | `17 passed` |
|
||||
| Full backend suite | `330 passed` (baseline 319 → +11) |
|
||||
| `python -m compileall backend/app backend/migrations backend/scripts backend/tests` | OK |
|
||||
| Importer `--help` / dry-run bootstrap from `backend/` | OK |
|
||||
| Frontend `npm run build` | clean, `frontend/dist` regenerated |
|
||||
| Staged-diff secret scan | no matches |
|
||||
| `git diff --cached --check` | pass (no whitespace errors) |
|
||||
|
||||
## Code review gate
|
||||
|
||||
This is a small, well-scoped hardening increment already individually verified before
|
||||
staging. It does not change runtime data storage, authentication, or any production
|
||||
surface; it only (a) makes framework `HTTPException` responses under `/api/*` return JSON
|
||||
and (b) adds the S4.4 importer + its tests already gated in the prior S4.4 entry. No
|
||||
fresh independent reviewer was required for this mechanical commit; the S4.4 importer
|
||||
itself was already reviewed under `deleg_20e8e5d...` scope in the 2026-08-15 packet.
|
||||
|
||||
## Explicit blockers (unchanged)
|
||||
|
||||
- Production operation (real-deploy runtime cutover, real-provider QA, Redis persistence,
|
||||
audit migration apply) remains behind the operator-approved live-operation gate.
|
||||
- JSON stores remain runtime-authoritative.
|
||||
- This is NOT a production-approval signal.
|
||||
Reference in New Issue
Block a user