# Engineering Log

## 2026-08-15 — Chatwoot privacy and rebrand audit

### Objective
Clone/inspect the official Chatwoot source and produce an evidence-based plan for a private-label community fork with no Chatwoot telemetry or visible product branding while preserving upstream syncability.

### Baseline
- Repository: `https://github.com/chatwoot/chatwoot.git`
- Branch: `develop`
- Commit: `9a73c1473ffa0ae6a9c7725046b8ca17922dcc83`
- Root license: MIT
- `enterprise/` license: proprietary and excluded from the recommended baseline

### Actions performed
1. Confirmed clone origin, branch, commit and clean working tree.
2. Mapped Rails, Sidekiq, Vue/Vite, widget, runtime configuration, manifests and deployment entry points.
3. Traced scheduled installation sync to `ChatwootHub.sync_with_hub`.
4. Confirmed from implementation and specs that `DISABLE_TELEMETRY` removes metrics but still POSTs installation configuration to Chatwoot Hub.
5. Traced Chatwoot-hosted push relay, remote changelog and optional onboarding registration.
6. Audited Amplitude, frontend/backend Sentry, optional APM integrations, arbitrary dashboard script injection and marketing tracking configuration.
7. Classified feature-required outbound integrations separately from unsolicited telemetry.
8. Mapped branding configuration, views, frontend components, locale copy, assets, manifest, mail/widget surfaces and technical compatibility identifiers.
9. Wrote implementation plan at `.hermes/plans/2026-08-15_092534-chatwoot-private-rebrand.md`.
10. Incorporated independent outbound-audit findings: silent cwctl command reporting, prechecked onboarding subscription, manual super-admin Hub refresh, and unconditional signup-domain website/DNS enrichment.
11. Split the master plan into a small-model execution package with an execution contract, dependency index, 12 runbooks and 58 unique bounded tasks covering baseline, privacy removals, observability, branding, E2E egress, release gates and upstream maintenance.

### Key decisions
- Use Community Edition as the legal/technical baseline.
- Remove telemetry code paths rather than rely on environment opt-out.
- Preserve internal namespaces and widget API aliases initially to avoid upstream conflict and customer embed breakage.
- Use no-op adapters/central policy for high-churn analytics callsites.
- Remove hosted push relay; require own push credentials.
- Remove cwctl event reporting and automatic signup-domain enrichment from the strict privacy baseline.
- Validate zero unexpected egress dynamically, not by grep alone.

### Source changes
Application changes are tracked in the verified checkpoints and the uncheckpointed working tree described below.

### Implementation state (updated 2026-08-16)
- SM-01 audit harness checkpoint: `8ebb320`.
- SM-02 Hub sync/registration/telemetry removal checkpoint: `832a7fd`; compatibility correction checkpoint: `1a3697f`.
- SM-03 push-relay removal checkpoint: `8b1a033`; exact post-commit independent review passed with empty security and logic blocking arrays.
- SM-04 cwctl privacy checkpoint: `3458272`; exact post-commit independent review `1a3697f..3458272` passed with empty security and logic blocking arrays.
- SM-05 community enrichment removal checkpoint: `d9bf4c4`; split exact post-commit reviews `deleg_5e2578c5` (backend) and `deleg_ee3e217f` (frontend) both passed with empty security and logic blocking arrays. Enterprise compatibility fails because proprietary code still depends on the deleted community base class.
- SM-05 boundary decision: do not modify `enterprise/` or restore a compatibility stub in the community surface. The community-only checkpoint is independently verified; the enterprise `WebsiteBrandingService` load failure remains an explicit architecture/legal blocker.
- SM-06 Product Analytics removal is verified at `ced77af`: `a96b977` removed the Amplitude/token delivery surface, and `ced77af` added the missing `GLOBAL_CONFIG_KEYS` allowlist before `window.globalConfig` serialization. Fresh pre-commit review `deleg_f9644984` and exact post-commit review `deleg_9b0a2f63` both passed with empty security and logic blocking arrays. SM-07 local-only observability remains uncheckpointed. SM-08 community Help Center GTM removal is committed as `5619cc3` and exact post-commit independent review `deleg_dd3db5b7` passed; fixed, allow-listed provider snippets remain explicit admin-configured traffic, while the GTM container path is removed because its remote container can execute arbitrary tags/scripts.
- SM-08 enterprise marketing conversion tracking was classified as fixed-endpoint, cloud-only, limited attribution traffic inside proprietary enterprise code; it remains untouched pending a separate legal/compatibility decision.
- SM-09/10 remain blocked until approved product brand inputs are supplied.
- No push or deployment has occurred.

### Current verification notes
- SM-06 exact committed range `a96b977..ced77af` changed one controller line. The root cause was raw `GlobalConfig.get(...).merge(app_config)` allowing an unrequested `CLOUD_ANALYTICS_TOKEN` from a stubbed/configured hash to reach serialized `window.globalConfig`; the remediation applies `.slice(*GLOBAL_CONFIG_KEYS)` before the explicit `app_config` merge. Raw merge/sliced-allowlist reproductions were RED/GREEN, committed Ruby syntax and `git diff --check` passed, and the added-line security scan was clean. Rails RSpec/RuboCop remain environment-blocked because the active Ruby is 2.6.10 and Bundler 2.5.16 is unavailable.
- SM-03 post-commit exact-range review `1a3697f..8b1a033`: passed; exactly five requested paths changed; 7 targeted RSpec examples, Ruby syntax, RuboCop and git diff checks passed; no secrets or Hub relay fallback remain.
- SM-04 post-commit exact-range review `1a3697f..3458272`: passed; exactly two requested deployment paths changed; Bash 3.2 syntax, normal privacy test, fail-closed grep-status checks, secret scan and diff checks passed.
- SM-05 pre-commit staged review: passed for exactly 16 community-only paths; commit hook completed ESLint/RuboCop checks.
- SM-05 post-commit split review: backend slice `deleg_5e2578c5` and frontend slice `deleg_ee3e217f` both passed with explicit empty blocking arrays; combined committed range is exactly the intended 16 paths.
- SM-05 frontend targeted run: 9 suites and 8 tests passed. Backend RSpec rerun was blocked by the local toolchain (installed Ruby 3.4.10 versus Gemfile Ruby 3.4.4; bundle environment has no usable `rspec` executable). Historical controller baseline remains 28 examples/5 failures versus HEAD baseline 27/5; the new no-enrichment regression passed.
- SM-07 targeted Rails specs: 7 examples, 0 failures; Ruby/JS lint and syntax checks pass.
- SM-08 dashboard controller specs: 7 examples, 0 failures; runtime denylist confirms no `DASHBOARD_SCRIPTS` reader or sink in community files.
- SM-08 Help Center correction: 57 portal model/controller examples, targeted GTM regression 3/3, Ruby/ERB syntax, RuboCop, ESLint/Prettier and exact staged security/scope scans pass; Vite production build passes. Full Vitest run with `TZ=UTC` (the env the `package.json` `test` script sets) passes: 414 files / 4176 tests passed, 0 failed. The earlier recorded "407 passed/15 failed" baseline was an artifact of running without `TZ=UTC`; the 15 timezone-sensitive failures all pass under UTC and none are in changed paths.
- Frontend privacy batch: 14 tests, 14 passed.

### SM-05 gate closure
- Exact committed range `3458272..d9bf4c4` is verified by two fresh read-only reviews: backend `deleg_5e2578c5` and frontend `deleg_ee3e217f`; both returned `passed=true`, `security_concerns=[]`, and `logic_errors=[]`.
- Community signup/onboarding no longer performs automatic website/branding enrichment or its Redis marker; enterprise compatibility remains intentionally unresolved and must not be repaired by restoring remote enrichment.

### Remaining gates
- Resolve the SM-05 enterprise/legal boundary without restoring remote enrichment.
- Complete the SM-07 local-only observability checkpoint and the remaining SM-08 dashboard/config script-injection work; keep provider comments, marketing conversion tracking, and enterprise compatibility as separate scoped decisions.
- Run full Rails/frontend suites, production-like copied-database checks, HTTP/DNS egress capture, upstream history fetch and merge rehearsal.

## 2026-08-16 — SM-06 checkpoint closure

### Root cause and fix
- The first SM-06 commit (`a96b977`) removed `CLOUD_ANALYTICS_TOKEN` from the requested config keys but left `GlobalConfig.get(*GLOBAL_CONFIG_KEYS).merge(app_config)` vulnerable to unrelated keys returned by a stub/config source being serialized through `window.globalConfig`.
- The regression was reproduced with `analytics-test-token`; the one-line remediation in `ced77af` applies `.slice(*GLOBAL_CONFIG_KEYS)` before merging the explicit application config.

### Verification
- Pre-commit independent review `deleg_f9644984`: `passed=true`, `security_concerns=[]`, `logic_errors=[]`.
- Exact post-commit review `deleg_9b0a2f63` over `a96b977..ced77af`: `passed=true`, `security_concerns=[]`, `logic_errors=[]`.
- Committed scope: one path, one line; Ruby syntax, diff check, and added-line security scan passed.
- Rails RSpec/RuboCop: blocked by local runtime mismatch (`ruby 2.6.10`, missing Bundler 2.5.16); no runtime pass is claimed.

### Next action
- Start SM-07 as a separate isolated task/checkpoint. Do not mix its unstaged files with the verified SM-06 commits.

## 2026-08-16 — SM-07/SM-08 closure and residual Hub gate

### Verified implementation checkpoints
- SM-07 committed at `19dc449`; exact post-commit review of `372f316..19dc449` returned a complete pass with `security_concerns=[]` and `logic_errors=[]`. Focused frontend tests passed 6/6, ESLint passed, Ruby syntax passed, and the offline frozen-lockfile install passed. Rails RSpec/RuboCop remain blocked by Ruby 2.6.10 and missing Bundler 2.5.16.
- SM-08 dashboard/script removal committed at `364e72f`; exact post-commit review returned a complete pass with empty blocking arrays. `DASHBOARD_SCRIPTS` reader, catalog entry, and HTML sink are absent. Vite production build completed with exit code 0. Full Vitest with `TZ=UTC` passes 414 files / 4176 tests, 0 failed (the "407/15" earlier baseline was a non-UTC run artifact; the 15 timezone-sensitive tests all pass under UTC).

### Residual Hub worktree gate
- Remaining Community Hub URL/push/changelog code was removed in the worktree. The shared Enterprise billing callers were preserved: `ChatwootHub.billing_url` now accepts only explicit HTTPS `CHATWOOT_BILLING_URL`; both shared views suppress the link when unset or invalid.
- Deterministic evidence passed: changelog Vitest 1/1, ESLint, Ruby/Bash syntax, isolated billing validation harness (unset/HTTPS/http/malformed/userinfo), cwctl privacy test, privacy audit harness, default privacy audit `findings=0`, and added-line security scan. The residual Vite production build also completed successfully with `4,736 modules transformed`, `✓ built in 3m 49s`, exit code 0; only Browserslist/chunk-size warnings remain.
- The residual diff was intentionally **not committed** until a fresh independent review returned a valid approval. Fail-closed policy treats interrupted/timeout/incomplete/no-payload reviews as no verdict.

## 2026-08-16 — SM-07 remediation and residual Hub gate closure

### SM-07 remediation (committed `2ef6fa5`)
- A fresh backend review (`deleg_96a5daef`, `372f316..19dc449`) returned `passed=false` with one logic blocker: `FilteredCountInstrumentation` discarded its documented local observability signals (computed `duration_ms` and discarded it, mutated a transient `Thread.current` summary then discarded it, and `increment` silently no-oped for non-aggregated signals).
- Remediation (user chose in-process local sink): added a thread-safe (`Mutex`) per-account in-process store with a public `last_request_summary(account_id:)` reader; `record_request_summary` now stores the completed summary (status + `duration_ms` + counts) instead of discarding it. Regression specs assert the completed summary is readable and an unknown account returns `nil`.
- Verification: Ruby syntax passed, isolated sink probe `SINK_PROBE=pass` (dedup/status/counts/duration/unknown-account all asserted), and a grep for NewRelic/`new_relic`/`record_metric`/`record_custom_event`/`record_event`/`Net::HTTP`/`RestClient`/`HTTPClient` returned zero matches (no egress). Rails RSpec/RuboCop remain blocked by Ruby 2.6.10 / missing Bundler 2.5.16; no runtime pass is claimed.
- Independent exact committed-diff review `deleg_087e4a3f` over `19dc449..2ef6fa5` is **pending**; the SM-07 gate is not fully closed until that returns `passed=true` with empty blocking arrays.

### Residual Hub removal (committed `8101395`)
- A fresh independent review `deleg_49d6ee2e` of `/tmp/chatwoot-residual-index-FINAL.txt` (regenerated from the exact current index; verified byte-identical to the staged blobs with zero mismatches) returned `passed=true`, `security_concerns=[]`, `logic_errors=[]`. It verified: Community Hub URL/push/sync/registration/event/changelog egress removed; `changelog.js` is a local empty-feed adapter exporting the default `ChangelogApi` with no axios/fetch/network and its spec traps both `global.fetch` and `axios.get`; `billing_url` is HTTPS-only with host and no userinfo and never Hub-falls-back; both ERB templates guard with `.present?` and escaped `<%= %>`; `chatwoot_hub_spec` uses `singleton_class.instance_methods(false)` for edition-safe `base_url` assertion without modifying the proprietary Enterprise file; `privacy_audit` uses a narrow per-file/per-rule (`hub-url`) Enterprise exception while all other rules still scan it; deployment privacy guard unchanged; no secrets/injection/dummy service/unrelated scope.
- Committed as `8101395` (10 paths) with the approval review ID referenced in the message. The older review `deleg_589b22d1` (also `passed=true`) covered a prior snapshot and is superseded by the fresh final-index gate. Commit is clean: only the 10 residual paths, with SM-07 already committed separately at `2ef6fa5`.
- A post-commit exact committed-diff gate over `2ef6fa5..8101395` remains to be run to fully close the committed-bytes gate (staged review saw the index; a range review sees the final committed bytes).

### Release and compatibility blockers
- SM-09/10 cannot change branding defaults/assets/metadata until approved product name, domains, logos/icons, sender, legal URLs and primary color are supplied. Current visible-branding report remains 2,878 findings; it is report-only, not a privacy-rule pass.
- Enterprise `WebsiteBrandingService` remains proprietary and currently fails with `NameError: uninitialized constant WebsiteBrandingService`; no dummy service or remote enrichment was added.
- SM-11/12 static inventory: repository is shallow (`git rev-parse --is-shallow-repository=true`); Docker, Syft, Trivy, Cosign and Gitleaks are unavailable; Dockerfile base images are mutable tags. Full-history fetch, isolated HTTP/DNS capture, production-like DB rehearsal, image digest and SBOM gates remain blocked. No network action was performed.

## 2026-08-22 — LLM chatbot runtime architecture discovery

### Objective
Compare Hermes, embedded n8n-like workflow functionality, and external n8n for the Chatwoot AgentBot path. This was a read-only architecture review; no production code was changed.

### Verified Chatwoot contract
- `AgentBot` currently has `bot_type: webhook`, `outgoing_url`, `bot_config`, secret, access token, account/inbox associations.
- `AgentBotListener` enqueues an asynchronous `AgentBots::WebhookJob`; it POSTs an event payload and does not wait for an LLM response.
- `Webhooks::Trigger` signs requests with `X-Chatwoot-Timestamp` and `X-Chatwoot-Signature`, retries 429/500 for agent bots, and treats the external bot as responsible for posting an outgoing message through Chatwoot API.
- Super Admin AgentBot is an admin CRUD surface. Per-account Bot Configuration is the correct customer-facing runtime boundary.
- No n8n integration exists in the current repository search.

### Verified Hermes contract
- Hermes API server exposes OpenAI-compatible `/v1/chat/completions`, Responses/Runs and Sessions APIs with bearer auth; Chat Completions is stateless unless history/response chaining is supplied.
- Hermes webhook adapter expects its own generic signature headers (`X-Webhook-Signature-V2`/`X-Webhook-Timestamp`), not Chatwoot's signature headers, and its configured delivery targets do not include Chatwoot message creation.
- Hermes memory/profiles are Hermes-scoped, not automatically Chatwoot-account-scoped. API server capabilities include terminal/tools, so a customer-facing deployment needs a restricted toolset, sandbox, and tenant isolation.

### Architecture finding
- Directly setting Chatwoot `outgoing_url` to Hermes is not a complete integration. A bridge/adapter is required to verify Chatwoot events, map account/inbox/conversation identity, invoke Hermes, enforce tool policy/idempotency, and POST the final message back to Chatwoot.
- n8n is appropriate as an external workflow/orchestration layer, but should not become the source of truth for conversation state or tenant knowledge.
- Embedding n8n UI/runtime in the product is a separate product/licensing/operations scope. n8n's official OEM docs require a commercial agreement when customers interact with the embedded editor; behind-the-scenes backend use is a distinct model.

### Decision candidate
Use a hybrid target: Chatwoot AgentBot → dedicated Hermes bridge → Hermes API for LLM/knowledge/tools; keep external n8n optional for cross-system workflows and side effects. Start with n8n as a P0 validation path only if it materially shortens learning; do not embed n8n or expose unrestricted Hermes tools to end users.

### Repository state
The worktree already contained unrelated uncommitted M1/M2 files before this review. Do not mix the future bridge implementation with those changes without an isolated worktree/branch.

## 2026-08-22 — Final decision: external n8n (Option 3)

- User selected Option 3: keep n8n as a separate service for the LLM chatbot/workflow path.
- No Hermes integration, embedded n8n/workflow builder, or Chatwoot production-code change is authorized in this decision.
- Existing Chatwoot AgentBot webhook contract remains the boundary; n8n receives the event, calls the configured LLM/external services, and posts the reply back through Chatwoot API.
- Treat the earlier Hermes hybrid architecture as analysis only, not an implementation decision. A Gateway/Adapter remains a future option if tenant isolation, idempotency, observability, or multi-runtime routing later require it.