moreminimore-chat/engineering-log.md

Engineering Log

2026-08-15 — Chatwoot privacy and rebrand audit

Objective

Clone/inspect the official Chatwoot source and produce an evidence-based plan for a private-label community fork with no Chatwoot telemetry or visible product branding while preserving upstream syncability.

Baseline

• Repository: https://github.com/chatwoot/chatwoot.git
• Branch: develop
• Commit: 9a73c1473ffa0ae6a9c7725046b8ca17922dcc83
• Root license: MIT
• enterprise/ license: proprietary and excluded from the recommended baseline

Actions performed

1. Confirmed clone origin, branch, commit and clean working tree.
2. Mapped Rails, Sidekiq, Vue/Vite, widget, runtime configuration, manifests and deployment entry points.
3. Traced scheduled installation sync to ChatwootHub.sync_with_hub.
4. Confirmed from implementation and specs that DISABLE_TELEMETRY removes metrics but still POSTs installation configuration to Chatwoot Hub.
5. Traced Chatwoot-hosted push relay, remote changelog and optional onboarding registration.
6. Audited Amplitude, frontend/backend Sentry, optional APM integrations, arbitrary dashboard script injection and marketing tracking configuration.
7. Classified feature-required outbound integrations separately from unsolicited telemetry.
8. Mapped branding configuration, views, frontend components, locale copy, assets, manifest, mail/widget surfaces and technical compatibility identifiers.
9. Wrote implementation plan at .hermes/plans/2026-08-15_092534-chatwoot-private-rebrand.md.
10. Incorporated independent outbound-audit findings: silent cwctl command reporting, prechecked onboarding subscription, manual super-admin Hub refresh, and unconditional signup-domain website/DNS enrichment.
11. Split the master plan into a small-model execution package with an execution contract, dependency index, 12 runbooks and 58 unique bounded tasks covering baseline, privacy removals, observability, branding, E2E egress, release gates and upstream maintenance.

Key decisions

• Use Community Edition as the legal/technical baseline.
• Remove telemetry code paths rather than rely on environment opt-out.
• Preserve internal namespaces and widget API aliases initially to avoid upstream conflict and customer embed breakage.
• Use no-op adapters/central policy for high-churn analytics callsites.
• Remove hosted push relay; require own push credentials.
• Remove cwctl event reporting and automatic signup-domain enrichment from the strict privacy baseline.
• Validate zero unexpected egress dynamically, not by grep alone.

Source changes

Application changes are tracked in the verified checkpoints and the uncheckpointed working tree described below.

Implementation state (updated 2026-08-16)

• SM-01 audit harness checkpoint: 8ebb320.
• SM-02 Hub sync/registration/telemetry removal checkpoint: 832a7fd; compatibility correction checkpoint: 1a3697f.
• SM-03 push-relay removal checkpoint: 8b1a033; exact post-commit independent review passed with empty security and logic blocking arrays.
• SM-04 cwctl privacy checkpoint: 3458272; exact post-commit independent review 1a3697f..3458272 passed with empty security and logic blocking arrays.
• SM-05 community enrichment removal checkpoint: d9bf4c4; split exact post-commit reviews deleg_5e2578c5 (backend) and deleg_ee3e217f (frontend) both passed with empty security and logic blocking arrays. Enterprise compatibility fails because proprietary code still depends on the deleted community base class.
• SM-05 boundary decision: do not modify enterprise/ or restore a compatibility stub in the community surface. The community-only checkpoint is independently verified; the enterprise WebsiteBrandingService load failure remains an explicit architecture/legal blocker.
• SM-06 Product Analytics removal is verified at ced77af: a96b977 removed the Amplitude/token delivery surface, and ced77af added the missing GLOBAL_CONFIG_KEYS allowlist before window.globalConfig serialization. Fresh pre-commit review deleg_f9644984 and exact post-commit review deleg_9b0a2f63 both passed with empty security and logic blocking arrays. SM-07 local-only observability remains uncheckpointed. SM-08 community Help Center GTM removal is committed as 5619cc3 and exact post-commit independent review deleg_dd3db5b7 passed; fixed, allow-listed provider snippets remain explicit admin-configured traffic, while the GTM container path is removed because its remote container can execute arbitrary tags/scripts.
• SM-08 enterprise marketing conversion tracking was classified as fixed-endpoint, cloud-only, limited attribution traffic inside proprietary enterprise code; it remains untouched pending a separate legal/compatibility decision.
• SM-09/10 remain blocked until approved product brand inputs are supplied.
• No push or deployment has occurred.

Current verification notes

• SM-06 exact committed range a96b977..ced77af changed one controller line. The root cause was raw GlobalConfig.get(...).merge(app_config) allowing an unrequested CLOUD_ANALYTICS_TOKEN from a stubbed/configured hash to reach serialized window.globalConfig; the remediation applies .slice(*GLOBAL_CONFIG_KEYS) before the explicit app_config merge. Raw merge/sliced-allowlist reproductions were RED/GREEN, committed Ruby syntax and git diff --check passed, and the added-line security scan was clean. Rails RSpec/RuboCop remain environment-blocked because the active Ruby is 2.6.10 and Bundler 2.5.16 is unavailable.
• SM-03 post-commit exact-range review 1a3697f..8b1a033: passed; exactly five requested paths changed; 7 targeted RSpec examples, Ruby syntax, RuboCop and git diff checks passed; no secrets or Hub relay fallback remain.
• SM-04 post-commit exact-range review 1a3697f..3458272: passed; exactly two requested deployment paths changed; Bash 3.2 syntax, normal privacy test, fail-closed grep-status checks, secret scan and diff checks passed.
• SM-05 pre-commit staged review: passed for exactly 16 community-only paths; commit hook completed ESLint/RuboCop checks.
• SM-05 post-commit split review: backend slice deleg_5e2578c5 and frontend slice deleg_ee3e217f both passed with explicit empty blocking arrays; combined committed range is exactly the intended 16 paths.
• SM-05 frontend targeted run: 9 suites and 8 tests passed. Backend RSpec rerun was blocked by the local toolchain (installed Ruby 3.4.10 versus Gemfile Ruby 3.4.4; bundle environment has no usable rspec executable). Historical controller baseline remains 28 examples/5 failures versus HEAD baseline 27/5; the new no-enrichment regression passed.
• SM-07 targeted Rails specs: 7 examples, 0 failures; Ruby/JS lint and syntax checks pass.
• SM-08 dashboard controller specs: 7 examples, 0 failures; runtime denylist confirms no DASHBOARD_SCRIPTS reader or sink in community files.
• SM-08 Help Center correction: 57 portal model/controller examples, targeted GTM regression 3/3, Ruby/ERB syntax, RuboCop, ESLint/Prettier and exact staged security/scope scans pass; Vite production build passes. Full Vitest run with TZ=UTC (the env the package.json test script sets) passes: 414 files / 4176 tests passed, 0 failed. The earlier recorded "407 passed/15 failed" baseline was an artifact of running without TZ=UTC; the 15 timezone-sensitive failures all pass under UTC and none are in changed paths.
• Frontend privacy batch: 14 tests, 14 passed.

SM-05 gate closure

• Exact committed range 3458272..d9bf4c4 is verified by two fresh read-only reviews: backend deleg_5e2578c5 and frontend deleg_ee3e217f; both returned passed=true, security_concerns=[], and logic_errors=[].
• Community signup/onboarding no longer performs automatic website/branding enrichment or its Redis marker; enterprise compatibility remains intentionally unresolved and must not be repaired by restoring remote enrichment.

Remaining gates

• Resolve the SM-05 enterprise/legal boundary without restoring remote enrichment.
• Complete the SM-07 local-only observability checkpoint and the remaining SM-08 dashboard/config script-injection work; keep provider comments, marketing conversion tracking, and enterprise compatibility as separate scoped decisions.
• Run full Rails/frontend suites, production-like copied-database checks, HTTP/DNS egress capture, upstream history fetch and merge rehearsal.

2026-08-16 — SM-06 checkpoint closure

Root cause and fix

• The first SM-06 commit (a96b977) removed CLOUD_ANALYTICS_TOKEN from the requested config keys but left GlobalConfig.get(*GLOBAL_CONFIG_KEYS).merge(app_config) vulnerable to unrelated keys returned by a stub/config source being serialized through window.globalConfig.
• The regression was reproduced with analytics-test-token; the one-line remediation in ced77af applies .slice(*GLOBAL_CONFIG_KEYS) before merging the explicit application config.

Verification

• Pre-commit independent review deleg_f9644984: passed=true, security_concerns=[], logic_errors=[].
• Exact post-commit review deleg_9b0a2f63 over a96b977..ced77af: passed=true, security_concerns=[], logic_errors=[].
• Committed scope: one path, one line; Ruby syntax, diff check, and added-line security scan passed.
• Rails RSpec/RuboCop: blocked by local runtime mismatch (ruby 2.6.10, missing Bundler 2.5.16); no runtime pass is claimed.

Next action

• Start SM-07 as a separate isolated task/checkpoint. Do not mix its unstaged files with the verified SM-06 commits.

2026-08-16 — SM-07/SM-08 closure and residual Hub gate

Verified implementation checkpoints

• SM-07 committed at 19dc449; exact post-commit review of 372f316..19dc449 returned a complete pass with security_concerns=[] and logic_errors=[]. Focused frontend tests passed 6/6, ESLint passed, Ruby syntax passed, and the offline frozen-lockfile install passed. Rails RSpec/RuboCop remain blocked by Ruby 2.6.10 and missing Bundler 2.5.16.
• SM-08 dashboard/script removal committed at 364e72f; exact post-commit review returned a complete pass with empty blocking arrays. DASHBOARD_SCRIPTS reader, catalog entry, and HTML sink are absent. Vite production build completed with exit code 0. Full Vitest with TZ=UTC passes 414 files / 4176 tests, 0 failed (the "407/15" earlier baseline was a non-UTC run artifact; the 15 timezone-sensitive tests all pass under UTC).

Residual Hub worktree gate

• Remaining Community Hub URL/push/changelog code was removed in the worktree. The shared Enterprise billing callers were preserved: ChatwootHub.billing_url now accepts only explicit HTTPS CHATWOOT_BILLING_URL; both shared views suppress the link when unset or invalid.
• Deterministic evidence passed: changelog Vitest 1/1, ESLint, Ruby/Bash syntax, isolated billing validation harness (unset/HTTPS/http/malformed/userinfo), cwctl privacy test, privacy audit harness, default privacy audit findings=0, and added-line security scan. The residual Vite production build also completed successfully with 4,736 modules transformed, ✓ built in 3m 49s, exit code 0; only Browserslist/chunk-size warnings remain.
• The residual diff was intentionally not committed until a fresh independent review returned a valid approval. Fail-closed policy treats interrupted/timeout/incomplete/no-payload reviews as no verdict.

2026-08-16 — SM-07 remediation and residual Hub gate closure

SM-07 remediation (committed 2ef6fa5)

• A fresh backend review (deleg_96a5daef, 372f316..19dc449) returned passed=false with one logic blocker: FilteredCountInstrumentation discarded its documented local observability signals (computed duration_ms and discarded it, mutated a transient Thread.current summary then discarded it, and increment silently no-oped for non-aggregated signals).
• Remediation (user chose in-process local sink): added a thread-safe (Mutex) per-account in-process store with a public last_request_summary(account_id:) reader; record_request_summary now stores the completed summary (status + duration_ms + counts) instead of discarding it. Regression specs assert the completed summary is readable and an unknown account returns nil.
• Verification: Ruby syntax passed, isolated sink probe SINK_PROBE=pass (dedup/status/counts/duration/unknown-account all asserted), and a grep for NewRelic/new_relic/record_metric/record_custom_event/record_event/Net::HTTP/RestClient/HTTPClient returned zero matches (no egress). Rails RSpec/RuboCop remain blocked by Ruby 2.6.10 / missing Bundler 2.5.16; no runtime pass is claimed.
• Independent exact committed-diff review deleg_087e4a3f over 19dc449..2ef6fa5 is pending; the SM-07 gate is not fully closed until that returns passed=true with empty blocking arrays.

Residual Hub removal (committed 8101395)

• A fresh independent review deleg_49d6ee2e of /tmp/chatwoot-residual-index-FINAL.txt (regenerated from the exact current index; verified byte-identical to the staged blobs with zero mismatches) returned passed=true, security_concerns=[], logic_errors=[]. It verified: Community Hub URL/push/sync/registration/event/changelog egress removed; changelog.js is a local empty-feed adapter exporting the default ChangelogApi with no axios/fetch/network and its spec traps both global.fetch and axios.get; billing_url is HTTPS-only with host and no userinfo and never Hub-falls-back; both ERB templates guard with .present? and escaped <%= %>; chatwoot_hub_spec uses singleton_class.instance_methods(false) for edition-safe base_url assertion without modifying the proprietary Enterprise file; privacy_audit uses a narrow per-file/per-rule (hub-url) Enterprise exception while all other rules still scan it; deployment privacy guard unchanged; no secrets/injection/dummy service/unrelated scope.
• Committed as 8101395 (10 paths) with the approval review ID referenced in the message. The older review deleg_589b22d1 (also passed=true) covered a prior snapshot and is superseded by the fresh final-index gate. Commit is clean: only the 10 residual paths, with SM-07 already committed separately at 2ef6fa5.
• A post-commit exact committed-diff gate over 2ef6fa5..8101395 remains to be run to fully close the committed-bytes gate (staged review saw the index; a range review sees the final committed bytes).

Release and compatibility blockers

• SM-09/10 cannot change branding defaults/assets/metadata until approved product name, domains, logos/icons, sender, legal URLs and primary color are supplied. Current visible-branding report remains 2,878 findings; it is report-only, not a privacy-rule pass.
• Enterprise WebsiteBrandingService remains proprietary and currently fails with NameError: uninitialized constant WebsiteBrandingService; no dummy service or remote enrichment was added.
• SM-11/12 static inventory: repository is shallow (git rev-parse --is-shallow-repository=true); Docker, Syft, Trivy, Cosign and Gitleaks are unavailable; Dockerfile base images are mutable tags. Full-history fetch, isolated HTTP/DNS capture, production-like DB rehearsal, image digest and SBOM gates remain blocked. No network action was performed.

2026-08-22 — LLM chatbot runtime architecture discovery

Objective

Compare Hermes, embedded n8n-like workflow functionality, and external n8n for the Chatwoot AgentBot path. This was a read-only architecture review; no production code was changed.

Verified Chatwoot contract

• AgentBot currently has bot_type: webhook, outgoing_url, bot_config, secret, access token, account/inbox associations.
• AgentBotListener enqueues an asynchronous AgentBots::WebhookJob; it POSTs an event payload and does not wait for an LLM response.
• Webhooks::Trigger signs requests with X-Chatwoot-Timestamp and X-Chatwoot-Signature, retries 429/500 for agent bots, and treats the external bot as responsible for posting an outgoing message through Chatwoot API.
• Super Admin AgentBot is an admin CRUD surface. Per-account Bot Configuration is the correct customer-facing runtime boundary.
• No n8n integration exists in the current repository search.

Verified Hermes contract

• Hermes API server exposes OpenAI-compatible /v1/chat/completions, Responses/Runs and Sessions APIs with bearer auth; Chat Completions is stateless unless history/response chaining is supplied.
• Hermes webhook adapter expects its own generic signature headers (X-Webhook-Signature-V2/X-Webhook-Timestamp), not Chatwoot's signature headers, and its configured delivery targets do not include Chatwoot message creation.
• Hermes memory/profiles are Hermes-scoped, not automatically Chatwoot-account-scoped. API server capabilities include terminal/tools, so a customer-facing deployment needs a restricted toolset, sandbox, and tenant isolation.

Architecture finding

• Directly setting Chatwoot outgoing_url to Hermes is not a complete integration. A bridge/adapter is required to verify Chatwoot events, map account/inbox/conversation identity, invoke Hermes, enforce tool policy/idempotency, and POST the final message back to Chatwoot.
• n8n is appropriate as an external workflow/orchestration layer, but should not become the source of truth for conversation state or tenant knowledge.
• Embedding n8n UI/runtime in the product is a separate product/licensing/operations scope. n8n's official OEM docs require a commercial agreement when customers interact with the embedded editor; behind-the-scenes backend use is a distinct model.

Decision candidate

Use a hybrid target: Chatwoot AgentBot → dedicated Hermes bridge → Hermes API for LLM/knowledge/tools; keep external n8n optional for cross-system workflows and side effects. Start with n8n as a P0 validation path only if it materially shortens learning; do not embed n8n or expose unrestricted Hermes tools to end users.

Repository state

The worktree already contained unrelated uncommitted M1/M2 files before this review. Do not mix the future bridge implementation with those changes without an isolated worktree/branch.

2026-08-22 — Final decision: external n8n (Option 3)

• User selected Option 3: keep n8n as a separate service for the LLM chatbot/workflow path.
• No Hermes integration, embedded n8n/workflow builder, or Chatwoot production-code change is authorized in this decision.
• Existing Chatwoot AgentBot webhook contract remains the boundary; n8n receives the event, calls the configured LLM/external services, and posts the reply back through Chatwoot API.
• Treat the earlier Hermes hybrid architecture as analysis only, not an implementation decision. A Gateway/Adapter remains a future option if tenant isolation, idempotency, observability, or multi-runtime routing later require it.