moreminimore-chat/.hermes/plans/2026-08-19-m2-self-improving-analytics.md

M2 — Self-Improving Chatbot Analytics (Moreminimore Chat)

Status: part2b ✅ · phase 2 ✅ · phase 3 import-backend ✅ (in review passed) · persona-eval/approval/import-UI next · Updated: 2026-08-19 Repo: /Users/kunthawat/Gitea/Chatwoot · branch develop

Goal

Every chat of every account is analyzed by an LLM daily → auto-tagged (topic + product) + deal outcome → aggregated into immutable daily metrics → admin-only reports + weekly persona improvement recommendation.

Design decisions (approved by user)

• Analyze EVERY chat of EVERY account daily (LLM cost accepted — selling point is self-improving chatbot).
• LLM cascade (lib/llm/resolver.rb): per-account openai hook → Captain config → nil (feature disabled). LLM effectively always on unless Captain unset.
• Original conversation tags: use Chatwoot's label_list (ActsAsTaggableOn) — do NOT add many tag fields. Product/sale tag goes into the same tag list, prefixed (e.g. group>subgroup>product joined, or สินค้า:.../กลุ่ม:...). top_tags aggregates from these.
• No tagged_at (skip time-tracking to avoid DB complexity).
• Daily stats are immutable snapshots; live tags get re-tagged later (e.g. session becomes product B → product A removed). Snapshot preserves earlier state.
• Report page admin-role ONLY. Filter agent/team/inbox/channel/tag/sale-tag/time + cross-analysis. Break down per CUSTOMER and per AGENT.
• Weekly (Mon 10:00) LLM summarizes 7 days → persona improvement recommendation (summary only, NOT full prompt). Delivered via Line (quick-replies) / Telegram (inline keyboard) / webhook. Admin approve / view-full-prompt / reject; viewing full prompt → re-approve; approve → webhook with prompt.
• Product list: per-account, admin-only menu. Import copy/paste or CSV/XLSX. Multi-level hierarchy (group>subgroup>product>display). LLM matches from lowest level first, tags ancestors; supports multiple products/groups per chat.

Data foundation (DONE — committed)

• conversation_daily_metrics (per account/day: message_count, conversation_count, resolved/unresolved, avg_session_duration, top_tags, sale_tags, deal_outcomes, peak hours, agent/team/channel/inbox breakdown) — 9fbb0da9b
• customer_daily_metrics (per customer/day: same core + agent_ids) — 0fad9ce6b
• product_catalog_entries (per-account hierarchical: group>subgroup>product+aliases; tag_hierarchy helper) — 161e4210b
• lib/llm/resolver.rb (cascade; https-only api_base) — ccff2dfca
• M1 openai custom base_url — f9628db0e

Remaining work

part2b-ii — LLM classifier service ✅ DONE (2026-08-19)

Read a conversation (latest message + history for context) → classify → tags (topic + product from catalog using > join) + deal (won/lost/undecided). Cascade via Llm::Resolver.

• lib/llm/analytics_classifier.rb (new): NOT Captain::BaseTaskService (no captain feature-flag gate — M2 is always-on via resolver).
• Uses Llm::Resolver.resolve(account) → Llm::Config.with_api_key(key, api_base:) + context.chat(model:).with_schema(...).ask(...) for structured JSON output.
• Matches product from ProductCatalogEntry via tag_hierarchy (index → path mapping with values_at+compact, safe against bad indexes).
• Writes nothing — pure classifier. Returns Result (topics/products/deal) or disabled: true when resolver is nil (fail-closed, no LLM call, no content sent).
• Also fixed: added has_many :product_catalog_entries to Account (foundation commit 161e4210b added the model+migration but never wired the reverse association → would have been NoMethodError).
• Verified: ruby -c clean (2.6.10 + 3.4.10); isolated smoke test 8/8 under Ruby 3.4.10 (disabled path, catalog join, fence sanitize, present path mapping + deal); static security scan clean. Independent reviewer deleg_47c1a2cb = complete 5-key PASS (empty security/logic; non-blocking em-dash escape fix applied).

part2b-iii — daily batch job ✅ DONE (2026-08-19)

sidekiq-cron ~02:30 in the SUPER ADMIN timezone (Asia/Bangkok, single TZ, no hourly-check) → auto-write tags + aggregate into conversation_daily_metrics + customer_daily_metrics (immutable counts; re-tag live).

• app/services/analytics/account_daily_processor.rb (new): per-account/date aggregation. Computes UTC day window from account.reporting_timezone (default UTC/Time.zone); selects conversations active in window; classifies each via Llm::AnalyticsClassifier (skips on error/disabled — fail-closed); applies tags (replaces prior M2-managed tags — topic: prefix + current catalog paths — preserves manual labels); accumulates per-conversation + per-customer metrics; upserts ConversationDailyMetric (unique account+date) + CustomerDailyMetric (unique account+contact+date).
• app/jobs/analytics/daily_metrics_job.rb (new): iterates all accounts with per-account rescue; computes closed day (yesterday) in account TZ.
• config/schedule.yml: adds analytics_daily_metrics_job at 30 2 * * * timezone Asia/Bangkok, queue scheduled_jobs.
• Verified: ruby -c clean (3.4.10); isolated smoke test 15/15 under Ruby 3.4.10 (day_window Bangkok UTC bounds, apply_tags re-tag preserving manual labels, accumulate conv+customer counts, upsert payload shape). Static scan clean; schedule.yml no dup keys. Independent reviewer deleg_fecf7c61 = complete 5-key PASS (empty security/logic; 4 non-blocking suggestions: add specs, N+1 in chat_message_count, deal_outcomes dual structure, apply_tags rescue scope).

phase 2 — rollup + report (BACKEND DONE + REVIEWED-PASS; frontend in review)

Backend (all committed-eligible, reviewer deleg_6800b012 = complete 5-key PASS):

• Analytics::ReportService — summary/timeseries/customers/agents from immutable conversation_daily_metrics + customer_daily_metrics (rollup/snapshot surface). Deal_outcomes normalization handles both 'totals'-nested (conv) and flat (customer).
• Analytics::DrilldownService — deep filterable drilldown over live Conversation records (since/until/agent_id/team_id/inbox_id/channel/tag/deal + pagination). Fail-closed label-join (scope.none when no label exists). Tag/deal match via acts_as_taggable_on :labels (Deal tag convention deal:won/lost/undecided).
• Api::V2::Accounts::AnalyticsReportsController — admin-only (authorize :report, :view? → ReportPolicy#view? = administrator?) with summary + drilldown endpoints; params validation (BadRequest on invalid date/int).
• config/routes.rb — resources :analytics_reports collection get :summary / get :drilldown under v2 accounts.
• AccountDailyProcessor#apply_tags — now also writes deal:<outcome> tag onto the live conversation (M2-managed, replaced on re-tag) so drilldown can filter by deal.
• Verified: ruby -c clean (3.4.10), smoke 12/12 (report summary/timeseries/deal-norm/top_tags/customers/agents/daterange + drilldown deal/channel/label_list/unknown-tag), routes parse, static scan clean. Reviewer deleg_6800b012 = complete 5-key PASS (4 non-blocking suggestions: N+1 in serialize, no specs, page not clamped, memoize labels).

Frontend (built, reviewer deleg_15f8447c pending):

• api/analyticsReports.js — v2 API wrapper (summary + drilldown).
• routes/dashboard/settings/reports/AnalyticsReports.vue — admin report dashboard: summary cards + filter bar (since/until/agent/team/inbox/channel/tag/deal) + drilldown table.
• reports.routes.js — new analytics_reports route under the (admin-only, REPORTS feature-flag) reports section.
• en/report.json — ANALYTICS_REPORTS i18n block.
• Verified: node --check on .js + JSON.parse on report.json. NOTE: full frontend build (eslint/vite/vitest) canNOT run locally (no node_modules) — flagged as limitation; run pnpm install && pnpm test before ship.

phase 3 — import + persona eval + approval

Import backend (DONE, reviewer deleg_7c595252 = complete 5-key PASS):

• Analytics::ProductCatalogImportService — imports product_catalog_entries from copy/paste / CSV / TSV / keyword (key: value) / named-header forms. Parsing: TSV vs '|' vs comma detection; keyword-mode + named-header column mapping (COLUMN_ALIASES group/product/subgroup/display); ordered 2-col=group+product / 3-col=group+subgroup+product; alias normalization (comma/semicolon split); upsert (same account+group+product → update, else create). Returns { imported, updated, errors[] }. Non-blocking reviewer fix applied: error line numbers now reference original input (header-shift offset).
• Api::V2::Accounts::ProductCatalogEntriesController — admin-only (ReportPolicy#view?): index (list) + import (POST content, BadRequest if blank).
• config/routes.rb — resources :product_catalog_entries only [:index] + collection post :import.
• Verified: ruby -c clean (3.4.10), smoke 10/10 (TSV/pipe/keyword/header/ordered/upsert/error), static scan clean. Reviewer deleg_7c595252 = complete 5-key PASS (4 non-blocking: no specs, line-offset [fixed], tab+pipe edge, XLSX unsupported).

XLSX support (DONE, reviewer deleg_e7e0879c = complete 5-key PASS): Added gem 'roo', '~> 2.10' to Gemfile; ProductCatalogImportService.import_file(account:, file_path:, filename:) + parse_spreadsheet (uses Roo::Spreadsheet.open lazily, converts sheet rows to pipe-delimited lines then reuses the shared parse_rows_from_lines parser; graceful rescue LoadError for missing roo + rescue StandardError for parse errors); extension() prefers original filename (Rack tempfiles lack extension). Controller import branches to file-upload (file.tempfile.path + file.original_filename) with a SUPPORTED_SPREADSHEET_EXTS whitelist (xlsx xls ods csv) rejecting other types, vs text content. Verified: ruby -c clean, stub smoke tests text 10/10 + xlsx 5/5 under plain Ruby 3.4.10 with a fake roo module, static scan clean, reviewer deleg_e7e0879c = complete 5-key PASS (3 non-blocking: no committed specs, file-ext whitelist [applied], extension() nil guard). REQUIRED deploy step: bundle install must run in the deploy env to resolve roo + update Gemfile.lock (cannot run here — Ruby 3.4.4 pin vs 3.4.10 installed).

Remaining phase 3 — weekly persona eval (DONE, reviewer deleg_b7d63fd7 = complete 5-key PASS):

• Analytics::WeeklyPersonaEvaluator — summarizes 7 days via Analytics::ReportService (aggregate counts/tags/deal_outcomes only, NO raw message content / NO customer PII) + LLM via Llm::Resolver cascade + with_schema → Result(summary, recommendations). Pure evaluator (no persistence). Fail-closed: no credential → disabled, never calls LLM.
• persona_evaluation admin-only endpoint (POST /analytics_reports/persona_evaluation) + route.
• Verified: ruby -c clean, smoke 8/8, static scan clean, reviewer deleg_b7d63fd7 = complete 5-key PASS (3 non-blocking: no specs, guard non-string recs, confirm .content — latter matches repo convention per Captain::ChatResponseHelper).

Remaining (NOT built):

• Frontend product-catalog import UI (DONE, reviewer deleg_0e9f9291 = PASS): api/productCatalog.js + productCatalog/Index.vue (paste + .xlsx/.csv upload + result + table) + route product_catalog_index (admin-only) wired into settings.routes.js. Verified: node --check. NOTE: full frontend build cannot run locally — run pnpm install && pnpm test before ship.
• Approval flow (DONE, all reviewers PASS) — user chose "LINE primary (quick-replies) + webhook fallback", then asked to add Telegram. Delivery precedence: LINE → Telegram → webhook. Analytics::PersonaApprovalService (deliver: LINE push to persona_line_user_id w/ quickReply buttons / Telegram sendMessage inline keyboard to persona_telegram_chat_id / webhook POST to persona_webhook_url + HMAC sig header; notify_approval: webhook-only sends the approval decision to the webhook) + Analytics::WeeklyPersonaEvaluationJob + controller actions (persona_evaluation_deliver, persona_approval_settings [line_user_id + telegram_chat_id + webhook_url, conditional], persona_decision) + routes + schedule.yml analytics_weekly_persona_evaluation_job (Mon 10:00 Asia/Bangkok). Config in Account#custom_attributes. Reviewers: deleg_1fe19871 FAIL-CLOSED (1 medium logic error :approve path — fixed) → re-review deleg_3052f7a4 PASS; Telegram channel deleg_34116d1a = complete 5-key PASS (3 non-blocking: i18n/dedup button labels, no specs). POST-MVP design note — prompt webhook handshake (DECIDED, NOT built; awaiting LLM-server details from user): User clarified the self-improvement loop: a SEPARATE LLM server (not this app) will hold/apply the actual prompt, and this app talks to it over webhook.
• Connect-first: on first webhook connection, the app should pull the current prompt via webhook (GET) and store it (snapshot).
• Approve → push: when the admin approves the weekly persona recommendation, the app POSTs the new prompt via webhook to the LLM server to apply.
• Analytics separate: this app only surfaces "what customers asked" (topics/products/deal) + recommendations; knowledge-prep/prompt-authoring is done/admin/LLM-server side, NOT here.
• Delivery-prompt-webhook design (REST 2-endpoint vs generic action-body) — DEFERRED, user said "document/design only, wait for details". Not built.
• Docs — NOT built.

Enterprise (EE) note

EE files still read CAPTAIN_OPEN_AI_ENDPOINT directly and won't honor per-account base_url gating — intentional (out of OSS scope), tracked as follow-up.

Verification notes

• Ruby env active is 2.6.10; target 3.4.4. Rails/RSpec/RuboCop cannot be reliably claimed passed — use ruby -c syntax + YAML validity + isolated smoke tests.
• Full Vitest clean under TZ=UTC: 414 files / 4176 tests, 0 failed (frontend untouched here).
• No secrets committed. Temp helpers under /tmp only.