diff --git a/.hermes/plans/2026-08-19-m2-self-improving-analytics.md b/.hermes/plans/2026-08-19-m2-self-improving-analytics.md new file mode 100644 index 000000000..2080fe378 --- /dev/null +++ b/.hermes/plans/2026-08-19-m2-self-improving-analytics.md @@ -0,0 +1,90 @@ +# M2 — Self-Improving Chatbot Analytics (Moreminimore Chat) + +Status: **part2b ✅ · phase 2 ✅ · phase 3 import-backend ✅ (in review passed) · persona-eval/approval/import-UI next** · Updated: 2026-08-19 +Repo: `/Users/kunthawat/Gitea/Chatwoot` · branch `develop` + +## Goal +Every chat of every account is analyzed by an LLM daily → auto-tagged (topic + product) + deal outcome → +aggregated into immutable daily metrics → admin-only reports + weekly persona improvement recommendation. + +## Design decisions (approved by user) +- **Analyze EVERY chat of EVERY account daily** (LLM cost accepted — selling point is self-improving chatbot). +- **LLM cascade** (`lib/llm/resolver.rb`): per-account openai hook → Captain config → nil (feature disabled). LLM effectively always on unless Captain unset. +- **Original conversation tags**: use Chatwoot's `label_list` (ActsAsTaggableOn) — do NOT add many tag fields. Product/sale tag goes into the same tag list, prefixed (e.g. `group>subgroup>product` joined, or `สินค้า:...`/`กลุ่ม:...`). `top_tags` aggregates from these. +- **No `tagged_at`** (skip time-tracking to avoid DB complexity). +- **Daily stats are immutable snapshots**; live tags get re-tagged later (e.g. session becomes product B → product A removed). Snapshot preserves earlier state. +- **Report page admin-role ONLY.** Filter agent/team/inbox/channel/tag/sale-tag/time + cross-analysis. Break down per CUSTOMER and per AGENT. +- **Weekly (Mon 10:00)** LLM summarizes 7 days → persona improvement recommendation (summary only, NOT full prompt). Delivered via Line (quick-replies) / Telegram (inline keyboard) / webhook. Admin approve / view-full-prompt / reject; viewing full prompt → re-approve; approve → webhook with prompt. +- **Product list**: per-account, admin-only menu. Import copy/paste or CSV/XLSX. Multi-level hierarchy (group>subgroup>product>display). LLM matches from lowest level first, tags ancestors; supports multiple products/groups per chat. + +## Data foundation (DONE — committed) +- `conversation_daily_metrics` (per account/day: message_count, conversation_count, resolved/unresolved, avg_session_duration, top_tags, sale_tags, deal_outcomes, peak hours, agent/team/channel/inbox breakdown) — `9fbb0da9b` +- `customer_daily_metrics` (per customer/day: same core + agent_ids) — `0fad9ce6b` +- `product_catalog_entries` (per-account hierarchical: group>subgroup>product+aliases; `tag_hierarchy` helper) — `161e4210b` +- `lib/llm/resolver.rb` (cascade; https-only api_base) — `ccff2dfca` +- M1 openai custom base_url — `f9628db0e` + +## Remaining work +### part2b-ii — LLM classifier service ✅ DONE (2026-08-19) +Read a conversation (latest message + history for context) → classify → tags (topic + product from catalog using `>` join) + deal (won/lost/undecided). Cascade via `Llm::Resolver`. +- **`lib/llm/analytics_classifier.rb`** (new): NOT `Captain::BaseTaskService` (no captain feature-flag gate — M2 is always-on via resolver). +- Uses `Llm::Resolver.resolve(account)` → `Llm::Config.with_api_key(key, api_base:)` + `context.chat(model:).with_schema(...).ask(...)` for structured JSON output. +- Matches product from `ProductCatalogEntry` via `tag_hierarchy` (index → path mapping with `values_at`+`compact`, safe against bad indexes). +- Writes nothing — pure classifier. Returns `Result` (topics/products/deal) or `disabled: true` when resolver is nil (fail-closed, no LLM call, no content sent). +- **Also fixed**: added `has_many :product_catalog_entries` to `Account` (foundation commit `161e4210b` added the model+migration but never wired the reverse association → would have been `NoMethodError`). +- **Verified**: `ruby -c` clean (2.6.10 + 3.4.10); isolated smoke test 8/8 under Ruby 3.4.10 (disabled path, catalog join, fence sanitize, present path mapping + deal); static security scan clean. Independent reviewer `deleg_47c1a2cb` = **complete 5-key PASS** (empty security/logic; non-blocking em-dash escape fix applied). + +### part2b-iii — daily batch job ✅ DONE (2026-08-19) +sidekiq-cron ~02:30 in the SUPER ADMIN timezone (`Asia/Bangkok`, single TZ, no hourly-check) → auto-write tags + aggregate into conversation_daily_metrics + customer_daily_metrics (immutable counts; re-tag live). +- **`app/services/analytics/account_daily_processor.rb`** (new): per-account/date aggregation. Computes UTC day window from `account.reporting_timezone` (default UTC/Time.zone); selects conversations active in window; classifies each via `Llm::AnalyticsClassifier` (skips on error/disabled — fail-closed); applies tags (replaces prior M2-managed tags — `topic:` prefix + current catalog paths — preserves manual labels); accumulates per-conversation + per-customer metrics; upserts `ConversationDailyMetric` (unique account+date) + `CustomerDailyMetric` (unique account+contact+date). +- **`app/jobs/analytics/daily_metrics_job.rb`** (new): iterates all accounts with per-account rescue; computes closed day (`yesterday`) in account TZ. +- **`config/schedule.yml`**: adds `analytics_daily_metrics_job` at `30 2 * * *` timezone `Asia/Bangkok`, queue scheduled_jobs. +- **Verified**: `ruby -c` clean (3.4.10); isolated smoke test 15/15 under Ruby 3.4.10 (day_window Bangkok UTC bounds, apply_tags re-tag preserving manual labels, accumulate conv+customer counts, upsert payload shape). Static scan clean; schedule.yml no dup keys. Independent reviewer `deleg_fecf7c61` = **complete 5-key PASS** (empty security/logic; 4 non-blocking suggestions: add specs, N+1 in chat_message_count, deal_outcomes dual structure, apply_tags rescue scope). + +### phase 2 — rollup + report (BACKEND DONE + REVIEWED-PASS; frontend in review) +**Backend (all committed-eligible, reviewer `deleg_6800b012` = complete 5-key PASS):** +- **`Analytics::ReportService`** — summary/timeseries/customers/agents from immutable conversation_daily_metrics + customer_daily_metrics (rollup/snapshot surface). Deal_outcomes normalization handles both 'totals'-nested (conv) and flat (customer). +- **`Analytics::DrilldownService`** — deep filterable drilldown over live Conversation records (since/until/agent_id/team_id/inbox_id/channel/tag/deal + pagination). Fail-closed label-join (scope.none when no label exists). Tag/deal match via `acts_as_taggable_on :labels` (Deal tag convention `deal:won/lost/undecided`). +- **`Api::V2::Accounts::AnalyticsReportsController`** — admin-only (`authorize :report, :view?` → ReportPolicy#view? = administrator?) with `summary` + `drilldown` endpoints; params validation (BadRequest on invalid date/int). +- **`config/routes.rb`** — `resources :analytics_reports` collection get :summary / get :drilldown under v2 accounts. +- **`AccountDailyProcessor#apply_tags`** — now also writes `deal:` tag onto the live conversation (M2-managed, replaced on re-tag) so drilldown can filter by deal. +- Verified: `ruby -c` clean (3.4.10), smoke 12/12 (report summary/timeseries/deal-norm/top_tags/customers/agents/daterange + drilldown deal/channel/label_list/unknown-tag), routes parse, static scan clean. Reviewer `deleg_6800b012` = complete 5-key PASS (4 non-blocking suggestions: N+1 in serialize, no specs, page not clamped, memoize labels). + +**Frontend (built, reviewer `deleg_15f8447c` pending):** +- **`api/analyticsReports.js`** — v2 API wrapper (summary + drilldown). +- **`routes/dashboard/settings/reports/AnalyticsReports.vue`** — admin report dashboard: summary cards + filter bar (since/until/agent/team/inbox/channel/tag/deal) + drilldown table. +- **`reports.routes.js`** — new `analytics_reports` route under the (admin-only, REPORTS feature-flag) reports section. +- **`en/report.json`** — `ANALYTICS_REPORTS` i18n block. +- Verified: `node --check` on .js + JSON.parse on report.json. NOTE: full frontend build (eslint/vite/vitest) canNOT run locally (no node_modules) — flagged as limitation; run `pnpm install && pnpm test` before ship. + +### phase 3 — import + persona eval + approval +**Import backend (DONE, reviewer `deleg_7c595252` = complete 5-key PASS):** +- **`Analytics::ProductCatalogImportService`** — imports `product_catalog_entries` from copy/paste / CSV / TSV / keyword (`key: value`) / named-header forms. Parsing: TSV vs '|' vs comma detection; keyword-mode + named-header column mapping (COLUMN_ALIASES group/product/subgroup/display); ordered 2-col=group+product / 3-col=group+subgroup+product; alias normalization (comma/semicolon split); upsert (same account+group+product → update, else create). Returns `{ imported, updated, errors[] }`. Non-blocking reviewer fix applied: error line numbers now reference original input (header-shift offset). +- **`Api::V2::Accounts::ProductCatalogEntriesController`** — admin-only (`ReportPolicy#view?`): `index` (list) + `import` (POST content, BadRequest if blank). +- **`config/routes.rb`** — `resources :product_catalog_entries` only [:index] + collection post :import. +- Verified: `ruby -c` clean (3.4.10), smoke 10/10 (TSV/pipe/keyword/header/ordered/upsert/error), static scan clean. Reviewer `deleg_7c595252` = complete 5-key PASS (4 non-blocking: no specs, line-offset [fixed], tab+pipe edge, XLSX unsupported). + +**XLSX support (DONE, reviewer `deleg_e7e0879c` = complete 5-key PASS):** Added `gem 'roo', '~> 2.10'` to Gemfile; `ProductCatalogImportService.import_file(account:, file_path:, filename:)` + `parse_spreadsheet` (uses `Roo::Spreadsheet.open` lazily, converts sheet rows to pipe-delimited lines then reuses the shared `parse_rows_from_lines` parser; graceful `rescue LoadError` for missing roo + `rescue StandardError` for parse errors); `extension()` prefers original filename (Rack tempfiles lack extension). Controller `import` branches to file-upload (`file.tempfile.path` + `file.original_filename`) with a `SUPPORTED_SPREADSHEET_EXTS` whitelist (`xlsx xls ods csv`) rejecting other types, vs text `content`. Verified: `ruby -c` clean, stub smoke tests text 10/10 + xlsx 5/5 under plain Ruby 3.4.10 with a fake roo module, static scan clean, reviewer `deleg_e7e0879c` = complete 5-key PASS (3 non-blocking: no committed specs, file-ext whitelist [applied], extension() nil guard). **REQUIRED deploy step**: `bundle install` must run in the deploy env to resolve `roo` + update `Gemfile.lock` (cannot run here — Ruby 3.4.4 pin vs 3.4.10 installed). + +**Remaining phase 3 — weekly persona eval (DONE, reviewer `deleg_b7d63fd7` = complete 5-key PASS):** +- **`Analytics::WeeklyPersonaEvaluator`** — summarizes 7 days via `Analytics::ReportService` (aggregate counts/tags/deal_outcomes only, NO raw message content / NO customer PII) + LLM via `Llm::Resolver` cascade + `with_schema` → `Result(summary, recommendations)`. Pure evaluator (no persistence). Fail-closed: no credential → disabled, never calls LLM. +- **`persona_evaluation`** admin-only endpoint (`POST /analytics_reports/persona_evaluation`) + route. +- Verified: `ruby -c` clean, smoke 8/8, static scan clean, reviewer `deleg_b7d63fd7` = complete 5-key PASS (3 non-blocking: no specs, guard non-string recs, confirm .content — latter matches repo convention per Captain::ChatResponseHelper). + +**Remaining (NOT built):** +- **Frontend product-catalog import UI (DONE, reviewer `deleg_0e9f9291` = PASS):** `api/productCatalog.js` + `productCatalog/Index.vue` (paste + .xlsx/.csv upload + result + table) + route `product_catalog_index` (admin-only) wired into `settings.routes.js`. Verified: `node --check`. NOTE: full frontend build cannot run locally — run `pnpm install && pnpm test` before ship. +- **Approval flow (DONE, all reviewers PASS)** — user chose **"LINE primary (quick-replies) + webhook fallback"**, then asked to add **Telegram**. Delivery precedence: **LINE → Telegram → webhook**. `Analytics::PersonaApprovalService` (`deliver`: LINE push to `persona_line_user_id` w/ quickReply buttons / Telegram `sendMessage` inline keyboard to `persona_telegram_chat_id` / webhook POST to `persona_webhook_url` + HMAC sig header; `notify_approval`: **webhook-only** sends the approval decision to the webhook) + `Analytics::WeeklyPersonaEvaluationJob` + controller actions (`persona_evaluation_deliver`, `persona_approval_settings` [line_user_id + telegram_chat_id + webhook_url, conditional], `persona_decision`) + routes + schedule.yml `analytics_weekly_persona_evaluation_job` (Mon 10:00 Asia/Bangkok). Config in `Account#custom_attributes`. Reviewers: `deleg_1fe19871` FAIL-CLOSED (1 medium logic error `:approve` path — fixed) → re-review `deleg_3052f7a4` PASS; Telegram channel `deleg_34116d1a` = **complete 5-key PASS** (3 non-blocking: i18n/dedup button labels, no specs). **POST-MVP design note — prompt webhook handshake (DECIDED, NOT built; awaiting LLM-server details from user):** +User clarified the self-improvement loop: a SEPARATE LLM server (not this app) will hold/apply the actual prompt, and this app talks to it over webhook. +- **Connect-first**: on first webhook connection, the app should **pull the current prompt** via webhook (GET) and store it (snapshot). +- **Approve → push**: when the admin approves the weekly persona recommendation, the app **POSTs the new prompt** via webhook to the LLM server to apply. +- **Analytics separate**: this app only surfaces "what customers asked" (topics/products/deal) + recommendations; knowledge-prep/prompt-authoring is done/admin/LLM-server side, NOT here. +- Delivery-prompt-webhook design (REST 2-endpoint vs generic action-body) — **DEFERRED**, user said "document/design only, wait for details". Not built. +- **Docs** — NOT built. + +## Enterprise (EE) note +EE files still read `CAPTAIN_OPEN_AI_ENDPOINT` directly and won't honor per-account base_url gating — intentional (out of OSS scope), tracked as follow-up. + +## Verification notes +- Ruby env active is 2.6.10; target 3.4.4. Rails/RSpec/RuboCop cannot be reliably claimed passed — use `ruby -c` syntax + YAML validity + isolated smoke tests. +- Full Vitest clean under TZ=UTC: 414 files / 4176 tests, 0 failed (frontend untouched here). +- No secrets committed. Temp helpers under /tmp only. diff --git a/.hermes/plans/2026-08-20-oss-chatbot-guardrail-knowledge.md b/.hermes/plans/2026-08-20-oss-chatbot-guardrail-knowledge.md new file mode 100644 index 000000000..bb87d942d --- /dev/null +++ b/.hermes/plans/2026-08-20-oss-chatbot-guardrail-knowledge.md @@ -0,0 +1,82 @@ +# Plan — OSS Self-Contained Chatbot (Guardrail + Knowledge + Llm::Resolver Answer) + +Repo: `/Users/kunthawat/Gitea/Chatwoot` · branch `develop` +Date: 2026-08-20 +Status: **PLAN — awaiting user approval before build** + +## Vision (from user) +Make this Chatwoot fork a self-contained OSS chatbot: every inbound chat → guardrail +(keep users on-topic, e.g. reject "บอกชะตารายวัน / ดูดวง" when it's a product/service bot) +→ if in-scope, answer with LLM using a knowledge base (md/csv/excel) + system prompt + +guardrail prompt + history. No EE, no external Captain. Answer via `Llm::Resolver` local. + +## Decisions locked (from clarify) +- **Not using EE** → Captain v2 (`enterprise/`) is out of scope. Use OSS path only. +- **Architecture**: build our own `Integrations::Chatbot::ProcessorService < Integrations::BotProcessorService` + (mirrors `Captain::ProcessorService` / `Dialogflow::ProcessorService`), overriding `get_response` + to answer via `Llm::Resolver` + knowledge base instead of an external webhook. Selectable per-inbox + (like Captain/Dialogflow choose processor by hook app_id). +- **Trigger stays** OSS AgentBot / pending flow (the "AI answers" = conversation pending). +- **LLM calls**: default **1 call** returning `{allowed, answer}`; config option to use **2 calls** + (guardrail check, then answer) for LLMs that need them. Backward-compatible. +- **KB retrieval**: **keyword + embedding** retrieval (top-k relevant chunks), NOT whole-KB-in-prompt. +- **Out-of-scope policy**: + - topic clearly off-guardrail (e.g. fortune-telling) → **refuse with template** + - topic related but not in KB / undecidable → **auto human handoff** (`bot_handoff!`) +- **KB storage**: extend `product_catalog_entries` (csv/xlsx product rows) + add **MD FAQ** table. + +## Current OSS bot flow (verified) +``` +inbound message (conversation pending?) + → ... AgentBots::WebhookJob / agent_bot_listener → Webhooks::Trigger → webhook (external) +``` +`BotProcessorService` base: `should_run_processor?` (message.reportable?, conversation.pending?) +→ `get_response(source_id, content)`; `process_action` handles :handoff → `bot_handoff!` / :resolve. +`Webhooks::Trigger#update_conversation_status`: on agent_bot failure, pending → open! (human takes over). + +## Architecture +``` +[our ProcessorService] < Integrations::BotProcessorService + get_response → guardrail? + ├─ in-scope → Llm::Resolver answer (KB + system prompt + guardrail + history) → reply + └─ out-of-scope → out-of-scope reply template (reject) / bot_handoff! +``` +Selected per inbox by hook app_id (same mechanism as Captain/Dialogflow). + +## Deliverables (build order) +1. **KB backend**: `KnowledgeBaseFaq` model+migration (account-scoped, md FAQ: title + content + topic tags) + + reuse `product_catalog_entries` for csv/xlsx product rows. Import service (extend + `ProductCatalogImportService` / add MD FAQ import via `roo`/CSV/stdlib). Admin endpoints (list/import) + routes. +2. **Retrieval service**: `Chatbot::KnowledgeRetriever` — keyword (+ optional embedding) top-k selection + over FAQ + product_catalog. Embedding vector column on FAQ table (nullable), keyword via SQL ILIKE/tsvector. +3. **Guardrail + answer**: `Chatbot::GuardrailService` + `Chatbot::AnswerService` — default 1 call + `{ allowed, answer }`; optional 2-call mode. Out-of-scope → refuse template; related-but-not-in-KB / + undecidable → human handoff. +4. **Chatbot processor**: `Integrations::Chatbot::ProcessorService < BotProcessorService` — get_response = + retrieve → guardrail/answer → reply | refuse | handoff. Wire selection per inbox. + - **Human handoff path adjusts chat status**: on related-but-not-in-KB / undecidable, call + `conversation.bot_handoff!` which **releases the bot (= `pending` → `open`), clears `assignee_agent_bot`, + sets `waiting_since`**, and dispatches the handoff event so a human agent queue/assignment picks it up. +5. **Account config**: per-account: system prompt + guardrail prompt + out-of-scope reply template + + call-mode (1 or 2) + enabled flag + which KB (folder/index). Store in Account#custom_attributes or settings model. + Admin endpoints + UI. +6. **Verify + review each part** (smoke / ruby -c / static scan / independent reviewer). + +## Open questions (mostly resolved; remaining minor) +- [x] LLM calls: 1 default, 2 optional (user decision). +- [x] KB retrieval: keyword + embedding, top-k (user decision). +- [x] Out-of-scope: refuse if clearly off-topic; auto handoff if related-but-not-in-KB/undecidable. +- [x] KB storage: extend product_catalog + add MD FAQ. +- [ ] Embedding: which provider/API to compute embeddings (Llm::Resolver? separate embedding model?). +- [ ] MD FAQ granularity: one row per file? per heading/section chunk? (affects retrieval + import). + +## Success criteria +- OSS inbox with our processor answers in-scope from KB (keyword+embedding) via Llm::Resolver (1 or 2 calls). ✅ built +- Clearly out-of-scope (e.g. fortune-telling) → templated refusal. ✅ +- Related-but-not-in-KB / undecidable → auto human handoff. ✅ (bot_handoff! → pending→open) +- Product rows (csv/xlsx) + MD FAQ both import + retrieve. ✅ +- Per-account config (system/guardrail/out-of-scope template/call mode/enabled) via admin endpoint. ✅ (ChatbotConfigController + ConfigService) +- Each deliverable passes smoke + static scan + independent reviewer. ✅ (final re-review deleg_a5c361cb = passed:true, end-to-end dispatchable) + +## Wiring (verified, end-to-end) +apps.yml chatbot(inbox) → HooksController create (ensure_hook_type=inbox) → HookListener supported_events_map['chatbot'] → HookJob INTEGRATION_PROCESSORS['chatbot'] → Chatbot::ProcessorService → DecisionService → Llm::Resolver. i18n chatbot added to en.yml integration_apps. diff --git a/Gemfile b/Gemfile index 5a010cc34..d572dd207 100644 --- a/Gemfile +++ b/Gemfile @@ -51,6 +51,8 @@ gem 'gmail_xoauth' gem 'net-smtp', '~> 0.3.4' # Prevent CSV injection gem 'csv-safe' +# XLSX / ODS / XLS spreadsheet reading (used by Analytics::ProductCatalogImportService) +gem 'roo', '~> 2.10' ##-- for active storage --## gem 'aws-sdk-s3', require: false diff --git a/HANDOFF.md b/HANDOFF.md index c0ea1c998..3e7fd987b 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,5 +1,24 @@ # HANDOFF +## ═══════ SESSION 2026-08-22 — Final decision: external n8n (Option 3) ═══════ + +- User selected Option 3: n8n remains a separate service for the LLM chatbot/workflow path. +- Do not integrate Hermes, embed n8n, or change Chatwoot production code for this decision. +- Keep the existing AgentBot async webhook boundary: Chatwoot → external n8n → configured LLM/external services → Chatwoot API reply. +- The Gateway/Adapter design is deferred; implement it only if later requirements justify centralized tenant isolation, idempotency, DLQ, observability, or provider routing. +- Only this handoff/log decision is updated; production source remains unchanged. + +## ═══════ SESSION 2026-08-22 — LLM chatbot runtime architecture discovery ═══════ + +- Read-only architecture review for `/Users/kunthawat/Gitea/Chatwoot`; no production code changed. +- Chatwoot's existing AgentBot path is already the intended boundary: async signed webhook out, external runtime processes, then external runtime posts an outgoing message through Chatwoot API. `outgoing_url` is not an LLM request/response endpoint. +- Hermes is technically feasible only behind a bridge. Hermes API server is the preferred execution interface; Hermes webhook adapter is not plug-compatible because signature headers and response delivery contract differ. +- Recommended target: `Chatwoot AgentBot → dedicated Hermes bridge → Hermes API → Chatwoot API`; optional external n8n remains for visual workflows, CRM/LINE/ERP side effects, and integrations. Do not embed n8n or give a customer-facing Hermes profile unrestricted terminal/file tools. +- Tenant rules: keep Chatwoot as transcript/source of truth; namespace every request by account/inbox/conversation; do not treat Hermes personal memory/profile files as a multi-tenant knowledge base. Use allow-listed tools, sandboxing, idempotency, HMAC verification, secret rotation, and human handoff. +- Next implementation gate: P0 one-account bridge proof (or n8n→Hermes proof), message-created only, no loop, signed webhook verification, reply via bot token, duplicate/retry tests, cross-tenant isolation test, and latency/error telemetry. +- Working tree was already dirty with unrelated M1/M2 changes; future bridge work must use a separate worktree/branch. + + ## ═══════ SESSION 2026-08-19 — Moreminimore Chat: rebrand + M1 + M2 work ═══════ ## Current state (2026-08-19) @@ -30,10 +49,16 @@ HEAD = `ccff2dfca`. Worktree clean except untracked `.hermes/plans/` (never comm - Product list: per-account, admin-only menu (like report), import via copy/paste or CSV/XLSX. Multi-level hierarchy (big group>subgroup>product>display) — LLM matches from lowest level first, tags ancestors; supports multiple products/groups per chat. ### NEXT TO DO (M2 remaining — biggest remaining work) -1. **part2b-ii**: LLM classifier service — read conversation (latest message + history for context), classify → tags (topic + product from catalog using '>' join) + deal (won/lost/undecided). Cascade via Llm::Resolver. -2. **part2b-iii**: daily batch job (sidekiq-cron) at ~02:00-03:00 in the SUPER ADMIN timezone (single TZ, no hourly-check) → auto-write tags to conversation, aggregate into conversation_daily_metrics + customer_daily_metrics (immutable counts; re-tag live). -3. **phase 2**: monthly/yearly rollup (sum daily) + admin-only report dashboard (filters agent/team/inbox/channel/tag/sale/time, cross-analysis) + deep per-customer/per-agent views. -4. **phase 3**: product-list import UI (admin-only, copy/paste + CSV + XLSX), weekly persona eval + approval flow (Telegram inline button, Line quick-reply, webhook) + docs. +1. **part2b-ii — LLM classifier service: ✅ DONE (2026-08-19)**. New `lib/llm/analytics_classifier.rb` — reads conversation, classifies via `Llm::Resolver` cascade + `with_schema` JSON output → `Result` (topics, products via tag_hierarchy path mapping, deal won/lost/undecided). Fail-closed on no-credential (returns `disabled`, never calls LLM / sends content). Also fixed: `Account` missing `has_many :product_catalog_entries` (foundation `161e4210b` wired model+migration but not the reverse association → would NoMethodError). Verified: `ruby -c` clean (2.6/3.4), isolated smoke 8/8 under Ruby 3.4.10 (found `/opt/homebrew/opt/ruby@3.4/bin/ruby` — real 3.4.10 available, `.ruby-version`=3.4.4), static scan clean, independent reviewer `deleg_47c1a2cb` = complete 5-key PASS (non-blocking em-dash escape fix applied). Plan doc: `.hermes/plans/2026-08-19-m2-self-improving-analytics.md`. +2. **part2b-iii — daily batch job: ✅ DONE (2026-08-19)**. New `app/services/analytics/account_daily_processor.rb` (per-account/date: UTC day window from `account.reporting_timezone`, selects active conversations, classifies via `Llm::AnalyticsClassifier` fail-closed, re-tags replacing M2-managed tags + preserving manual labels, upserts ConversationDailyMetric unique account+date + CustomerDailyMetric unique account+contact+date) + `app/jobs/analytics/daily_metrics_job.rb` (per-account rescue, yesterday in account TZ) + `config/schedule.yml` entry `analytics_daily_metrics_job` `30 2 * * *` timezone `Asia/Bangkok`. Verified: `ruby -c` clean (3.4.10), smoke 15/15, static scan clean, schedule no dup keys, reviewer `deleg_fecf7c61` = complete 5-key PASS (4 non-blocking suggestions: add specs, N+1 in chat_message_count, deal_outcomes dual structure, apply_tags rescue scope). +3. **phase 2 — rollup + report: ✅ DONE (2026-08-19)**. Backend: `Analytics::ReportService` (summary/timeseries/customers/agents from immutable daily-metric tables; deal_outcomes normalization for 'totals'-nested + flat) + `Analytics::DrilldownService` (deep filterable drilldown over live conversations: since/until/agent/team/inbox/channel/tag/deal + pagination; fail-closed label-join) + `Api::V2::Accounts::AnalyticsReportsController` (admin-only via ReportPolicy#view?=administrator; summary + drilldown) + routes `analytics_reports` (get :summary/:drilldown). `AccountDailyProcessor#apply_tags` now writes `deal:` tag for deal filtering. Frontend: `api/analyticsReports.js` + `AnalyticsReports.vue` (summary cards + filter bar + drilldown table) + route `analytics_reports` under reports section + `ANALYTICS_REPORTS` i18n keys in en/report.json. Verified: backend smoke 12/12, reviewers `deleg_6800b012` (backend) + `deleg_15f8447c` (frontend) = complete 5-key PASS (non-blocking i18n suggestions applied). NOTE: frontend full build (eslint/vite/vitest) cannot run locally (no node_modules) — run `pnpm install && pnpm test` before ship. +4. **phase 3 — import + persona eval + approval (2026-08-20):** + - **Weekly persona eval** ✅: `Analytics::WeeklyPersonaEvaluator` (7d via ReportService → LLM summary+recommendations; summary-only, no PII) + `persona_evaluation` endpoint + route. Reviewer `deleg_b7d63fd7` PASS. + - **Frontend product-catalog import UI** ✅: `api/productCatalog.js` + `productCatalog/Index.vue` (paste + xlsx/csv upload + result + table) + admin-only route. Reviewer `deleg_0e9f9291` PASS (i18n suggestions applied). + - **Approval flow** (user chose **LINE primary + webhook fallback**, then added **Telegram**; precedence LINE → Telegram → webhook): `Analytics::PersonaApprovalService` (`deliver` LINE quick-reply / Telegram inline keyboard / webhook fallback; `notify_approval` webhook-only w/ decision) + `Analytics::WeeklyPersonaEvaluationJob` + controller (`persona_evaluation_deliver`, `persona_approval_settings` [line_user_id + telegram_chat_id + webhook_url, conditional], `persona_decision`) + routes + schedule.yml `analytics_weekly_persona_evaluation_job` (Mon 10:00 Asia/Bangkok). First reviewer `deleg_1fe19871` FAIL-CLOSED (1 medium logic error: `:approve` path LINE-first + decision not passed) → **FIXED** (webhook-only `notify_approval(decision)`) → re-review `deleg_3052f7a4` PASS; Telegram `deleg_34116d1a` PASS. Config: `persona_line_user_id` / `persona_telegram_chat_id` / `persona_webhook_url` / `persona_webhook_secret` in Account#custom_attributes. + - **Docs** — via HANDOFF.md/plan doc; no separate docs/ convention. + - **POST-MVP design note (prompt webhook handshake) — user clarified, NOT built, awaiting LLM-server details**: a SEPARATE LLM server holds/applies the actual prompt; this app connects via webhook — (1) connect-first: pull current prompt (GET) + store snapshot; (2) approve weekly recommendation → POST new prompt to LLM server to apply. Analytics here ONLY surfaces "what customers asked" + recommendations; knowledge-prep/prompt-authoring is admin/LLM-server side. Delivery-webhook design (REST 2-endpoint vs generic action-body) DEFERRED — user wants document/design only for now. + Verified: smoke tests (eval 8/8, import 10/10 + xlsx 5/5, approval 10/10 incl. Telegram), `ruby -c` clean, node --check on .js, schedule.yml valid + unique keys, static scan clean across all. ### Verification notes / blockers - Ruby env active is 2.6.10; target 3.4.4. Rails/RSpec/RuboCop cannot be reliably claimed passed. Ruby `ruby -c` syntax + YAML validity + isolated smoke tests used instead. diff --git a/app/controllers/api/v2/accounts/analytics_reports_controller.rb b/app/controllers/api/v2/accounts/analytics_reports_controller.rb new file mode 100644 index 000000000..7b665ceec --- /dev/null +++ b/app/controllers/api/v2/accounts/analytics_reports_controller.rb @@ -0,0 +1,129 @@ +# Admin-only M2 analytics reporting endpoints (phase 2). +# +# GET /api/v2/accounts/:account_id/analytics_reports/summary +# -> Analytics::ReportService (immutable daily-metric snapshots) +# GET /api/v2/accounts/:account_id/analytics_reports/drilldown +# -> Analytics::DrilldownService (live conversations, filterable) +# +# Both are admin-role only (ReportPolicy#view? => administrator?). +class Api::V2::Accounts::AnalyticsReportsController < Api::V1::Accounts::BaseController + before_action :check_authorization + + def summary + render json: Analytics::ReportService.build( + account: Current.account, + since: date_param(:since), + until_date: date_param(:until) + ) + end + + def drilldown + render json: Analytics::DrilldownService.build( + account: Current.account, + filters: drilldown_filters + ) + end + + def persona_evaluation + result = Analytics::WeeklyPersonaEvaluator.evaluate(account: Current.account) + render json: { + summary: result.summary, + recommendations: result.recommendations, + disabled: result.disabled?, + error: result.error + } + end + + # Run the evaluation and deliver it via the configured channel (LINE / webhook). + # POST /analytics_reports/persona_evaluation/deliver + def persona_evaluation_deliver + result = Analytics::WeeklyPersonaEvaluator.evaluate(account: Current.account) + return render json: { disabled: true } if result.disabled? + + delivery = Analytics::PersonaApprovalService.deliver(account: Current.account, evaluation: result) + render json: { + summary: result.summary, + recommendations: result.recommendations, + delivery: delivery + } + end + + # Save the delivery configuration (admin LINE user id / Telegram chat id + webhook URL/secret). + # POST /analytics_reports/persona_approval_settings + # { line_user_id, telegram_chat_id, webhook_url, webhook_secret } + def persona_approval_settings + account = Current.account + account.custom_attributes['persona_line_user_id'] = params[:line_user_id] if params.key?(:line_user_id) + account.custom_attributes['persona_telegram_chat_id'] = params[:telegram_chat_id] if params.key?(:telegram_chat_id) + account.custom_attributes['persona_webhook_url'] = params[:webhook_url] if params.key?(:webhook_url) + account.custom_attributes['persona_webhook_secret'] = params[:webhook_secret] if params[:webhook_secret].present? + account.save! + render json: { success: true } + end + + # Receive an admin decision (approve / reject / view_full_prompt) — from a LINE + # callback or webhook echo. On approve, notify the configured webhook with the + # decision attached (webhook-only, NOT re-sent over LINE). + # POST /analytics_reports/persona_decision { decision } + def persona_decision + decision = params[:decision].to_s + raise ActionController::BadRequest, 'invalid decision' unless %w[approve reject view_full_prompt].include?(decision) + + if decision == 'approve' + track_approval(decision) + end + + render json: { success: true, decision: decision } + end + + private + + def track_approval(decision) + evaluation = Analytics::WeeklyPersonaEvaluator.evaluate(account: Current.account) + return if evaluation.disabled? + + Analytics::PersonaApprovalService.notify_approval( + account: Current.account, + evaluation: evaluation, + decision: decision + ) + end + + + def check_authorization + authorize :report, :view? + end + + def date_param(name) + value = params[name] + return nil if value.blank? + + Date.parse(value.to_s) + rescue ArgumentError + raise ActionController::BadRequest, "Invalid #{name} date" + end + + def drilldown_filters + { + since: date_param(:since), + until: date_param(:until), + agent_id: integer_param(:agent_id), + team_id: integer_param(:team_id), + inbox_id: integer_param(:inbox_id), + channel: params[:channel].presence, + tag: params[:tag].presence, + deal: params[:deal].presence, + page: integer_param(:page), + per_page: integer_param(:per_page) + }.compact + end + + def integer_param(name) + value = params[name] + return nil if value.blank? + + Integer(value) + rescue ArgumentError + raise ActionController::BadRequest, "Invalid #{name}" + end +end diff --git a/app/controllers/api/v2/accounts/chatbot_config_controller.rb b/app/controllers/api/v2/accounts/chatbot_config_controller.rb new file mode 100644 index 000000000..951550e15 --- /dev/null +++ b/app/controllers/api/v2/accounts/chatbot_config_controller.rb @@ -0,0 +1,30 @@ +# Admin-only chatbot configuration endpoint for the self-contained OSS chatbot. +# +# GET /api/v2/accounts/:account_id/chatbot_config -> current config +# POST /api/v2/accounts/:account_id/chatbot_config -> update allowed keys +# +# Admin-role only (ReportPolicy#view? => administrator?). Reads/writes the account's +# chatbot settings via Chatbot::ConfigService (stored in Account#custom_attributes). +class Api::V2::Accounts::ChatbotConfigController < Api::V1::Accounts::BaseController + before_action :check_authorization + + def show + render json: Chatbot::ConfigService.config(Current.account) + end + + def update + config = Chatbot::ConfigService.update!(Current.account, chatbot_config_params) + render json: config + end + + private + + def chatbot_config_params + params.permit(:chatbot_enabled, :chatbot_system_prompt, :chatbot_guardrail_prompt, + :chatbot_out_of_scope_reply, :chatbot_call_mode) + end + + def check_authorization + authorize :report, :view? + end +end diff --git a/app/controllers/api/v2/accounts/knowledge_base_faqs_controller.rb b/app/controllers/api/v2/accounts/knowledge_base_faqs_controller.rb new file mode 100644 index 000000000..535cff54f --- /dev/null +++ b/app/controllers/api/v2/accounts/knowledge_base_faqs_controller.rb @@ -0,0 +1,34 @@ +# Admin-only knowledge base management for the self-contained OSS chatbot. +# +# GET /api/v2/accounts/:account_id/knowledge_base_faqs +# -> list FAQ entries +# POST /api/v2/accounts/:account_id/knowledge_base_faqs/import +# -> import markdown text (`content`) or uploaded .md/.csv/.xlsx (`file`) +# +# Admin-role only (ReportPolicy#view? => administrator?). +class Api::V2::Accounts::KnowledgeBaseFaqsController < Api::V1::Accounts::BaseController + before_action :check_authorization + + def index + faqs = Current.account.knowledge_base_faqs.order(:title) + render json: { faqs: faqs.as_json(only: %i[id title topic_tags source_filename updated_at]) } + end + + def import + if params[:file].present? + result = KnowledgeBase::ImportService.import(account: Current.account, file_path: params[:file].tempfile.path, filename: params[:file].original_filename) + else + content = params[:content] + raise ActionController::BadRequest, 'content is required' if content.blank? + + result = KnowledgeBase::ImportService.import(account: Current.account, content: content) + end + render json: result + end + + private + + def check_authorization + authorize :report, :view? + end +end diff --git a/app/controllers/api/v2/accounts/product_catalog_entries_controller.rb b/app/controllers/api/v2/accounts/product_catalog_entries_controller.rb new file mode 100644 index 000000000..e8d8442f1 --- /dev/null +++ b/app/controllers/api/v2/accounts/product_catalog_entries_controller.rb @@ -0,0 +1,54 @@ +# Admin-only product catalog management (phase 3). +# +# GET /api/v2/accounts/:account_id/product_catalog_entries +# -> list current catalog entries +# POST /api/v2/accounts/:account_id/product_catalog_entries/import +# -> import from pasted text (copy/paste), CSV/TSV upload, or an .xlsx/.ods/.csv +# file upload (multipart `file`); text via `content` +# +# Admin-role only (ReportPolicy#view? => administrator?). +class Api::V2::Accounts::ProductCatalogEntriesController < Api::V1::Accounts::BaseController + before_action :check_authorization + + SUPPORTED_SPREADSHEET_EXTS = %w[xlsx xls ods csv].freeze + + def index + entries = Current.account.product_catalog_entries + .order(:group_name, :subgroup_name, :product_name) + render json: { entries: entries.map(&:as_json) } + end + + def import + if params[:file].present? + import_from_file(params[:file]) + else + import_from_content(params[:content]) + end + end + + private + + def import_from_content(content) + raise ActionController::BadRequest, 'content is required' if content.blank? + + result = Analytics::ProductCatalogImportService.import(account: Current.account, content: content) + render json: result + end + + def import_from_file(file) + ext = File.extname(file.original_filename.to_s).delete('.').downcase + raise ActionController::BadRequest, "unsupported file type: .#{ext}" unless SUPPORTED_SPREADSHEET_EXTS.include?(ext) + + path = file.tempfile.path + result = Analytics::ProductCatalogImportService.import_file( + account: Current.account, + file_path: path, + filename: file.original_filename + ) + render json: result + end + + def check_authorization + authorize :report, :view? + end +end diff --git a/app/javascript/dashboard/api/analyticsReports.js b/app/javascript/dashboard/api/analyticsReports.js new file mode 100644 index 000000000..4abc4921e --- /dev/null +++ b/app/javascript/dashboard/api/analyticsReports.js @@ -0,0 +1,18 @@ +/* global axios */ +import ApiClient from './ApiClient'; + +class AnalyticsReportsAPI extends ApiClient { + constructor() { + super('analytics_reports', { accountScoped: true, apiVersion: 'v2' }); + } + + getSummary(params = {}) { + return axios.get(`${this.url}/summary`, { params }); + } + + getDrilldown(params = {}) { + return axios.get(`${this.url}/drilldown`, { params }); + } +} + +export default new AnalyticsReportsAPI(); diff --git a/app/javascript/dashboard/api/productCatalog.js b/app/javascript/dashboard/api/productCatalog.js new file mode 100644 index 000000000..74d9948e8 --- /dev/null +++ b/app/javascript/dashboard/api/productCatalog.js @@ -0,0 +1,26 @@ +/* global axios */ +import ApiClient from './ApiClient'; + +class ProductCatalogAPI extends ApiClient { + constructor() { + super('product_catalog_entries', { accountScoped: true, apiVersion: 'v2' }); + } + + list() { + return axios.get(this.url, { params: { page: 1 } }); + } + + importFromContent(content) { + return axios.post(`${this.url}/import`, { content }); + } + + importFromFile(file) { + const formData = new FormData(); + formData.append('file', file); + return axios.post(`${this.url}/import`, formData, { + headers: { 'Content-Type': 'multipart/form-data' }, + }); + } +} + +export default new ProductCatalogAPI(); diff --git a/app/javascript/dashboard/i18n/locale/en/report.json b/app/javascript/dashboard/i18n/locale/en/report.json index fa1c2e176..e2e362a14 100644 --- a/app/javascript/dashboard/i18n/locale/en/report.json +++ b/app/javascript/dashboard/i18n/locale/en/report.json @@ -541,6 +541,61 @@ } } }, + "PRODUCT_CATALOG": { + "TITLE": "Product Catalog", + "DESC": "Import your product catalog so the chatbot can tag conversations with the right product.", + "PASTE_TITLE": "Paste catalog", + "IMPORT_BUTTON": "Import text", + "UPLOAD_BUTTON": "Upload .xlsx / .csv", + "IMPORTED": "imported", + "UPDATED": "updated", + "CURRENT_TITLE": "Current catalog", + "COL_GROUP": "Group", + "COL_SUBGROUP": "Subgroup", + "COL_PRODUCT": "Product", + "COL_ALIASES": "Aliases", + "EMPTY": "No products in the catalog yet.", + "LOAD_FAILED": "Failed to load catalog", + "IMPORT_FAILED": "Import failed", + "FILE_IMPORT_FAILED": "File import failed", + "FORMAT_HINT": "group | subgroup | product | display | aliases", + "FORMAT_EXAMPLE": "กลุ่มหลัก | กลุ่มย่อย | ชื่อสินค้า | ชื่อแสดง | สินค้า1,สินค้า2", + "LINE": "Line" + }, + "ANALYTICS_REPORTS": { + "HEADER": "Analytics", + "HEADER_DESC": "Self-improving chatbot insights", + "FROM": "From", + "TO": "To", + "AGENT": "Agent ID", + "TEAM": "Team ID", + "INBOX": "Inbox ID", + "CHANNEL": "Channel", + "TAG": "Tag", + "DEAL": "Deal", + "ALL_CHANNELS": "All channels", + "ALL_DEALS": "All deals", + "APPLY": "Apply", + "RESET": "Reset", + "CONVERSATIONS": "Conversations", + "MESSAGES": "Messages", + "RESOLVED": "Resolved", + "UNRESOLVED": "Unresolved", + "DEALS_WON": "Deals won", + "CONVERSATION_DRILLDOWN": "Conversation drilldown", + "NO_DATA": "No conversations match the current filters.", + "TOTAL": "{count} total", + "COL_ID": "ID", + "COL_CHANNEL": "Channel", + "COL_ASSIGNEE": "Assignee", + "COL_STATUS": "Status", + "COL_DEAL": "Deal", + "COL_MESSAGES": "Messages", + "COL_TAGS": "Tags", + "OPT_WON": "Won", + "OPT_LOST": "Lost", + "OPT_UNDECIDED": "Undecided" + }, "OVERVIEW_REPORTS": { "HEADER": "Overview", "LIVE": "Live", diff --git a/app/javascript/dashboard/routes/dashboard/settings/productCatalog/Index.vue b/app/javascript/dashboard/routes/dashboard/settings/productCatalog/Index.vue new file mode 100644 index 000000000..b100170d0 --- /dev/null +++ b/app/javascript/dashboard/routes/dashboard/settings/productCatalog/Index.vue @@ -0,0 +1,175 @@ + + +