develop
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Aakash Bakhle
|
13de83d1dc |
refactor(captain): route conversation completion by feature (#15317)
Conversation completion evaluations now use a dedicated internal LLM feature with GPT 4.1 as the default. The internal route keeps the completion model separate from the installation wide Captain model override and from the assistant route, which can use GPT 5.2 for Captain V2 accounts. Evaluations continue to use the installation API key and do not consume Captain response credits. ## What changed Added an internal `conversation_completion` feature to the LLM model config and excluded internal features from account preferences, the Captain settings API, and Super Admin model overrides. Updated `Captain::ConversationCompletionService` to resolve its model through `Llm::FeatureRouter`. Added focused service and request coverage for model routing and settings visibility. |
||
|
Aakash Bakhle
|
e10b236871 |
feat(captain): group conversation FAQ signals (2/4) (#14978)
Captain can now turn resolved conversations into FAQ suggestions that people can review. When a human support agent gives a reusable answer, Captain saves the question and answer as an observation. Captain groups matching observations into one suggestion instead of creating a pending FAQ for every conversation. This PR does not approve or publish FAQs. PR [#14977](https://github.com/chatwoot/chatwoot/pull/14977) adds the data model and should be reviewed first. The controller and UI PRs will add the review flow. The three PRs should merge together. ## Closes [CW-7495](https://linear.app/chatwoot/issue/CW-7495/backend-llm-changes-to-make-conversation-faqs-as-signalssuggestions). This is PR 2 of 3. The issue is complete after the full stack lands. ## What changed 1. Runs FAQ generation in the low priority queue after a conversation is resolved. 2. Reads only customer messages and answers written by human support agents. 3. Uses the assistant's product details, instructions, response rules, and guardrails to reject spam and unrelated conversations. 4. Stores each reusable question and answer as an observation. 5. Uses exact text similarity search within the conversation language to find likely matches. 6. Asks the LLM whether both FAQs ask the same question and give the same answer. 7. Does not create a new suggestion when an approved FAQ already covers the observation. 8. Adds matching observations to an existing open suggestion and updates its source count. 9. Does not suggest the same FAQ again after someone dismisses it. 10. Creates a new open suggestion only when no approved FAQ or existing suggestion covers the observation. ## How to test 1. Resolve a conversation where a human support agent gives a reusable answer. Confirm that Captain creates one open suggestion with one source. 2. Resolve another conversation with the same question and answer. Confirm that Captain adds a source to the existing suggestion instead of creating another suggestion. 3. Resolve a conversation that is already covered by an approved FAQ. Confirm that Captain creates no new suggestion. 4. Dismiss a suggestion, then resolve another conversation with the same question and answer. Confirm that Captain does not suggest the FAQ again. 5. Resolve a spam or unrelated conversation. Confirm that Captain creates no suggestion. --------- Co-authored-by: Sony Mathew <sony@chatwoot.com> Co-authored-by: Sivin Varghese <64252451+iamsivin@users.noreply.github.com> |
||
|
Aakash Bakhle
|
d57354c8b5 |
feat: tighten conversation FAQ generation prompt (#14957)
Tightens the resolved-conversation FAQ generator so it only proposes durable, reusable FAQ candidates supported by human support-agent messages. The implementation now sends a conversation-FAQ-specific transcript to the LLM: customer messages plus real human support-agent messages only, excluding bot, private, activity, and template messages. ## Closes - https://linear.app/chatwoot/issue/CW-7494/tighten-conversation-faq-generation-prompt ## What changed - Added a human-only transcript builder in `ConversationFaqService` instead of using the generic `conversation.to_llm_text` output. - Excluded bot/agent-bot messages before the LLM call, which removes the main bot-line leakage class deterministically. - Preserved native-channel human replies where outgoing messages are stored as `external_echo` without a `User` sender. - Kept a prompt decision gate requiring each FAQ to be backed by a complete public human-agent answer. - Added generic no-FAQ classes for spam, wrong-service conversations, private account/payment/order/certificate/troubleshooting cases, support workflow mechanics, and direct-link/file/quote outputs. - Added a separate `conversation_faq_generation` model route defaulting to `gpt-5.2`, while keeping `document_faq_generation` on its existing `gpt-4.1-mini` default. Conversation FAQ generation passes that feature default ahead of the legacy global `CAPTAIN_OPEN_AI_MODEL` setting unless an account-level override is configured. - Kept the prompt domain-neutral so it can still generate reusable product, service, policy, setup, and process FAQs outside SaaS contexts. ## Sampling notes - Production Langfuse traces showed `llm.captain.conversation_faq` calls using `gpt-4.1` in the sampled account set. - Locally, `Llm::FeatureRouter.resolve(feature: 'conversation_faq_generation')` now resolves to `gpt-5.2`. - Reviewed recent production `llm.captain.conversation_faq` traces across 13+ accounts in compact form. - Replayed 20 full traces across 10 accounts/domains, including education, hosting, retail/auto, APIs, logistics, tax/fiscal workflows, and Chatwoot account 1. - Explicit `gpt-5.2` replay with human-only conversation history returned no FAQ for 15/20 traces. - A comparison replay with `gpt-4.1-mini` returned no FAQ for only 7/20 traces, bringing back several private/order/payment/support-workflow cases. - Remaining non-empty `gpt-5.2` outputs are now mostly borderline/possibly useful human-agent-derived FAQs rather than obvious bot-sourced answers. ## How to test - Resolve conversations where the answer came only from the bot; no pending FAQ should be generated. - Resolve spam, unrelated, wrong-service, or private payment/order/account conversations; no pending FAQ should be generated. - Resolve conversations that require account/order/payment/login/private verification or a human handoff; no pending FAQ should be generated. - Resolve a conversation where a human agent gives a stable, reusable help-center answer; the generated pending FAQ should be general and self-contained. |
||
|
Sony Mathew
|
cf134deb37 |
fix: Preserve Captain LLM defaults (#14858)
# Pull Request Template ## Description Adjusts the Captain LLM feature defaults after feature routing so the defaults stay intentional and avoid unintended high-cost model upgrades. Assistant, copilot, and onboarding content generation now default to `gpt-4.1`; audio transcription keeps `gpt-4o-mini-transcribe` as the default while exposing `whisper-1` as an available account override option. Related: https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [x] Bug fix (non-breaking change which fixes an issue) ## How Has This Been Tested? - `ruby -ryaml -e 'yaml = YAML.load_file("config/llm.yml"); features = yaml.fetch("features"); models = yaml.fetch("models"); features.each { |name, cfg| missing = Array(cfg["models"]) - models.keys; raise "#{name}: missing #{missing.join(",")}" if missing.any?; raise "#{name}: default not in models" unless Array(cfg["models"]).include?(cfg["default"]) }'` - `node -e "JSON.parse(require('fs').readFileSync('app/javascript/dashboard/i18n/locale/en/settings.json', 'utf8'))"` - `pnpm exec prettier --check app/javascript/dashboard/i18n/locale/en/settings.json` - `bundle exec rspec spec/lib/llm/feature_router_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/models/concerns/agentable_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/models/account_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/controllers/super_admin/accounts_controller_spec.rb` - `bundle exec rspec spec/enterprise/services/captain/llm/article_translation_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb` - `RUBOCOP_CACHE_ROOT=/private/tmp/rubocop_cache bundle exec rubocop enterprise/app/services/captain/llm/article_translation_service.rb enterprise/app/services/messages/audio_transcription_service.rb spec/enterprise/services/captain/llm/article_translation_service_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony Mathew
|
4e26c5b4bb |
feat: Route system LLM jobs (4/6) (#14843)
## Description Routes the remaining system-only and legacy-sensitive LLM jobs through feature-level model configuration, while preserving system credential usage and usage-accounting behavior. This adds dedicated defaults for help center article generation, onboarding content generation, query translation, transcription, and search embeddings so these flows can be configured per account without falling back to installation-wide model settings. Fixes https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? Verified the feature routing defaults and account overrides for the touched Captain/system LLM paths, including the legacy OpenAI transcription and paginated FAQ services. - `eval "$(rbenv init -)" && bundle exec rspec spec/lib/captain/base_task_service_spec.rb spec/lib/llm/models_spec.rb spec/models/concerns/captain_featurable_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/enterprise/services/captain/llm/paginated_faq_generator_service_spec.rb spec/enterprise/services/captain/llm/pdf_processing_service_spec.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/enterprise/services/onboarding/help_center_article_builder_spec.rb spec/enterprise/services/captain/onboarding/website_analyzer_service_spec.rb` - `eval "$(rbenv init -)" && bundle exec rubocop app/controllers/api/v1/accounts/captain/preferences_controller.rb app/models/concerns/account_settings_schema.rb lib/captain/base_task_service.rb enterprise/app/services/captain/llm/article_translation_service.rb enterprise/app/services/captain/llm/article_writer_service.rb enterprise/app/services/captain/llm/embedding_service.rb enterprise/app/services/captain/llm/help_center_curation_service.rb enterprise/app/services/captain/llm/paginated_faq_generator_service.rb enterprise/app/services/captain/llm/translate_query_service.rb enterprise/app/services/captain/llm/widget_tagline_service.rb enterprise/app/services/captain/onboarding/website_analyzer_service.rb enterprise/app/services/messages/audio_transcription_service.rb spec/enterprise/services/messages/audio_transcription_service_spec.rb spec/lib/captain/base_task_service_spec.rb` - `ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); %w[document_faq_generation help_center_article_generation onboarding_content_generation help_center_query_translation audio_transcription help_center_search].each { |feature| abort(%(missing #{feature})) unless config.dig('features', feature) }; abort('wrong article default') unless config.dig('features', 'help_center_article_generation', 'default') == 'gpt-5.2'; puts 'llm.yml ok'"` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony Mathew
|
43e0f8a141 |
feat: Route Enterprise LLM services (3/6) (#14841)
# Pull Request Template ## Description Routes Enterprise assistant, copilot, FAQ, contact memory, action-classifier, and false-promise detector LLM paths through feature-specific model resolution. `Llm::BaseAiService` now accepts feature/account context and uses `Llm::FeatureRouter` when that context is present, while retaining the installation-model fallback for unmigrated callers. This also adds a `document_faq_generation` feature default for generative FAQ/document content. Linear: https://linear.app/chatwoot/issue/CW-7425/test-new-models Depends on #14840 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? - `bundle exec rspec spec/lib/llm/models_spec.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with 112 examples, 0 failures. - `bundle exec rspec spec/models/concerns/captain_featurable_spec.rb spec/models/account_spec.rb spec/controllers/api/v1/accounts/captain/preferences_controller_spec.rb spec/lib/llm/feature_router_spec.rb` passed with 87 examples, 0 failures. - `bundle exec rubocop enterprise/app/services/llm/base_ai_service.rb enterprise/app/services/captain/copilot/chat_service.rb enterprise/app/services/captain/llm/assistant_chat_service.rb enterprise/app/services/captain/llm/faq_generator_service.rb enterprise/app/services/captain/llm/conversation_faq_service.rb enterprise/app/services/captain/llm/contact_notes_service.rb enterprise/app/services/captain/llm/contact_attributes_service.rb enterprise/app/services/captain/llm/assistant_action_classifier_service.rb enterprise/app/services/captain/llm/assistant_false_promise_service.rb spec/enterprise/services/llm/base_ai_service_spec.rb spec/enterprise/services/captain/copilot/chat_service_spec.rb spec/enterprise/services/captain/llm/assistant_chat_service_spec.rb spec/enterprise/services/captain/llm/faq_generator_service_spec.rb spec/enterprise/services/captain/llm/conversation_faq_service_spec.rb spec/enterprise/services/captain/llm/assistant_action_classifier_service_spec.rb spec/enterprise/services/captain/llm/assistant_false_promise_service_spec.rb spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb` passed with no offenses. - `bundle exec ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); abort('missing document_faq_generation') unless config.dig('features', 'document_faq_generation'); abort('missing default') unless config.dig('features', 'document_faq_generation', 'default'); puts 'llm.yml ok'"` passed. - `git diff --check` passed. ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Sony Mathew
|
8b977b35a8 |
feat: Add LLM feature router (1/6) (#14839)
## Description Adds the foundation for feature-specific LLM model routing so Captain AI features can resolve their effective provider/model from code defaults and account-level overrides. This fixes the provider metadata key in `config/llm.yml`, adds `Llm::FeatureRouter`, and routes existing `CaptainFeaturable` model defaults through the shared resolver. Fixes https://linear.app/chatwoot/issue/CW-7425/test-new-models ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality not to work as expected) - [ ] This change requires a documentation update ## How Has This Been Tested? - `bundle exec rspec spec/lib/llm/models_spec.rb spec/lib/llm/feature_router_spec.rb spec/models/concerns/captain_featurable_spec.rb` - 23 examples, 0 failures - `bundle exec rubocop lib/llm/models.rb lib/llm/feature_router.rb app/models/concerns/captain_featurable.rb spec/lib/llm/models_spec.rb spec/lib/llm/feature_router_spec.rb spec/models/concerns/captain_featurable_spec.rb` - no offenses - `bundle exec ruby -e "require 'yaml'; config = YAML.load_file('config/llm.yml'); abort('missing providers') unless config['providers']; abort('missing models') unless config['models']; abort('missing features') unless config['features']; puts 'llm.yml ok'"` - `git diff --check` ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [ ] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules |
||
|
Shivam Mishra
|
34b42a1ce1 |
feat: add global config for captain settings (#13141)
Co-authored-by: aakashb95 <aakashbakhle@gmail.com> Co-authored-by: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com> |