From 342f0a399c11d0a71c1ae2fcbea78391dca66cab Mon Sep 17 00:00:00 2001 From: Aakash Bakhle <48802744+aakashb95@users.noreply.github.com> Date: Wed, 5 Aug 2026 10:29:43 +0530 Subject: [PATCH] fix(captain): resolve V2 FAQ citations from trusted sources (#15159) Captain V2 now adds FAQ citations from a structured model response. The model returns ordered response parts with citation indexes, and Chatwoot turns only trusted indexes into customer links. ## Before Captain V2 asked the model to copy text markers such as `[[faq:1]]`. Chatwoot used one regular expression to replace those markers with links in the outgoing message and another regular expression to remove the rendered links before the next model turn. Long conversations depended on parsing the customer message to recover plain model context. ## After The FAQ lookup tool now gives each eligible source document a numeric index and never gives the model a URL. FAQ results from the same document reuse the same index. The model returns `response_parts`, where each part contains customer text and the supporting citation indexes. Chatwoot checks every index against the document IDs registered during the current run. Only stored HTTP or HTTPS web-document links without embedded credentials can appear in the customer reply. Blank links, PDF sources, attachments, non-HTTP storage links, and unknown indexes do not create links. Sources receive display numbers in the order they first appear, and repeated sources keep the same display number. Chatwoot saves the structured response parts with each newly generated Captain message. Later Captain V2 turns use the saved plain text for those messages, so they never need to parse rendered links. Existing messages remain unchanged and continue to use their stored content. When citations are disabled, Chatwoot clears citation indexes before it returns or saves the response. Captain V1, Copilot, legacy prompts, legacy tools, and the playground response contract are unchanged. The playground continues to show the plain `response` field. ## Closes [AI-138](https://linear.app/chatwoot/issue/AI-138/faq-citation-fix) ## How to test 1. Open a conversation handled by a Captain V2 assistant and turn citations off. Ask a greeting, an FAQ question, a code question, and a follow up question. Confirm that the assistant answers normally and shows no source links. 2. Turn citations on and ask a question that matches one public web document. Confirm that the reply shows the stored public link after the supported text. 3. Ask a question that needs two public web documents. Confirm that the response order stays correct, each link appears after the supported text, and repeated sources keep the same display number. 4. Ask a question that retrieves several FAQ results from one document. Confirm that the reply shows the document once at each supported response part rather than exposing separate FAQ sources. 5. Ask a question supported by a PDF, attachment, blank link, or non-HTTP storage link. Confirm that Captain can use the information but does not show a customer link. 6. Ask for a fenced code example with a citation. Confirm that the code block stays complete and the citation appears after the closing fence. 7. Continue the conversation with a follow up question. Confirm that Captain uses the earlier plain response text and does not receive or repeat rendered citation links. 8. Test a scenario handoff in a conversation. Confirm that the handoff and final response still work. --- .../message/CaptainGenerationDetails.vue | 13 +- ...dd_cited_document_ids_to_agent_sessions.rb | 5 + ...ex_on_agent_sessions_cited_document_ids.rb | 7 + ...0002_add_used_faq_ids_to_agent_sessions.rb | 5 + ...dd_index_on_agent_sessions_used_faq_ids.rb | 7 + db/schema.rb | 6 +- .../captain/agent_sessions_controller.rb | 8 +- .../captain/conversation/message_builder.rb | 13 +- .../app/models/captain/agent_session.rb | 38 +-- enterprise/app/models/captain/assistant.rb | 21 ++ .../app/models/captain/assistant_response.rb | 4 + enterprise/app/models/captain/document.rb | 35 +++ enterprise/app/models/captain/scenario.rb | 1 + .../captain/assistant/agent_run_response.rb | 67 +++++ .../captain/assistant/agent_runner_service.rb | 54 +--- .../captain/assistant/response_parts.rb | 70 ++++++ .../captain/assistant/response_rewriter.rb | 57 +++-- .../assistant/session_capture_service.rb | 14 ++ .../message_history_builder_service.rb | 13 +- .../captain/agent_sessions/show.json.jbuilder | 8 +- .../lib/captain/prompts/assistant.liquid | 2 + .../lib/captain/prompts/scenario.liquid | 2 + .../captain/prompts/snippets/citations.liquid | 4 + enterprise/lib/captain/response_schema.rb | 12 +- .../lib/captain/tools/faq_lookup_tool.rb | 42 ++-- .../captain/agent_sessions_controller_spec.rb | 30 +-- .../captain/assistants_controller_spec.rb | 8 +- .../conversation/response_builder_job_spec.rb | 230 +++++++++++++++++- .../lib/captain/prompt_renderer_spec.rb | 11 + .../lib/captain/tools/faq_lookup_tool_spec.rb | 64 ++++- .../models/captain/agent_session_spec.rb | 4 +- .../models/captain/document_spec.rb | 32 +++ .../models/concerns/agentable_spec.rb | 13 + .../assistant/agent_runner_service_spec.rb | 189 +++++++++++++- .../assistant/session_capture_service_spec.rb | 49 ++++ 35 files changed, 996 insertions(+), 142 deletions(-) create mode 100644 db/migrate/20260804000000_add_cited_document_ids_to_agent_sessions.rb create mode 100644 db/migrate/20260804000001_add_index_on_agent_sessions_cited_document_ids.rb create mode 100644 db/migrate/20260804000002_add_used_faq_ids_to_agent_sessions.rb create mode 100644 db/migrate/20260804000003_add_index_on_agent_sessions_used_faq_ids.rb create mode 100644 enterprise/app/services/captain/assistant/agent_run_response.rb create mode 100644 enterprise/app/services/captain/assistant/response_parts.rb create mode 100644 enterprise/lib/captain/prompts/snippets/citations.liquid diff --git a/app/javascript/dashboard/components-next/message/CaptainGenerationDetails.vue b/app/javascript/dashboard/components-next/message/CaptainGenerationDetails.vue index 752616109..85e1fdf2d 100644 --- a/app/javascript/dashboard/components-next/message/CaptainGenerationDetails.vue +++ b/app/javascript/dashboard/components-next/message/CaptainGenerationDetails.vue @@ -37,6 +37,11 @@ const isLoading = computed( ); const citations = computed(() => session.value?.citations || []); +const usedFaqs = computed(() => session.value?.usedFaqs || []); +const knowledgeSources = computed(() => [ + ...citations.value.map(source => ({ ...source, type: 'document' })), + ...usedFaqs.value.map(source => ({ ...source, type: 'faq' })), +]); const scenarioTitles = computed(() => (session.value?.scenarios || []).reduce((map, scenario) => { @@ -289,7 +294,7 @@ const onPopoverHide = () => { -
[1]
") + expect(rendered_content).to include('Continue with the next step.') + end + + it 'preserves response part order and numbers trusted sources by first appearance' do + v2_response_parts.replace( + [ + { 'text' => 'Email changes are in settings.', 'citation_indexes' => [2] }, + { 'text' => 'Password resets use the reset link.', 'citation_indexes' => [1] } + ] + ) + + described_class.perform_now(conversation, assistant) + + expect(conversation.messages.outgoing.last.content).to eq( + "Email changes are in settings. [[1](https://help.example.com/email)]\n\n" \ + 'Password resets use the reset link. [[2](https://help.example.com/password)]' + ) + end + + it 'reuses a display number when a response cites the same source more than once' do + v2_response_parts.replace( + [ + { 'text' => 'Use the reset link.', 'citation_indexes' => [1, 1] }, + { 'text' => 'The same page has the next steps.', 'citation_indexes' => [1] } + ] + ) + + described_class.perform_now(conversation, assistant) + + expect(conversation.messages.outgoing.last.content).to eq( + "Use the reset link. [[1](https://help.example.com/password)]\n\n" \ + 'The same page has the next steps. [[1](https://help.example.com/password)]' + ) + end + + it 'ignores unknown indexes and non-numeric citation values' do + v2_response_parts.first['citation_indexes'] = [99, 'https://model.example/source'] + + described_class.perform_now(conversation, assistant) + + expect(conversation.messages.outgoing.last.content).to eq('Hey, welcome to Captain V2') + end + + it 'renders a wrong but valid index only through its trusted source mapping' do + v2_response_parts.first['citation_indexes'] = [2, 'https://model.example/source'] + + described_class.perform_now(conversation, assistant) + + content = conversation.messages.outgoing.last.content + expect(content).to eq('Hey, welcome to Captain V2 [[1](https://help.example.com/email)]') + expect(content).not_to include('model.example') + end + + it 'does not render mapped PDF or non-public sources' do + pdf_document = create( + :captain_document, + assistant: assistant, + external_link: 'https://storage.example.com/private-file.pdf?token=secret' + ) + citation_sources = mock_agent_runner_service.last_run_result.context[:state][Captain::Assistant::CITATION_SOURCES_STATE_KEY] + citation_sources[3] = pdf_document.id + citation_sources[4] = 0 + v2_response_parts.first['citation_indexes'] = [3, 4] + + described_class.perform_now(conversation, assistant) + + expect(conversation.messages.outgoing.last.content).to eq('Hey, welcome to Captain V2') + end + + it 'joins valid markdown parts and drops blank or invalid parts' do + v2_response_parts.replace( + [ + { 'text' => " **Steps**\n\n- Open settings ", 'citation_indexes' => [] }, + { 'text' => ' ', 'citation_indexes' => [1] }, + 'invalid', + { 'text' => 'Save the change.', 'citation_indexes' => ['1'] } + ] + ) + + described_class.perform_now(conversation, assistant) + + message = conversation.messages.outgoing.last + expect(message.content).to eq("**Steps**\n\n- Open settings\n\nSave the change.") + expect(message.additional_attributes[Captain::Assistant::ResponseParts::MESSAGE_ATTRIBUTE_KEY]).to eq( + [ + { 'text' => "**Steps**\n\n- Open settings", 'citation_indexes' => [] }, + { 'text' => 'Save the change.', 'citation_indexes' => [] } + ] + ) + end + end + it 'emits a response completed event' do expect(Captain::ConversationEvents).to receive(:response_completed) .with(conversation: conversation, assistant: assistant, message: kind_of(Message), at: kind_of(Time)) @@ -653,6 +870,7 @@ RSpec.describe Captain::Conversation::ResponseBuilderJob, type: :job do llm_model: 'openai-gpt-5.2', credits_consumed: 1.0, faq_ids: [7, 9], + cited_document_ids: [], document_ids: [3], scenario_ids: [], user_id: nil diff --git a/spec/enterprise/lib/captain/prompt_renderer_spec.rb b/spec/enterprise/lib/captain/prompt_renderer_spec.rb index 910319253..9898ad8da 100644 --- a/spec/enterprise/lib/captain/prompt_renderer_spec.rb +++ b/spec/enterprise/lib/captain/prompt_renderer_spec.rb @@ -95,6 +95,17 @@ RSpec.describe Captain::PromptRenderer do expect(result).to include('Liquid error') end + + it 'renders structured citation instructions only when citations are enabled' do + allow(File).to receive(:read).with(template_path).and_return("{% render 'citations', citation_enabled: citation_enabled %}") + + enabled_result = described_class.render(template_name, { citation_enabled: true }) + disabled_result = described_class.render(template_name, { citation_enabled: false }) + + expect(enabled_result).to include('citation_indexes', 'numeric citation index') + expect(enabled_result).not_to include('[[faq:') + expect(disabled_result).to be_blank + end end describe '.load_template' do diff --git a/spec/enterprise/lib/captain/tools/faq_lookup_tool_spec.rb b/spec/enterprise/lib/captain/tools/faq_lookup_tool_spec.rb index dc2b9f6c6..869b16e64 100644 --- a/spec/enterprise/lib/captain/tools/faq_lookup_tool_spec.rb +++ b/spec/enterprise/lib/captain/tools/faq_lookup_tool_spec.rb @@ -47,10 +47,13 @@ RSpec.describe Captain::Tools::FaqLookupTool, type: :model do assistant: assistant, question: 'How to change email?', answer: 'Go to settings and update email', + documentable: document, status: 'approved') end before do + allow(Resolv).to receive(:getaddresses).and_return(['93.184.216.34']) + # Mock nearest_neighbors to return our test responses allow(Captain::AssistantResponse).to receive(:nearest_neighbors).and_return( Captain::AssistantResponse.where(id: [response1.id, response2.id]) @@ -64,14 +67,61 @@ RSpec.describe Captain::Tools::FaqLookupTool, type: :model do expect(result).to include('Answer: Click on forgot password link') expect(result).to include('Question: How to change email?') expect(result).to include('Answer: Go to settings and update email') + expect(result).not_to include('Citation index:') end - it 'includes source link when document has external_link' do + it 'does not assign indexes to attachments or storage links' do + assistant.update!(config: assistant.config.merge('feature_citation' => true)) + document.pdf_file.attach( + io: StringIO.new('PDF content'), + filename: 'private-file.pdf', + content_type: 'application/pdf' + ) + + pdf_result = tool.perform(tool_context, query: 'private document') + + expect(pdf_result).not_to include('Citation index:') + expect(tool_context.state[Captain::Assistant::CITATION_SOURCES_STATE_KEY]).to be_nil + + document.pdf_file.detach + document.update!(external_link: 's3://private-bucket/password') + + storage_result = tool.perform(tool_context, query: 'private storage document') + + expect(storage_result).not_to include('Citation index:') + expect(tool_context.state[Captain::Assistant::CITATION_SOURCES_STATE_KEY]).to be_nil + + document.update!(external_link: 'https://storage.example.com/private-file.pdf?token=secret') + + pdf_url_result = tool.perform(tool_context, query: 'private PDF URL') + + expect(pdf_url_result).not_to include('Citation index:') + expect(tool_context.state[Captain::Assistant::CITATION_SOURCES_STATE_KEY]).to be_nil + end + + it 'assigns stable numeric indexes to customer-visible document links' do + assistant.update!(config: assistant.config.merge('feature_citation' => true)) document.update!(external_link: 'https://help.example.com/password') result = tool.perform(tool_context, query: 'password') + repeated_result = tool.perform(tool_context, query: 'password again') - expect(result).to include('Source: https://help.example.com/password') + expect(result).to include('Citation index: 1') + expect(repeated_result).to include('Citation index: 1') + expect(result).not_to include('https://help.example.com/password') + expect(result.scan('Citation index: 1').size).to eq(2) + expect(result).not_to include('Citation index: 2') + expect(tool_context.state[Captain::Assistant::CITATION_SOURCES_STATE_KEY]).to eq(1 => document.id) + end + + it 'does not cite document URLs containing credentials' do + assistant.update!(config: assistant.config.merge('feature_citation' => true)) + document.update!(external_link: 'https://user:pass@help.example.com/password') + + result = tool.perform(tool_context, query: 'password') + + expect(result).not_to include('Citation index:') + expect(tool_context.state[Captain::Assistant::CITATION_SOURCES_STATE_KEY]).to be_nil end it 'logs tool usage for search' do @@ -84,7 +134,15 @@ RSpec.describe Captain::Tools::FaqLookupTool, type: :model do it 'records retrieved faq ids and document ids into Chatwoot metadata' do tool.perform(tool_context, query: 'password reset') - expect(tool_context.state.dig(:cw_metadata, :faq_ids)).to contain_exactly(response1.id, response2.id) + user = create(:user, account: account) + user_faq = create(:captain_assistant_response, assistant: assistant, documentable: user, status: :approved) + allow(Captain::AssistantResponse).to receive(:nearest_neighbors).and_return( + Captain::AssistantResponse.where(id: user_faq.id) + ) + tool.perform(tool_context, query: 'user faq') + + expect(tool_context.state.dig(:cw_metadata, :faq_ids)).to contain_exactly(response1.id, response2.id, user_faq.id) + expect(tool_context.state.dig(:cw_metadata, :used_faq_ids)).to contain_exactly(user_faq.id) expect(tool_context.state.dig(:cw_metadata, :document_ids)).to contain_exactly(document.id) end diff --git a/spec/enterprise/models/captain/agent_session_spec.rb b/spec/enterprise/models/captain/agent_session_spec.rb index b4306a11e..bd71f069f 100644 --- a/spec/enterprise/models/captain/agent_session_spec.rb +++ b/spec/enterprise/models/captain/agent_session_spec.rb @@ -129,10 +129,12 @@ RSpec.describe Captain::AgentSession, type: :model do end describe 'defaults' do - it 'defaults faq_ids, document_ids, scenario_ids and run_context' do + it 'defaults source ids, scenario ids and run_context' do session = create(:captain_agent_session, account: account, assistant: assistant) expect(session.faq_ids).to eq([]) + expect(session.used_faq_ids).to eq([]) + expect(session.cited_document_ids).to eq([]) expect(session.document_ids).to eq([]) expect(session.scenario_ids).to eq([]) expect(session.run_context).to eq({}) diff --git a/spec/enterprise/models/captain/document_spec.rb b/spec/enterprise/models/captain/document_spec.rb index af6c1cb58..04779dc6b 100644 --- a/spec/enterprise/models/captain/document_spec.rb +++ b/spec/enterprise/models/captain/document_spec.rb @@ -15,6 +15,38 @@ RSpec.describe Captain::Document, type: :model do end end + describe '#customer_visible_source_url' do + let(:document) do + build(:captain_document, assistant: assistant, account: account, external_link: 'https://example.com/runbooks') + end + + it 'returns URLs that resolve only to public addresses' do + allow(Resolv).to receive(:getaddresses).with('example.com').and_return(['93.184.216.34']) + + expect(document.customer_visible_source_url).to eq('https://example.com/runbooks') + end + + it 'rejects URLs that resolve to private, loopback, or link-local addresses' do + ['10.0.0.5', '127.0.0.1', '169.254.1.1', 'fc00::1', 'fe80::1'].each do |address| + allow(Resolv).to receive(:getaddresses).with('example.com').and_return([address]) + + expect(document.customer_visible_source_url).to be_nil + end + end + + it 'rejects URLs with mixed public and private addresses' do + allow(Resolv).to receive(:getaddresses).with('example.com').and_return(['93.184.216.34', '10.0.0.5']) + + expect(document.customer_visible_source_url).to be_nil + end + + it 'rejects URLs with unresolved hosts' do + allow(Resolv).to receive(:getaddresses).with('example.com').and_return([]) + + expect(document.customer_visible_source_url).to be_nil + end + end + describe 'PDF support' do let(:pdf_document) do doc = build(:captain_document, assistant: assistant, account: account) diff --git a/spec/enterprise/models/concerns/agentable_spec.rb b/spec/enterprise/models/concerns/agentable_spec.rb index f2145ff85..c423c51ec 100644 --- a/spec/enterprise/models/concerns/agentable_spec.rb +++ b/spec/enterprise/models/concerns/agentable_spec.rb @@ -198,6 +198,19 @@ RSpec.describe Concerns::Agentable do it 'returns Captain::ResponseSchema' do expect(dummy_instance.send(:agent_response_schema)).to eq(Captain::ResponseSchema) end + + it 'defines complete structured response parts with nested citation indexes' do + schema = Captain::ResponseSchema.new.to_json_schema[:schema] + response_parts = schema.dig(:properties, :response_parts) + response_part = response_parts.dig(:items, :properties) + + expect(schema[:required]).to contain_exactly(:response_parts, :reasoning) + expect(response_parts).to include(type: 'array', minItems: 1) + expect(response_part.dig(:text, :type)).to eq('string') + expect(response_part.dig(:citation_indexes, :items, :type)).to eq('integer') + expect(response_part.dig(:citation_indexes, :items, :minimum)).to eq(1) + expect(schema[:properties]).not_to have_key(:response) + end end describe 'required methods' do diff --git a/spec/enterprise/services/captain/assistant/agent_runner_service_spec.rb b/spec/enterprise/services/captain/assistant/agent_runner_service_spec.rb index 494d60dbc..093cbccf4 100644 --- a/spec/enterprise/services/captain/assistant/agent_runner_service_spec.rb +++ b/spec/enterprise/services/captain/assistant/agent_runner_service_spec.rb @@ -13,7 +13,16 @@ RSpec.describe Captain::Assistant::AgentRunnerService do let(:mock_runner) { instance_double(Agents::AgentRunner) } let(:mock_agent) { instance_double(Agents::Agent) } let(:mock_scenario_agent) { instance_double(Agents::Agent) } - let(:mock_result) { instance_double(Agents::RunResult, output: { 'response' => 'Test response' }, context: nil) } + let(:mock_result) do + instance_double( + Agents::RunResult, + output: { + 'response_parts' => [{ 'text' => 'Test response', 'citation_indexes' => [] }], + 'reasoning' => 'Test reasoning' + }, + context: nil + ) + end let(:message_history) do [ @@ -185,7 +194,13 @@ RSpec.describe Captain::Assistant::AgentRunnerService do it 'processes and formats agent result' do result = service.generate_response(message_history: message_history) - expect(result).to eq({ 'response' => 'Test response', 'agent_name' => nil, 'handoff_tool_called' => false }) + expect(result).to eq({ + 'response_parts' => [{ 'text' => 'Test response', 'citation_indexes' => [] }], + 'response' => 'Test response', + 'reasoning' => 'Test reasoning', + 'agent_name' => nil, + 'handoff_tool_called' => false + }) end it 'exposes the raw run result via last_run_result' do @@ -197,7 +212,14 @@ RSpec.describe Captain::Assistant::AgentRunnerService do context 'when handoff tool was called during agent execution' do let(:runner_context) { { captain_v2_handoff_tool_called: true } } let(:mock_result) do - instance_double(Agents::RunResult, output: { 'response' => 'Let me connect you' }, context: runner_context) + instance_double( + Agents::RunResult, + output: { + 'response_parts' => [{ 'text' => 'Let me connect you', 'citation_indexes' => [] }], + 'reasoning' => 'A human is needed' + }, + context: runner_context + ) end it 'includes handoff_tool_called flag in response' do @@ -205,6 +227,8 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq({ 'response' => 'Let me connect you', + 'response_parts' => [{ 'text' => 'Let me connect you', 'citation_indexes' => [] }], + 'reasoning' => 'A human is needed', 'agent_name' => nil, 'handoff_tool_called' => true }) @@ -234,6 +258,7 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq({ 'response' => 'Simple string response', + 'response_parts' => [{ 'text' => 'Simple string response', 'citation_indexes' => [] }], 'reasoning' => 'Processed by agent', 'agent_name' => nil, 'handoff_tool_called' => false @@ -241,6 +266,149 @@ RSpec.describe Captain::Assistant::AgentRunnerService do end end + context 'when structured response parts contain invalid values' do + let(:assistant) { create(:captain_assistant, account: account, config: { 'feature_citation' => true }) } + let(:mock_result) do + instance_double( + Agents::RunResult, + output: { + 'response_parts' => [ + { 'text' => ' First part ', 'citation_indexes' => [2, 2, 0, '1', 'https://model.example/source'] }, + { 'text' => ' ', 'citation_indexes' => [1] }, + 'invalid', + { 'text' => 'Second part', 'citation_indexes' => [1] } + ], + 'reasoning' => 'Test reasoning' + }, + context: nil + ) + end + + it 'keeps valid ordered parts and derives the compatibility response string' do + result = service.generate_response(message_history: message_history) + + expect(result['response_parts']).to eq( + [ + { 'text' => 'First part', 'citation_indexes' => [2] }, + { 'text' => 'Second part', 'citation_indexes' => [1] } + ] + ) + expect(result['response']).to eq("First part\n\nSecond part") + end + end + + context 'when citations are disabled' do + let(:mock_result) do + instance_double( + Agents::RunResult, + output: { + 'response_parts' => [{ 'text' => 'FAQ-backed response', 'citation_indexes' => [1] }], + 'reasoning' => 'Test reasoning' + }, + context: nil + ) + end + + it 'discards model-provided citation indexes' do + result = service.generate_response(message_history: message_history) + + expect(result['response_parts']).to eq( + [{ 'text' => 'FAQ-backed response', 'citation_indexes' => [] }] + ) + end + end + + context 'when a structured response exceeds the channel limit' do + let(:channel_limit_rewrite) do + original_model_output = { + 'response_parts' => [ + { 'text' => 'The first answer has more detail than this channel can accept.', 'citation_indexes' => [2] }, + { 'text' => 'The second answer also has more detail than this channel can accept.', 'citation_indexes' => [1] } + ], + 'reasoning' => 'Used two FAQ results' + } + rewritten_model_output = { + 'response_parts' => [ + { 'text' => 'Short first answer.', 'citation_indexes' => [2] }, + { 'text' => 'Short second answer.', 'citation_indexes' => [1] } + ], + 'reasoning' => 'Shortened both parts' + } + runner_context = { + session_id: "#{account.id}_#{conversation.display_id}", + state: { Captain::Assistant::CITATION_SOURCES_STATE_KEY => { 1 => 101, 2 => 202 } }, + conversation_history: [{ role: :assistant, content: original_model_output }] + } + runner_messages = [{ role: :assistant, content: original_model_output }] + + { + original_run_result: Agents::RunResult.new(output: original_model_output, messages: runner_messages, context: runner_context), + rewrite_run_result: Agents::RunResult.new(output: rewritten_model_output, messages: [], context: {}), + citation_urls: { + 1 => 'https://help.example.com/first', + 2 => 'https://help.example.com/second' + }, + runner_context: runner_context + } + end + let(:mock_result) { channel_limit_rewrite[:original_run_result] } + + before do + allow(assistant).to receive(:config).and_return('feature_citation' => true) + allow(assistant).to receive(:customer_visible_citation_urls).and_return(channel_limit_rewrite[:citation_urls]) + allow(Captain::MessageLengthLimit).to receive(:for).with(conversation).and_return(140) + allow(mock_runner).to receive(:run).and_return(mock_result, channel_limit_rewrite[:rewrite_run_result]) + end + + it 'reserves space for rendered citations and restores their original indexes' do + original_response_parts = Captain::Assistant::ResponseParts.from_response(mock_result.output) + original_customer_message = original_response_parts.customer_message_content(citation_urls: channel_limit_rewrite[:citation_urls]) + citation_markup_length = original_customer_message.length - original_response_parts.plain_text.length + response_text_limit = 140 - citation_markup_length + + expect(original_response_parts.plain_text.length).to be < 140 + expect(original_customer_message.length).to be > 140 + + result = service.generate_response(message_history: message_history) + + expect(result['response_parts']).to eq( + [ + { 'text' => 'Short first answer.', 'citation_indexes' => [2] }, + { 'text' => 'Short second answer.', 'citation_indexes' => [1] } + ] + ) + expect(channel_limit_rewrite[:runner_context][:conversation_history].last[:content]['response_parts']).to eq(result['response_parts']) + expect( + Captain::Assistant::ResponseParts.from_response(result).customer_message_content( + citation_urls: channel_limit_rewrite[:citation_urls] + ).length + ).to be <= 140 + expect(mock_runner).to have_received(:run).with( + include( + "must be at most #{response_text_limit} characters", + 'Keep the same number and order of response parts', + '"citation_indexes":[2]', + '"citation_indexes":[1]' + ), + context: hash_including(state: channel_limit_rewrite[:runner_context][:state]), + max_turns: 1 + ) + expect(mock_runner).not_to have_received(:run).with(include('help.example.com'), any_args) + end + + it 'rejects a rewrite that changes the response part citation order' do + channel_limit_rewrite[:rewrite_run_result].output['response_parts'].reverse! + allow(ChatwootExceptionTracker).to receive(:new).and_return( + instance_double(ChatwootExceptionTracker, capture_exception: true) + ) + + result = service.generate_response(message_history: message_history) + + expect(result['response']).to eq('conversation_handoff') + expect(result['reasoning']).to eq('Error occurred: Captain response rewrite changed the response part citation order') + end + end + context 'when an error occurs' do let(:error) { StandardError.new('Test error') } @@ -258,6 +426,7 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq({ 'response' => 'conversation_handoff', + 'response_parts' => [{ 'text' => 'conversation_handoff', 'citation_indexes' => [] }], 'reasoning' => 'Error occurred: Test error', 'error' => true, 'error_reason' => 'standard_error', @@ -288,6 +457,7 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq({ 'response' => 'conversation_handoff', + 'response_parts' => [{ 'text' => 'conversation_handoff', 'citation_indexes' => [] }], 'reasoning' => 'Error occurred: Test error', 'error' => true, 'error_reason' => 'standard_error', @@ -314,6 +484,7 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq({ 'response' => 'conversation_handoff', + 'response_parts' => [{ 'text' => 'conversation_handoff', 'citation_indexes' => [] }], 'reasoning' => 'Error occurred: Test error', 'error' => true, 'error_reason' => 'standard_error', @@ -406,6 +577,18 @@ RSpec.describe Captain::Assistant::AgentRunnerService do expect(result).to eq('Hash response') end + it 'joins structured response parts from hash content' do + content = { + 'response_parts' => [ + { 'text' => 'First response part', 'citation_indexes' => [1] }, + { 'text' => 'Second response part', 'citation_indexes' => [] } + ] + } + result = service.send(:extract_text_from_content, content) + + expect(result).to eq("First response part\n\nSecond response part") + end + it 'extracts text from multimodal array content' do content = [ { type: 'text', text: 'First part' }, diff --git a/spec/enterprise/services/captain/assistant/session_capture_service_spec.rb b/spec/enterprise/services/captain/assistant/session_capture_service_spec.rb index daf38c8d6..239053d6f 100644 --- a/spec/enterprise/services/captain/assistant/session_capture_service_spec.rb +++ b/spec/enterprise/services/captain/assistant/session_capture_service_spec.rb @@ -49,6 +49,7 @@ RSpec.describe Captain::Assistant::SessionCaptureService do before do allow(assistant).to receive(:agent_model).and_return('gpt-5.2') + allow(Resolv).to receive(:getaddresses).and_return(['93.184.216.34']) end describe '#capture' do @@ -98,6 +99,8 @@ RSpec.describe Captain::Assistant::SessionCaptureService do llm_model: 'openai-gpt-5.2', credits_consumed: 1.0, faq_ids: [11, 12], + used_faq_ids: [], + cited_document_ids: [], document_ids: [5], scenario_ids: [], user_id: nil @@ -149,10 +152,56 @@ RSpec.describe Captain::Assistant::SessionCaptureService do expect(session.result).to eq(result_message) expect(session.faq_ids).to eq([]) + expect(session.used_faq_ids).to eq([]) + expect(session.cited_document_ids).to eq([]) expect(session.document_ids).to eq([]) expect(session.run_context).to eq([]) end + it 'stores only selected customer-visible document citations' do + cited_document = create(:captain_document, assistant: assistant, external_link: 'https://help.example.com/reset-password') + cited_faq = create(:captain_assistant_response, assistant: assistant, documentable: cited_document) + retrieved_document = create(:captain_document, assistant: assistant, external_link: 'https://help.example.com/change-email') + retrieved_faq = create(:captain_assistant_response, assistant: assistant, documentable: retrieved_document) + user = create(:user, account: account) + used_faq = create(:captain_assistant_response, assistant: assistant, documentable: user, status: :approved) + assistant.update!(config: assistant.config.merge('feature_citation' => true)) + run_context[:state] = { + :cw_metadata => { + faq_ids: [cited_faq.id, retrieved_faq.id, used_faq.id], + used_faq_ids: [used_faq.id], + document_ids: [cited_document.id, retrieved_document.id] + }, + Captain::Assistant::CITATION_SOURCES_STATE_KEY => { 1 => cited_document.id, 2 => retrieved_document.id } + } + run_result.output = { + 'response_parts' => [{ 'text' => 'Reset your password from settings.', 'citation_indexes' => [1] }], + 'reasoning' => 'Used the password FAQ' + } + result_message.update!( + additional_attributes: { + Captain::Assistant::ResponseParts::MESSAGE_ATTRIBUTE_KEY => run_result.output['response_parts'] + } + ) + + session = service.capture! + + expect(session.faq_ids).to contain_exactly(cited_faq.id, retrieved_faq.id, used_faq.id) + expect(session.used_faq_ids).to eq([used_faq.id]) + expect(session.cited_document_ids).to eq([cited_document.id]) + end + + it 'does not store citations when citations are disabled' do + document = create(:captain_document, assistant: assistant, external_link: 'https://help.example.com/reset-password') + run_context[:state][Captain::Assistant::CITATION_SOURCES_STATE_KEY] = { 1 => document.id } + run_result.output = { + 'response_parts' => [{ 'text' => 'Reset your password from settings.', 'citation_indexes' => [1] }], + 'reasoning' => 'Used the password FAQ' + } + + expect(service.capture!.cited_document_ids).to eq([]) + end + it 'extracts every scenario that authored a message in the current turn' do first_scenario = create(:captain_scenario, assistant: assistant, account: account) second_scenario = create(:captain_scenario, assistant: assistant, account: account)