kunthawat/moreminimore-chat
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

feat: record Captain conversation outcome episodes from lifecycle events [CW-7792] (#15316)

Browse Source 
Records Captain conversation outcomes at episode grain so reporting can
distinguish initial demand from reopened conversations and measure
replies, handoffs, resolutions, human follow-up, and CSAT.

Eligibility creates the episode at demand time. Message-derived fields
are snapshotted from persisted messages at handoff or resolution,
keeping terminal analytics accurate without writing outcomes for every
message. Outcome tracking remains reporting-only and fail-open.

Builds on the episode-grain schema from #15315.

## Closes

- https://linear.app/chatwoot/issue/CW-7792

## How to test

1. Enable `captain_integration_v2` and connect a Captain assistant to an
inbox.
2. Send an inbound customer message and confirm an initial outcome
episode is created at the message timestamp.
3. Let Captain reply and then resolve or hand off the conversation.
Confirm the episode records Captain reply counts and timestamps, the
outcome timestamp, and the handoff category where applicable.
4. Reply after resolution and confirm a `reopen` episode is created
while preserving the previous episode.
5. Resolve the reopened conversation and submit CSAT. Confirm the
response is attributed to the episode that issued the survey.

## What changed

- Creates the initial episode from demand-level eligibility and appends
a new episode when a resolved conversation reopens.
- Snapshots Captain replies and the first qualifying human reply from
persisted messages at handoff and resolution.
- Attributes asynchronous resolution events using the episode active at
the event timestamp.
- Records later CSAT responses using the survey message timestamp.
- Keeps boundary writes transactional and fail-open without retries,
advisory locks, late-boundary repair, or handoff self-healing.
- Adds schema-constrained handoff reason categories, including lifecycle
coverage for incomplete V2 tool fallback handoffs.

Open, non-terminal episodes may retain empty or stale message-derived
fields until handoff or resolution.
This commit is contained in:
Shivam Mishra
2026-08-11 14:57:23 +05:30
committed by GitHub
parent 1693525124
commit 0a2293f921
18 changed files with 608 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
spec/enterprise/jobs/captain/conversation/response_builder_job_spec.rb
View File

@@ -817,6 +817,8 @@ RSpec.describe Captain::Conversation::ResponseBuilderJob, type: :job do
'response' => 'I tried to hand off',
'handoff_tool_called' => true
})
expect(Captain::ConversationEvents).to receive(:handed_off)
.with(conversation: conversation, assistant: assistant, source: 'tool', reason_category: :tool_failure, at: kind_of(Time))
described_class.perform_now(conversation, assistant)

63
spec/enterprise/lib/captain/tools/handoff_tool_spec.rb
View File

@@ -16,13 +16,23 @@ RSpec.describe Captain::Tools::HandoffTool, type: :model do
end
end
describe '#parameters' do
it 'returns the correct parameters' do
expect(tool.parameters).to have_key(:reason)
expect(tool.parameters[:reason].name).to eq(:reason)
expect(tool.parameters[:reason].type).to eq('string')
expect(tool.parameters[:reason].description).to eq('The reason why handoff is needed (optional)')
expect(tool.parameters[:reason].required).to be false
describe '#params_schema' do
it 'constrains the reason category to the supported values and keeps the reason optional' do
schema = tool.params_schema
expect(schema['properties']['reason']['type']).to eq('string')
expect(schema['properties']['reason_category']['enum']).to eq(described_class::REASON_CATEGORIES)
expect(schema['required']).to eq(['reason_category'])
end
it 'uses only reason categories supported by conversation outcomes' do
expect(ConversationOutcome::HANDOFF_REASON_CATEGORIES).to include(*described_class::REASON_CATEGORIES)
end
it 'survives Agents::ToolWrapper reading the class params at wrap time' do
described_class.params
expect(described_class.new(assistant).params_schema['properties'].keys).to contain_exactly('reason', 'reason_category')
end
end
@@ -96,9 +106,16 @@ RSpec.describe Captain::Tools::HandoffTool, type: :model do
it 'emits a captain handoff event with the tool source after the locked handoff completes' do
expect(Captain::ConversationEvents).to receive(:handed_off)
.with(conversation: conversation, assistant: assistant, source: 'tool', at: kind_of(Time))
.with(conversation: conversation, assistant: assistant, source: 'tool', reason_category: 'customer_request', at: kind_of(Time))
tool.perform(tool_context, reason: 'Customer needs specialized support')
tool.perform(tool_context, reason: 'Customer needs specialized support', reason_category: 'customer_request')
end
it 'emits an unclassified handoff when the model supplies an unknown reason category' do
expect(Captain::ConversationEvents).to receive(:handed_off)
.with(conversation: conversation, assistant: assistant, source: 'tool', reason_category: nil, at: kind_of(Time))
tool.perform(tool_context, reason: 'Customer needs specialized support', reason_category: 'hallucinated_category')
end
it 'does not emit a captain handoff event when the handoff is skipped as stale' do
@@ -170,11 +187,33 @@ RSpec.describe Captain::Tools::HandoffTool, type: :model do
tool.perform(tool_context, reason: 'Test reason')
end
it 'emits a captain handoff event with the tool source' do
it 'emits a captain handoff event with the tool source and reason category' do
expect(Captain::ConversationEvents).to receive(:handed_off)
.with(conversation: conversation, assistant: assistant, source: 'tool', at: kind_of(Time))
.with(conversation: conversation, assistant: assistant, source: 'tool', reason_category: 'unsupported_request', at: kind_of(Time))
tool.perform(tool_context, reason: 'Test reason')
tool.perform(tool_context, reason: 'Test reason', reason_category: 'unsupported_request')
end
it 'records the handoff on an existing V2 outcome' do
account.enable_features!('captain_integration_v2')
create(
:conversation_outcome,
account: account,
assistant: assistant,
conversation: conversation,
inbox: inbox
)
tool.perform(
tool_context,
reason: 'Customer needs specialized support',
reason_category: 'unsupported_request'
)
expect(ConversationOutcome.last).to have_attributes(
handoff_reason_category: 'unsupported_request',
handoff_at: be_present
)
end
it 'creates a conversation_bot_handoff reporting event' do

71
spec/enterprise/listeners/captain_listener_spec.rb
View File

@@ -11,7 +11,7 @@ describe CaptainListener do
let(:conversation) { create(:conversation, account: account, inbox: inbox, assignee: user) }
let(:event_name) { :conversation_resolved }
let(:event) { Events::Base.new(event_name, Time.zone.now, conversation: conversation) }
let(:event) { Events::Base.new(event_name, Time.zone.now.change(usec: 0), conversation: conversation) }
before do
create(:captain_inbox, captain_assistant: assistant, inbox: inbox)
@@ -49,5 +49,74 @@ describe CaptainListener do
listener.conversation_resolved(event)
end
end
it 'records the resolution on an existing V2 outcome' do
assistant.update!(config: {})
outcome = create(
:conversation_outcome,
account: account,
assistant: assistant,
conversation: conversation,
inbox: inbox,
started_at: 10.minutes.ago
)
captain_reply = create(
:message,
account: account,
inbox: inbox,
conversation: conversation,
sender: assistant,
message_type: :outgoing,
created_at: 5.minutes.ago.change(usec: 0)
)
listener.conversation_resolved(event)
expect(outcome.reload).to have_attributes(
captain_reply_count: 1,
first_captain_reply_at: captain_reply.created_at,
resolved_at: event.timestamp
)
end
end
describe '#message_updated' do
let(:conversation) { create(:conversation, account: account, inbox: inbox) }
let!(:outcome) do
create(
:conversation_outcome,
account: account,
assistant: assistant,
conversation: conversation,
inbox: inbox
)
end
it 'records a submitted CSAT response' do
message = create(
:message,
account: account,
inbox: inbox,
conversation: conversation,
content_type: :input_csat,
message_type: :outgoing
)
response = create(
:csat_survey_response,
account: account,
conversation: conversation,
contact: conversation.contact,
message: message,
rating: 5
)
event = Events::Base.new(:message_updated, Time.current, message: message)
listener.message_updated(event)
expect(outcome.reload).to have_attributes(
csat_rating: 5,
csat_received_at: response.created_at
)
end
end
end

16
spec/enterprise/services/captain/conversation_events_spec.rb
View File

@@ -6,18 +6,6 @@ RSpec.describe Captain::ConversationEvents do
let(:assistant) { create(:captain_assistant, account: account) }
let(:timestamp) { Time.zone.now }
describe '.engaged' do
it 'dispatches the engagement event with normalized context' do
expect(Rails.configuration.dispatcher).to receive(:dispatch).with(
Events::Types::CAPTAIN_CONVERSATION_ENGAGED,
timestamp,
{ conversation: conversation, assistant: assistant }
)
described_class.engaged(conversation: conversation, assistant: assistant, at: timestamp)
end
end
describe '.handed_off' do
it 'dispatches the handoff event with source and reason category' do
expect(Rails.configuration.dispatcher).to receive(:dispatch).with(
@@ -69,7 +57,9 @@ RSpec.describe Captain::ConversationEvents do
.and_return(instance_double(ChatwootExceptionTracker, capture_exception: true))
expect do
described_class.engaged(conversation: conversation, assistant: assistant, at: timestamp)
described_class.handed_off(
conversation: conversation, assistant: assistant, source: 'tool', reason_category: nil, at: timestamp
)
end.not_to raise_error
end
end

195
spec/enterprise/services/captain/conversation_outcome_tracker_spec.rb Normal file
View File

@@ -0,0 +1,195 @@
require 'rails_helper'
RSpec.describe Captain::ConversationOutcomeTracker do
let(:account) { create(:account) }
let(:assistant) { create(:captain_assistant, account: account) }
let(:inbox) { create(:inbox, account: account) }
let(:conversation) { create(:conversation, account: account, inbox: inbox, status: :pending) }
let(:tracker) { described_class.new(conversation: conversation, assistant: assistant) }
def episodes
ConversationOutcome.where(conversation_id: conversation.id).order(:started_at)
end
before do
account.enable_features!('captain_integration_v2')
create(:captain_inbox, captain_assistant: assistant, inbox: inbox)
end
describe '#record_eligibility' do
it 'creates the initial episode anchored to the demand time' do
eligible_at = 2.minutes.ago.change(usec: 0)
episode = tracker.record_eligibility(at: eligible_at)
expect(episode.reload).to have_attributes(
account: account,
assistant: assistant,
conversation: conversation,
inbox: inbox,
episode_trigger: 'initial',
started_at: eligible_at,
ended_at: nil
)
end
it 'is idempotent' do
tracker.record_eligibility(at: Time.current)
expect do
tracker.record_eligibility(at: Time.current)
end.not_to change(ConversationOutcome, :count)
end
end
describe '#record_reopen' do
let(:initial_at) { 30.minutes.ago.change(usec: 0) }
let!(:initial) { tracker.record_eligibility(at: initial_at) }
it 'closes the open episode and opens a reopen episode' do
boundary_at = 5.minutes.ago.change(usec: 0)
tracker.record_reopen(at: boundary_at)
expect(initial.reload.ended_at).to eq(boundary_at)
expect(episodes.last).to have_attributes(
episode_trigger: 'reopen',
assistant: assistant,
started_at: boundary_at,
ended_at: nil
)
end
it 'creates exactly one episode per boundary regardless of history' do
tracker.record_reopen(at: 20.minutes.ago)
tracker.record_reopen(at: 10.minutes.ago)
expect do
tracker.record_reopen(at: 1.minute.ago)
end.to change(ConversationOutcome, :count).by(1)
end
it 'ignores boundaries for conversations without Captain history' do
ConversationOutcome.delete_all
expect do
tracker.record_reopen(at: Time.current)
end.not_to change(ConversationOutcome, :count)
end
it 'captures boundary failures without raising' do
error = ActiveRecord::StatementInvalid.new('database unavailable')
exception_tracker = instance_double(ChatwootExceptionTracker)
allow(ConversationOutcome).to receive(:create!).and_raise(error)
expect(ChatwootExceptionTracker).to receive(:new).with(error, account: account).and_return(exception_tracker)
expect(exception_tracker).to receive(:capture_exception)
expect do
tracker.record_reopen(at: Time.current)
end.not_to raise_error
end
end
describe '#record_handoff' do
let!(:initial) { tracker.record_eligibility(at: 30.minutes.ago) }
it 'snapshots message facts and records the handoff reason' do
first_reply = create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: assistant, message_type: :outgoing, created_at: 20.minutes.ago.change(usec: 0)
)
last_reply = create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: assistant, message_type: :outgoing, created_at: 10.minutes.ago.change(usec: 0)
)
handoff_at = 5.minutes.ago.change(usec: 0)
tracker.record_handoff(at: handoff_at, reason_category: 'missing_knowledge')
expect(initial.reload).to have_attributes(
captain_reply_count: 2,
first_captain_reply_at: first_reply.created_at,
last_captain_reply_at: last_reply.created_at,
handoff_at: handoff_at,
handoff_reason_category: 'missing_knowledge'
)
end
end
describe '#record_resolution' do
let!(:initial) { tracker.record_eligibility(at: 30.minutes.ago) }
it 'snapshots the episode active at the resolution event time' do
captain_reply = create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: assistant, message_type: :outgoing, created_at: 20.minutes.ago.change(usec: 0)
)
resolved_at = 15.minutes.ago.change(usec: 0)
tracker.record_reopen(at: 10.minutes.ago.change(usec: 0))
tracker.record_resolution(at: resolved_at)
expect(initial.reload).to have_attributes(
captain_reply_count: 1,
first_captain_reply_at: captain_reply.created_at,
resolved_at: resolved_at
)
expect(episodes.last.resolved_at).to be_nil
end
it 'snapshots Captain and human replies through resolution' do
captain_reply = create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: assistant, message_type: :outgoing, created_at: 20.minutes.ago.change(usec: 0)
)
agent = create(:user, account: account)
create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: agent, message_type: :outgoing, created_at: 15.minutes.ago,
content_attributes: { automation_rule_id: 1 }
)
human_reply = create(
:message,
account: account, inbox: inbox, conversation: conversation,
sender: agent, message_type: :outgoing, created_at: 10.minutes.ago.change(usec: 0)
)
resolved_at = 5.minutes.ago.change(usec: 0)
tracker.record_resolution(at: resolved_at)
expect(initial.reload).to have_attributes(
captain_reply_count: 1,
first_captain_reply_at: captain_reply.created_at,
last_captain_reply_at: captain_reply.created_at,
first_human_reply_at: human_reply.created_at,
resolved_at: resolved_at
)
end
end
describe '#record_csat' do
let!(:initial) { tracker.record_eligibility(at: 30.minutes.ago) }
it 'attributes the response to the episode that issued its survey' do
survey_message = create(
:message,
account: account, inbox: inbox, conversation: conversation,
content_type: :input_csat, created_at: 20.minutes.ago
)
tracker.record_reopen(at: 10.minutes.ago)
response = create(
:csat_survey_response,
account: account, conversation: conversation, message: survey_message, rating: 4
)
tracker.record_csat(response: response)
expect(initial.reload).to have_attributes(csat_rating: 4, csat_received_at: response.created_at)
expect(episodes.last.csat_rating).to be_nil
end
end
end

51
spec/enterprise/services/enterprise/message_templates/hook_execution_service_spec.rb
View File

@@ -137,19 +137,23 @@ RSpec.describe MessageTemplates::HookExecutionService do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end
it 'emits the engagement event when captain V2 is enabled' do
it 'records a conversation outcome when captain V2 is enabled' do
account.enable_features!('captain_integration_v2')
expect(Captain::ConversationEvents).to receive(:engaged)
.with(conversation: conversation, assistant: assistant, at: kind_of(Time))
expect do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end.to change(ConversationOutcome, :count).by(1)
create(:message, conversation: conversation, message_type: :incoming, account: account)
expect(ConversationOutcome.last).to have_attributes(
assistant: assistant,
conversation: conversation
)
end
it 'does not emit the engagement event when captain V2 is disabled' do
expect(Captain::ConversationEvents).not_to receive(:engaged)
create(:message, conversation: conversation, message_type: :incoming, account: account)
it 'does not record a conversation outcome when captain V2 is disabled' do
expect do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end.not_to change(ConversationOutcome, :count)
end
end
@@ -173,19 +177,30 @@ RSpec.describe MessageTemplates::HookExecutionService do
expect(conversation.reload.status).to eq('open')
end
it 'emits a usage limit handoff event when captain V2 is enabled' do
account.enable_features!('captain_integration_v2')
it 'emits a usage limit handoff event' do
expect(Captain::ConversationEvents).to receive(:handed_off)
.with(conversation: conversation, assistant: assistant, source: 'usage_limit', reason_category: :usage_limit, at: kind_of(Time))
create(:message, conversation: conversation, message_type: :incoming, account: account)
end
it 'does not emit a handoff event when captain V2 is disabled' do
expect(Captain::ConversationEvents).not_to receive(:handed_off)
it 'records the handoff on the outcome when captain V2 is enabled' do
account.enable_features!('captain_integration_v2')
create(:message, conversation: conversation, message_type: :incoming, account: account)
expect do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end.to change(ConversationOutcome, :count).by(1)
expect(ConversationOutcome.last).to have_attributes(
handoff_reason_category: 'usage_limit',
handoff_at: be_present
)
end
it 'does not record an outcome when captain V2 is disabled' do
expect do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end.not_to change(ConversationOutcome, :count)
end
end
end
@@ -222,6 +237,14 @@ RSpec.describe MessageTemplates::HookExecutionService do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end
it 'still records the conversation as eligible demand when captain V2 is enabled' do
account.enable_features!('captain_integration_v2')
expect do
create(:message, conversation: conversation, message_type: :incoming, account: account)
end.to change(ConversationOutcome, :count).by(1)
end
end
context 'when the contact is inside the assistant audience' do