kunthawat/moreminimore-chat
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

feat: add Captain outcome reporting builders (#15425)

Browse Source 
This PR adds outcome-based reporting builders for the redesigned Captain
overview, resolution flow, and resolution trend. These builders are not
wired to controllers or the frontend yet, so the existing Captain
metrics remain unchanged.

## AssistantOverviewStatsBuilder

Builds current and previous reporting-window metrics, including the
comparison trend, for the overview and CSAT cards.

| Statistic | Description |
| --- | --- |
| Conversations handled | Counts outcome episodes in which Captain
replied or performed a non-usage-limit handoff. |
| Auto-resolution rate | Shows autonomous resolutions as a percentage of
conversations handled. |
| Autonomous resolutions | Counts episodes resolved by Captain without a
handoff or an earlier human reply. |
| Handoff rate | Shows involved handoffs as a percentage of
conversations handled. |
| Handoff count | Counts involved handoffs while excluding demand
blocked by usage limits. |
| Hours saved | Estimates displaced agent effort from Captain's public
replies at two minutes per reply. |
| Reopen rate | Shows the share of autonomous resolutions followed by
another episode. |
| Conversation depth | Shows the average number of public Captain
replies per replied-to conversation. |
| Durable resolution rate | Shows autonomous resolutions that remained
closed for at least seven days among resolutions old enough to assess. |
| Autonomous CSAT score | Averages CSAT ratings from conversations
resolved autonomously by Captain. |
| Assisted CSAT score | Averages CSAT ratings from resolved
conversations where Captain participated alongside a human. |
| Human-only CSAT score | Averages account CSAT from conversations where
Captain never participated. |
| Median resolution time | Reports the median elapsed seconds from
demand start to resolution for handled episodes. |

## AssistantResolutionFlowBuilder

Builds the current-window Sankey data and a matching handoff-reason
distribution from the same outcome cohort.

| Statistic | Description |
| --- | --- |
| Conversations handled | Provides the Sankey entry count for episodes
where Captain participated. |
| Resolved by Captain | Counts handled episodes resolved autonomously by
Captain. |
| Handed off | Counts handled episodes transferred to a human for a
non-usage-limit reason. |
| Closed with team | Counts handled episodes outside the
autonomous-resolution and handoff branches. |
| Reopened within seven days | Counts Captain resolutions followed by a
new episode before the seven-day durability boundary. |
| Stayed closed | Counts Captain resolutions with no reopen inside seven
days. |
| Handoff reason nodes | Shows the two largest handoff categories and
combines the remainder as other reasons. |
| Handoff distribution | Returns every involved handoff category with
its count and percentage, including unclassified handoffs. |

## AssistantResolutionTrendStatsBuilder

Builds a zero-filled, timezone-aware resolution series in one outcome
query, using daily buckets for windows of 15 days or less and weekly
buckets for longer windows.

| Statistic | Description |
| --- | --- |
| Granularity | Identifies whether the response contains daily or weekly
buckets. |
| Bucket range | Returns the start and end date represented by each
bucket. |
| Conversations handled | Counts Captain-involved outcome episodes whose
demand started in each bucket. |
| Resolved by Captain | Counts autonomously resolved outcome episodes
whose demand started in each bucket. |
This commit is contained in:
Shivam Mishra
2026-08-13 18:29:22 +05:30
committed by GitHub
parent a929955cc3
commit 40f5381da6
7 changed files with 1152 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

252
spec/enterprise/builders/captain/assistant_overview_stats_builder_spec.rb Normal file
View File

@@ -0,0 +1,252 @@
require 'rails_helper'
RSpec.describe Captain::AssistantOverviewStatsBuilder do
subject(:metrics) { described_class.new(assistant, '30').metrics }
let(:account) { create(:account) }
let(:assistant) { create(:captain_assistant, account: account) }
let(:inbox) { create(:inbox, account: account) }
context 'with no outcomes' do
it 'returns only the overview metrics' do
expect(metrics.keys).to contain_exactly(
:conversations_handled,
:auto_resolution_rate,
:autonomous_resolutions,
:handoff_rate,
:handoff_count,
:hours_saved,
:reopen_rate,
:conversation_depth,
:durable_resolution_rate,
:autonomous_csat_score,
:assisted_csat_score,
:human_only_csat_score,
:median_resolution_seconds
)
end
it 'returns zeroed metrics' do
expect(metrics[:conversations_handled]).to eq(current: 0, previous: 0, trend: 0)
expect(metrics[:handoff_count][:current]).to eq(0)
expect(metrics[:durable_resolution_rate][:current]).to eq(0)
expect(metrics[:autonomous_csat_score][:current]).to eq(0)
expect(metrics[:assisted_csat_score][:current]).to eq(0)
end
end
context 'with demand in both windows' do
before do
demand_start = 20.days.ago.change(usec: 0)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: demand_start,
first_captain_reply_at: demand_start + 30.seconds,
captain_reply_count: 1,
resolved_at: demand_start + 100.seconds,
csat_rating: 5
)
demand_start = 20.days.ago.change(usec: 0)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: demand_start,
first_captain_reply_at: demand_start + 60.seconds,
captain_reply_count: 1,
resolved_at: demand_start + 200.seconds,
ended_at: demand_start + 1.day
)
demand_start = 6.days.ago.change(usec: 0)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: demand_start,
first_captain_reply_at: demand_start + 90.seconds,
captain_reply_count: 1,
resolved_at: demand_start + 300.seconds
)
demand_start = 10.days.ago.change(usec: 0)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: demand_start,
first_captain_reply_at: demand_start + 120.seconds,
captain_reply_count: 2,
handoff_at: demand_start + 150.seconds,
handoff_reason_category: 'missing_knowledge',
first_human_reply_at: demand_start + 250.seconds,
resolved_at: demand_start + 400.seconds,
csat_rating: 4
)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: 5.days.ago,
handoff_at: 5.days.ago,
handoff_reason_category: 'usage_limit'
)
create(:conversation_outcome, account: account, assistant: assistant, inbox: inbox, started_at: 4.days.ago)
demand_start = 45.days.ago.change(usec: 0)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: demand_start,
first_captain_reply_at: demand_start + 45.seconds,
captain_reply_count: 1,
resolved_at: demand_start + 500.seconds
)
end
it 'computes resolution counts and durability' do
expect(metrics[:autonomous_resolutions][:current]).to eq(3)
expect(metrics[:durable_resolution_rate][:current]).to eq(50.0)
end
it 'derives the legacy funnel metrics from outcome episodes' do
expect(metrics[:conversations_handled]).to eq(current: 4, previous: 1, trend: 300.0)
expect(metrics[:auto_resolution_rate][:current]).to eq(75.0)
expect(metrics[:handoff_rate][:current]).to eq(25.0)
expect(metrics[:handoff_count][:current]).to eq(1)
expect(metrics[:reopen_rate][:current]).to eq(33.3)
end
it 'splits autonomous and assisted CSAT and computes the median resolution time' do
expect(metrics[:autonomous_csat_score][:current]).to eq(5.0)
expect(metrics[:assisted_csat_score][:current]).to eq(4.0)
expect(metrics[:median_resolution_seconds][:current]).to eq(250)
end
end
it 'counts an unclassified handoff as Captain involvement' do
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: 1.day.ago,
handoff_at: 1.day.ago
)
expect(metrics[:conversations_handled][:current]).to eq(1)
expect(metrics[:handoff_rate][:current]).to eq(100.0)
expect(metrics[:handoff_count][:current]).to eq(1)
end
it 'derives reply activity from messages while an outcome episode is still active' do
stub_const("#{described_class}::SECONDS_SAVED_PER_REPLY", 20.minutes.to_i)
first_outcome = create(:conversation_outcome, account: account, assistant: assistant, inbox: inbox, started_at: 2.days.ago)
second_outcome = create(:conversation_outcome, account: account, assistant: assistant, inbox: inbox, started_at: 1.day.ago)
create_list(
:message,
2,
account: account,
inbox: inbox,
conversation: first_outcome.conversation,
sender: assistant,
message_type: :outgoing,
private: false,
created_at: 1.day.ago
)
create(
:message,
account: account,
inbox: inbox,
conversation: second_outcome.conversation,
sender: assistant,
message_type: :outgoing,
private: false,
created_at: 1.day.ago
)
create(
:message,
account: account,
inbox: inbox,
conversation: first_outcome.conversation,
sender: assistant,
message_type: :outgoing,
private: true,
created_at: 1.day.ago
)
expect(metrics[:hours_saved]).to eq(current: 1, previous: 0, trend: 1)
expect(metrics[:conversation_depth][:current]).to eq(1.5)
end
it 'does not count an episode in both adjacent day windows' do
travel_to Time.zone.parse('2026-08-12 12:00:00') do
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: 7.days.ago,
first_captain_reply_at: 7.days.ago
)
boundary_metrics = described_class.new(assistant, '7').metrics
expect(boundary_metrics[:conversations_handled]).to include(current: 1, previous: 0)
end
end
it 'compares Captain CSAT with conversations where Captain never participated' do
human_conversation = create(:conversation, account: account, inbox: inbox)
human_survey = create(:message, account: account, inbox: inbox, conversation: human_conversation, content_type: :input_csat)
create(
:csat_survey_response,
account: account,
conversation: human_conversation,
contact: human_conversation.contact,
message: human_survey,
rating: 3,
created_at: 5.days.ago
)
involved_outcome = create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: 5.days.ago,
first_captain_reply_at: 5.days.ago,
csat_rating: 5
)
involved_survey = create(
:message,
account: account,
inbox: inbox,
conversation: involved_outcome.conversation,
content_type: :input_csat
)
create(
:csat_survey_response,
account: account,
conversation: involved_outcome.conversation,
contact: involved_outcome.conversation.contact,
message: involved_survey,
rating: 5,
created_at: 5.days.ago
)
expect(metrics[:human_only_csat_score][:current]).to eq(3.0)
end
end

187
spec/enterprise/builders/captain/assistant_resolution_flow_builder_spec.rb Normal file
View File

@@ -0,0 +1,187 @@
require 'rails_helper'
RSpec.describe Captain::AssistantResolutionFlowBuilder do
subject(:resolution_flow) { described_class.new(assistant, '30').build }
let(:account) { create(:account) }
let(:assistant) { create(:captain_assistant, account: account) }
let(:inbox) { create(:inbox, account: account) }
context 'with no outcomes' do
it 'returns an empty flow with stable nodes and links' do
expect(resolution_flow).to eq(
sankey: {
nodes: [
{ id: :conversations_handled, count: 0 },
{ id: :resolved_by_captain, count: 0 },
{ id: :handed_off, count: 0 },
{ id: :closed_with_team, count: 0 },
{ id: :reopened_within_7_days, count: 0 },
{ id: :stayed_closed, count: 0 }
],
links: [
{ source: :conversations_handled, target: :resolved_by_captain, value: 0 },
{ source: :conversations_handled, target: :handed_off, value: 0 },
{ source: :conversations_handled, target: :closed_with_team, value: 0 },
{ source: :resolved_by_captain, target: :reopened_within_7_days, value: 0 },
{ source: :resolved_by_captain, target: :stayed_closed, value: 0 }
]
},
handoff_distribution: []
)
end
end
context 'with handled outcomes in the current window' do
before do
now = Time.current.change(usec: 0)
reopened_start = now - 20.days
reopened_resolution = reopened_start + 1.hour
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: reopened_start,
first_captain_reply_at: reopened_start + 1.minute,
resolved_at: reopened_resolution,
ended_at: reopened_resolution + 2.days
)
boundary_start = now - 15.days
boundary_resolution = boundary_start + 1.hour
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: boundary_start,
first_captain_reply_at: boundary_start + 1.minute,
resolved_at: boundary_resolution,
ended_at: boundary_resolution + 7.days
)
recent_start = now - 3.days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: recent_start,
first_captain_reply_at: recent_start + 1.minute,
resolved_at: recent_start + 1.hour
)
assisted_start = now - 2.days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: assisted_start,
first_captain_reply_at: assisted_start + 1.minute,
first_human_reply_at: assisted_start + 2.minutes,
resolved_at: assisted_start + 1.hour
)
2.times do |index|
started_at = now - (index + 4).days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: started_at,
first_captain_reply_at: started_at + 1.minute,
handoff_at: started_at + 2.minutes,
handoff_reason_category: 'customer_request'
)
end
%w[missing_knowledge policy_restriction].each_with_index do |category, index|
started_at = now - (index + 6).days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: started_at,
first_captain_reply_at: started_at + 1.minute,
handoff_at: started_at + 2.minutes,
handoff_reason_category: category
)
end
unclassified_start = now - 8.days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: unclassified_start,
handoff_at: unclassified_start + 2.minutes
)
usage_limit_start = now - 9.days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: usage_limit_start,
handoff_at: usage_limit_start + 2.minutes,
handoff_reason_category: 'usage_limit'
)
untouched_start = now - 10.days
create(:conversation_outcome, account: account, assistant: assistant, inbox: inbox, started_at: untouched_start)
outside_window_start = now - 45.days
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: outside_window_start,
first_captain_reply_at: outside_window_start + 1.minute,
resolved_at: outside_window_start + 1.hour
)
end
it 'builds a balanced resolution flow' do
expect(resolution_flow[:sankey]).to eq(
nodes: [
{ id: :conversations_handled, count: 9 },
{ id: :resolved_by_captain, count: 3 },
{ id: :handed_off, count: 5 },
{ id: :closed_with_team, count: 1 },
{ id: :reopened_within_7_days, count: 1 },
{ id: :stayed_closed, count: 2 },
{ id: :handoff_reason_customer_request, count: 2 },
{ id: :handoff_reason_missing_knowledge, count: 1 },
{ id: :other_reasons, count: 2 }
],
links: [
{ source: :conversations_handled, target: :resolved_by_captain, value: 3 },
{ source: :conversations_handled, target: :handed_off, value: 5 },
{ source: :conversations_handled, target: :closed_with_team, value: 1 },
{ source: :resolved_by_captain, target: :reopened_within_7_days, value: 1 },
{ source: :resolved_by_captain, target: :stayed_closed, value: 2 },
{ source: :handed_off, target: :handoff_reason_customer_request, value: 2 },
{ source: :handed_off, target: :handoff_reason_missing_knowledge, value: 1 },
{ source: :handed_off, target: :other_reasons, value: 2 }
]
)
end
it 'returns every handoff reason with its share of involved handoffs' do
expect(resolution_flow[:handoff_distribution]).to eq([
{ category: 'customer_request', count: 2, percentage: 40.0 },
{ category: 'missing_knowledge', count: 1, percentage: 20.0 },
{ category: 'policy_restriction', count: 1, percentage: 20.0 },
{ category: 'unclassified', count: 1, percentage: 20.0 }
])
end
end
end

189
spec/enterprise/builders/captain/assistant_resolution_trend_stats_builder_spec.rb Normal file
View File

@@ -0,0 +1,189 @@
require 'rails_helper'
RSpec.describe Captain::AssistantResolutionTrendStatsBuilder do
subject(:metrics) { described_class.new(assistant, range, timezone_offset).metrics }
let(:account) { create(:account) }
let(:assistant) { create(:captain_assistant, account: account) }
let(:inbox) { create(:inbox, account: account) }
let(:range) { 'this_month' }
let(:timezone_offset) { 0 }
let(:now) { Time.zone.parse('2025-06-30 12:00:00') }
around do |example|
travel_to(now) { example.run }
end
context 'with no outcomes' do
it 'returns zeroed weekly buckets for the complete reporting window' do
expect(metrics).to eq(
granularity: :week,
buckets: [
{ starts_on: Date.new(2025, 6, 1), ends_on: Date.new(2025, 6, 7), conversations_handled: 0, resolved_by_captain: 0 },
{ starts_on: Date.new(2025, 6, 8), ends_on: Date.new(2025, 6, 14), conversations_handled: 0, resolved_by_captain: 0 },
{ starts_on: Date.new(2025, 6, 15), ends_on: Date.new(2025, 6, 21), conversations_handled: 0, resolved_by_captain: 0 },
{ starts_on: Date.new(2025, 6, 22), ends_on: Date.new(2025, 6, 28), conversations_handled: 0, resolved_by_captain: 0 },
{ starts_on: Date.new(2025, 6, 29), ends_on: Date.new(2025, 6, 30), conversations_handled: 0, resolved_by_captain: 0 }
]
)
end
end
context 'when the reporting window spans 15 days or less' do
let(:now) { Time.zone.parse('2025-06-15 12:00:00') }
it 'returns zeroed daily buckets' do
expect(metrics[:granularity]).to eq(:day)
expect(metrics[:buckets].map { |bucket| bucket.values_at(:starts_on, :ends_on) }).to eq(
(Date.new(2025, 6, 1)..Date.new(2025, 6, 15)).map { |date| [date, date] }
)
expect(metrics[:buckets]).to all(include(conversations_handled: 0, resolved_by_captain: 0))
end
it 'computes every bucket in one outcome query' do
queries = []
subscriber = lambda do |_name, _started, _finished, _unique_id, payload|
queries << payload[:sql] if payload[:sql].include?('FROM "conversation_outcomes"') && !payload[:cached]
end
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') { metrics }
expect(queries.size).to eq(1)
end
end
context 'with outcomes across the reporting window' do
before do
first_week_start = Time.zone.parse('2025-06-01 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: first_week_start,
first_captain_reply_at: first_week_start + 1.minute,
resolved_at: first_week_start + 1.hour
)
second_week_start = Time.zone.parse('2025-06-08 00:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: second_week_start,
handoff_at: second_week_start + 1.minute,
handoff_reason_category: 'customer_request'
)
usage_limit_start = Time.zone.parse('2025-06-09 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: usage_limit_start,
handoff_at: usage_limit_start + 1.minute,
handoff_reason_category: 'usage_limit'
)
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: Time.zone.parse('2025-06-15 12:00:00')
)
fourth_week_start = Time.zone.parse('2025-06-22 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: fourth_week_start,
first_captain_reply_at: fourth_week_start + 1.minute,
first_human_reply_at: fourth_week_start + 2.minutes,
resolved_at: fourth_week_start + 1.hour
)
final_week_start = Time.zone.parse('2025-06-29 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: final_week_start,
first_captain_reply_at: final_week_start + 1.minute,
resolved_at: final_week_start + 1.hour
)
outside_window_start = Time.zone.parse('2025-05-31 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: outside_window_start,
first_captain_reply_at: outside_window_start + 1.minute,
resolved_at: outside_window_start + 1.hour
)
other_assistant = create(:captain_assistant, account: account)
other_assistant_start = Time.zone.parse('2025-06-02 12:00:00')
create(
:conversation_outcome,
account: account,
assistant: other_assistant,
inbox: inbox,
started_at: other_assistant_start,
first_captain_reply_at: other_assistant_start + 1.minute,
resolved_at: other_assistant_start + 1.hour
)
end
it 'counts handled and Captain-resolved outcomes in their demand week' do
bucket_counts = metrics[:buckets].map { |bucket| bucket.values_at(:conversations_handled, :resolved_by_captain) }
expect(bucket_counts).to eq([
[1, 1],
[1, 0],
[0, 0],
[1, 0],
[1, 1]
])
end
it 'computes every bucket in one outcome query' do
queries = []
subscriber = lambda do |_name, _started, _finished, _unique_id, payload|
queries << payload[:sql] if payload[:sql].include?('FROM "conversation_outcomes"') && !payload[:cached]
end
ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') { metrics }
expect(queries.size).to eq(1)
end
end
it 'anchors calendar buckets to the viewer timezone' do
started_at = Time.zone.parse('2025-05-31 19:00:00')
create(
:conversation_outcome,
account: account,
assistant: assistant,
inbox: inbox,
started_at: started_at,
first_captain_reply_at: started_at + 1.minute,
resolved_at: started_at + 1.hour
)
timezone_metrics = described_class.new(assistant, range, 5.5).metrics
expect(timezone_metrics[:buckets].first).to include(
starts_on: Date.new(2025, 6, 1),
conversations_handled: 1,
resolved_by_captain: 1
)
end
end