feat(data-imports): add Freshdesk migration (1/3) (#15261)

## Description

Adds Freshdesk as an integration import source so administrators can
validate a Freshdesk domain and API key, then import contacts, tickets,
public replies, customer replies, and private notes while tracking
progress from Data Imports.

The integration has now been validated against a live Freshdesk trial
tenant with contacts, Web Chat and phone tickets, public replies,
customer replies, a private note, pagination, requester expansion, and
attachment metadata. That validation found and fixed the current Web
Chat source mapping and prevented the ticket description from
duplicating the initial Web Chat message.

Related: #15116

## Closes

Closes
[CW-7639](https://linear.app/chatwoot/issue/CW-7639/freshdesk-freshworks-migration)

## Type of change

- [x] New feature (non-breaking change which adds functionality)

## What changed

- Added a shared source adapter, importer, job, retry, restart,
creation, and placeholder inbox contract used by Intercom and Freshdesk.
- Added Freshdesk API authentication, contact and ticket pagination,
requester expansion, conversation retrieval, normalization, channel
grouping, and error handling.
- Added current Freshdesk source identifiers through SMS, including Web
Chat source `15`, and grouped equivalent sources into placeholder
inboxes.
- Used Web Chat conversation events as the complete message history so
the generated ticket description does not duplicate the initial customer
message.
- Preserved Freshdesk ticket subjects in source metadata and added a
sanitized live-derived Web Chat fixture with structured bodies and
attachment metadata.
- Added Freshdesk selection, domain and API key validation, and
provider-neutral import status handling in the Data Imports UI.

## How to test

1. Enable the data_import feature for an account and open Settings >
Data > New import.
2. Select Freshdesk and enter a Freshdesk domain and API key.
3. Select contacts and/or conversations, validate the credentials, and
start the import.
4. Confirm progress is displayed and imported tickets appear as resolved
conversations in Freshdesk placeholder inboxes with public replies and
private notes preserved.
5. Verify Web Chat tickets appear in the Chat placeholder inbox and the
initial customer message is imported once.
6. Verify an abandoned import can be restarted and a stalled import can
be retried.

## Current scope

- **Product decision:** Attachment binaries are intentionally not
imported in the current migration scope. Attachment metadata is
preserved and messages include a skipped-attachment marker.
- Adaptive Retry-After scheduling and handling the 30,000-ticket listing
ceiling are covered by stacked follow-up PRs.

## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented on my code, particularly in hard-to-understand
areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged and published in downstream
modules
This commit is contained in:
Sony Mathew
2026-08-05 16:25:30 +05:30
committed by GitHub
parent bab2f99004
commit 56be133bb0
64 changed files with 4538 additions and 1814 deletions

View File

@@ -19,14 +19,12 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
end
def validate_source
totals = validate_intercom_source
totals = source_class.credentials_validator(source_params: permitted_params.to_h, import_types: import_types).perform
render json: { valid: true, totals: totals }
rescue DataImports::Intercom::Client::AuthenticationError
render_source_validation_error('We could not validate this Intercom access key. Check the key and its permissions.')
rescue DataImports::Intercom::Client::Error
render_source_validation_error('Intercom could not be reached. Please try again.')
rescue ArgumentError => e
render_source_validation_error(e.message)
rescue StandardError => e
render_source_client_error(e)
end
def create
@@ -36,44 +34,42 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
return
end
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id)
enqueue_import(@data_import)
render_show
rescue DataImports::Intercom::Client::AuthenticationError
render_source_validation_error('We could not validate this Intercom access key. Check the key and its permissions.')
rescue DataImports::Intercom::Client::Error
render_source_validation_error('Intercom could not be reached. Please try again.')
rescue ArgumentError => e
render_source_validation_error(e.message)
rescue StandardError => e
render_source_client_error(e)
end
def start
restart_service = DataImports::Intercom::RestartService.new(account: Current.account, data_import: @data_import)
restart_service = DataImports::RestartService.new(account: Current.account, data_import: @data_import)
restart_result = restart_service.perform
@data_import = restart_service.data_import
if restart_result == :access_token_missing
render json: { message: 'The Intercom access key for this import is unavailable.' }, status: :unprocessable_entity
render json: { message: "The #{source_name} #{credential_name} for this import is unavailable." }, status: :unprocessable_entity
return
end
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id) if restart_result == :enqueue
enqueue_import(@data_import) if restart_result == :enqueue
render_show
end
def retry_import
retry_service = DataImports::Intercom::RetryService.new(account: Current.account, data_import: @data_import)
retry_service = DataImports::RetryService.new(account: Current.account, data_import: @data_import)
retry_result = retry_service.perform
@data_import = retry_service.data_import
case retry_result
when :enqueue
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id)
enqueue_import(@data_import)
render_show
when :not_stalled
render json: { message: 'This Intercom import is no longer stalled.' }, status: :unprocessable_entity
render json: { message: "This #{source_name} import is no longer stalled." }, status: :unprocessable_entity
when :active_import_exists
render json: { message: 'Another Intercom import is already in progress.' }, status: :unprocessable_entity
render json: { message: "Another #{source_name} import is already in progress." }, status: :unprocessable_entity
when :access_token_missing
render json: { message: 'The Intercom access key for this import is unavailable.' }, status: :unprocessable_entity
render json: { message: "The #{source_name} #{credential_name} for this import is unavailable." }, status: :unprocessable_entity
end
end
@@ -113,11 +109,11 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
end
def permitted_params
params.permit(:name, :source_provider, :access_token, import_types: [])
params.permit(:name, :source_provider, :access_token, :domain, import_types: [])
end
def creation_service
DataImports::Intercom::CreationService.new(
DataImports::CreationService.new(
account: Current.account,
initiated_by: Current.user,
source_params: permitted_params.to_h
@@ -125,24 +121,50 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
end
def import_types
return DataImports::Intercom::Importer::DEFAULT_IMPORT_TYPES unless permitted_params.key?(:import_types)
return DataImports::Importer::DEFAULT_IMPORT_TYPES unless permitted_params.key?(:import_types)
Array(permitted_params[:import_types]).compact_blank
end
def validate_intercom_source
raise ArgumentError, 'Unsupported import source.' unless permitted_params[:source_provider] == 'intercom'
def source_class
provider = @data_import&.source_provider || permitted_params[:source_provider]
DataImports::Source.source_class(provider)
end
DataImports::Intercom::CredentialsValidator.new(
access_token: permitted_params[:access_token],
import_types: import_types
).perform
def source_name
return 'Integration' unless DataImports::Source.supported?(@data_import&.source_provider || permitted_params[:source_provider])
source_class::DISPLAY_NAME
end
def credential_name
return 'credential' unless DataImports::Source.supported?(@data_import&.source_provider || permitted_params[:source_provider])
source_class.credential_name
end
def enqueue_import(data_import)
DataImports::Source.source_class(data_import.source_provider).import_job_class.perform_later(
data_import,
data_import.active_import_run_id
)
end
def render_source_validation_error(message)
render json: { valid: false, message: message }, status: :unprocessable_entity
end
def render_source_client_error(error)
raise error unless source_class.client_error?(error)
message = if source_class.authentication_error?(error)
"We could not validate this #{source_name} #{credential_name}. Check the key and its permissions."
else
"#{source_name} could not be reached. Please try again."
end
render_source_validation_error(message)
end
def render_show
@import_errors_finder = DataImportErrorFinder.new(@data_import)
@skip_logs_finder = DataImportSkipLogFinder.new(@data_import, params)

View File

@@ -421,6 +421,10 @@
"DESCRIPTION": "Bring your existing contacts and past conversations into this account from another support tool. Each import runs in the background, so you can keep working while it finishes, track its progress, and review anything that was skipped along the way.",
"LOADING": "Fetching imports",
"DEFAULT_IMPORT_NAME": "Intercom import",
"DEFAULT_IMPORT_NAMES": {
"INTERCOM": "Intercom import",
"FRESHDESK": "Freshdesk import"
},
"TABS": {
"IMPORT": "Import",
"EXPORT": "Export"
@@ -434,9 +438,15 @@
"TITLE": "New import",
"SOURCE": "Source",
"NAME": "Import name",
"NAME_PLACEHOLDER": "July Intercom migration",
"NAME_PLACEHOLDER": "July support migration",
"ACCESS_KEY": "Intercom access key",
"ACCESS_KEY_PLACEHOLDER": "Paste your Intercom access key",
"INTERCOM_ACCESS_KEY": "Intercom access key",
"INTERCOM_ACCESS_KEY_PLACEHOLDER": "Paste your Intercom access key",
"FRESHDESK_API_KEY": "Freshdesk API key",
"FRESHDESK_API_KEY_PLACEHOLDER": "Paste your Freshdesk API key",
"FRESHDESK_DOMAIN": "Freshdesk domain",
"FRESHDESK_DOMAIN_PLACEHOLDER": "acme.freshdesk.com",
"DATA_TYPES": "Data to import",
"VALIDATING": "Validating access key...",
"VALID_KEY": "Access key validated.",
@@ -510,11 +520,11 @@
}
},
"ALERTS": {
"IMPORT_STARTED": "Intercom import has started.",
"IMPORT_RETRIED": "Intercom import has been queued to resume.",
"IMPORT_RETRY_FAILED": "Could not retry the Intercom import.",
"IMPORT_ABANDONED": "Intercom import has been abandoned.",
"IMPORT_FAILED": "Could not start the Intercom import."
"IMPORT_STARTED": "Import has started.",
"IMPORT_RETRIED": "Import has been queued to resume.",
"IMPORT_RETRY_FAILED": "Could not retry the import.",
"IMPORT_ABANDONED": "Import has been abandoned.",
"IMPORT_FAILED": "Could not start the import."
}
},
"CAPTAIN_SETTINGS": {

View File

@@ -24,7 +24,7 @@ import {
formatStatus,
importedCount,
isActiveImport,
isActiveIntercomImport,
isActiveIntegrationImport,
statusDotClass,
} from './importStatus';
@@ -53,8 +53,8 @@ const activeTabIndex = computed(() =>
);
const hasActiveImport = computed(() => dataImports.value.some(isActiveImport));
const hasActiveIntercomImport = computed(() =>
dataImports.value.some(isActiveIntercomImport)
const hasActiveIntegrationImport = computed(() =>
dataImports.value.some(isActiveIntegrationImport)
);
const dataImportRoute = dataImport => ({
@@ -137,7 +137,7 @@ const openImport = dataImport => {
};
const openImportDrawer = () => {
if (!hasActiveIntercomImport.value) showImportDrawer.value = true;
if (!hasActiveIntegrationImport.value) showImportDrawer.value = true;
};
const onImportCreated = dataImportId => {
@@ -227,9 +227,9 @@ onBeforeUnmount(() => {
<Button
size="sm"
:label="$t('DATA_IMPORTS.TABLE.NEW_IMPORT')"
:disabled="hasActiveIntercomImport"
:disabled="hasActiveIntegrationImport"
:title="
hasActiveIntercomImport
hasActiveIntegrationImport
? $t('DATA_IMPORTS.DRAWER.ACTIVE_IMPORT')
: undefined
"
@@ -371,7 +371,7 @@ onBeforeUnmount(() => {
<NewImportDialog
:show="showImportDrawer"
:has-active-import="hasActiveIntercomImport"
:has-active-import="hasActiveIntegrationImport"
@close="showImportDrawer = false"
@created="onImportCreated"
/>

View File

@@ -8,7 +8,7 @@ import Checkbox from 'dashboard/components-next/checkbox/Checkbox.vue';
import Input from 'dashboard/components-next/input/Input.vue';
import Select from 'dashboard/components-next/select/Select.vue';
import DataImportsAPI from 'dashboard/api/dataImports';
import { IMPORT_SOURCES } from './importSources';
import { IMPORT_SOURCES, importSourceConfigFor } from './importSources';
const props = defineProps({
show: { type: Boolean, default: false },
@@ -20,8 +20,17 @@ const emit = defineEmits(['close', 'created']);
const { t } = useI18n();
const dialogRef = ref(null);
const sourceProvider = ref('intercom');
const importName = ref(t('DATA_IMPORTS.DEFAULT_IMPORT_NAME'));
const sourceConfig = computed(
() => importSourceConfigFor(sourceProvider.value) || IMPORT_SOURCES[0]
);
const defaultImportName = computed(() =>
sourceProvider.value === 'freshdesk'
? t('DATA_IMPORTS.DEFAULT_IMPORT_NAMES.FRESHDESK')
: t('DATA_IMPORTS.DEFAULT_IMPORT_NAMES.INTERCOM')
);
const importName = ref(defaultImportName.value);
const accessToken = ref('');
const domain = ref('');
const selectedImportTypes = ref(['contacts', 'conversations']);
const validationState = ref('idle');
const validationMessage = ref('');
@@ -34,6 +43,17 @@ const sourceOptions = computed(() =>
IMPORT_SOURCES.map(({ value, label }) => ({ value, label }))
);
const credentialPlaceholder = computed(() =>
sourceProvider.value === 'freshdesk'
? t('DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY_PLACEHOLDER')
: t('DATA_IMPORTS.DRAWER.INTERCOM_ACCESS_KEY_PLACEHOLDER')
);
const credentialLabel = computed(() =>
sourceProvider.value === 'freshdesk'
? t('DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY')
: t('DATA_IMPORTS.DRAWER.INTERCOM_ACCESS_KEY')
);
const tokenMessageType = computed(() => {
if (validationState.value === 'valid') return 'success';
if (validationState.value === 'invalid') return 'error';
@@ -51,9 +71,14 @@ const canCreate = computed(
const validationPayload = () => ({
source_provider: sourceProvider.value,
access_token: accessToken.value.trim(),
...(sourceConfig.value.requiresDomain ? { domain: domain.value.trim() } : {}),
import_types: selectedImportTypes.value,
});
const hasRequiredCredentials = () =>
accessToken.value.trim() &&
(!sourceConfig.value.requiresDomain || domain.value.trim());
const invalidateValidation = () => {
validationRequestId += 1;
validationState.value = 'idle';
@@ -61,7 +86,7 @@ const invalidateValidation = () => {
};
const validateSource = async () => {
if (!accessToken.value.trim() || !selectedImportTypes.value.length) {
if (!hasRequiredCredentials() || !selectedImportTypes.value.length) {
invalidateValidation();
return;
}
@@ -98,7 +123,7 @@ const createImport = async () => {
try {
const response = await DataImportsAPI.create({
...validationPayload(),
name: importName.value.trim() || t('DATA_IMPORTS.DEFAULT_IMPORT_NAME'),
name: importName.value.trim() || defaultImportName.value,
});
useAlert(t('DATA_IMPORTS.ALERTS.IMPORT_STARTED'));
emit('created', response.data.id);
@@ -112,10 +137,18 @@ const createImport = async () => {
};
watch(accessToken, invalidateValidation);
watch(domain, invalidateValidation);
watch(sourceProvider, () => {
importName.value = defaultImportName.value;
accessToken.value = '';
domain.value = '';
invalidateValidation();
});
watch(selectedImportTypes, () => {
invalidateValidation();
if (accessToken.value.trim() && selectedImportTypes.value.length) {
if (hasRequiredCredentials() && selectedImportTypes.value.length) {
validateSource();
}
});
@@ -130,6 +163,7 @@ watch(
dialogRef.value?.close();
accessToken.value = '';
domain.value = '';
validationState.value = 'idle';
validationMessage.value = '';
}
@@ -164,12 +198,21 @@ watch(
:placeholder="$t('DATA_IMPORTS.DRAWER.NAME_PLACEHOLDER')"
/>
<Input
v-if="sourceConfig.requiresDomain"
v-model="domain"
autocomplete="off"
:label="$t('DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN')"
:placeholder="$t('DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN_PLACEHOLDER')"
@blur="validateSource"
/>
<Input
v-model="accessToken"
type="password"
autocomplete="off"
:label="$t('DATA_IMPORTS.DRAWER.ACCESS_KEY')"
:placeholder="$t('DATA_IMPORTS.DRAWER.ACCESS_KEY_PLACEHOLDER')"
:label="credentialLabel"
:placeholder="credentialPlaceholder"
:message="validationMessage"
:message-type="tokenMessageType"
@blur="validateSource"

View File

@@ -4,8 +4,17 @@ export const IMPORT_SOURCES = [
label: 'Intercom',
icon: '/dashboard/images/integrations/intercom.png',
},
{
value: 'freshdesk',
label: 'Freshdesk',
iconClass: 'i-lucide-life-buoy',
requiresDomain: true,
},
];
export const importSourceConfigFor = provider =>
IMPORT_SOURCES.find(source => source.value === provider);
const DEFAULT_IMPORT_SOURCE = {
value: 'file',
label: 'File import',
@@ -13,5 +22,4 @@ const DEFAULT_IMPORT_SOURCE = {
};
export const importSourceFor = dataImport =>
IMPORT_SOURCES.find(source => source.value === dataImport?.source_provider) ||
DEFAULT_IMPORT_SOURCE;
importSourceConfigFor(dataImport?.source_provider) || DEFAULT_IMPORT_SOURCE;

View File

@@ -9,14 +9,21 @@ export const isIntercomImport = dataImport =>
dataImport?.data_type === 'intercom' &&
dataImport?.source_provider === 'intercom';
export const isIntegrationImport = dataImport =>
['freshdesk', 'intercom'].includes(dataImport?.data_type) &&
dataImport?.data_type === dataImport?.source_provider;
export const isActiveIntercomImport = dataImport =>
isIntercomImport(dataImport) && isActiveImport(dataImport);
export const isActiveIntegrationImport = dataImport =>
isIntegrationImport(dataImport) && isActiveImport(dataImport);
export const isAbandonableImport = dataImport =>
isActiveIntercomImport(dataImport);
isActiveIntegrationImport(dataImport);
export const importedCount = dataImport => {
if (!isIntercomImport(dataImport)) {
if (!isIntegrationImport(dataImport)) {
return Number(dataImport?.processed_records || 0);
}

View File

@@ -0,0 +1,104 @@
import { flushPromises, mount } from '@vue/test-utils';
import { nextTick } from 'vue';
import DataImportsAPI from 'dashboard/api/dataImports';
import NewImportDialog from '../NewImportDialog.vue';
vi.mock('dashboard/api/dataImports', () => ({
default: {
validateSource: vi.fn(),
create: vi.fn(),
},
}));
vi.mock('dashboard/composables', () => ({
useAlert: vi.fn(),
}));
vi.mock('vue-i18n', () => ({
useI18n: () => ({ t: key => key }),
}));
const DialogStub = {
name: 'Dialog',
props: {
disableConfirmButton: { type: Boolean, default: false },
},
emits: ['close', 'confirm'],
methods: {
open() {},
close() {},
},
template: `
<section>
<slot />
<button
data-test="confirm"
:disabled="disableConfirmButton"
@click="$emit('confirm')"
/>
</section>
`,
};
const mountDialog = () =>
mount(NewImportDialog, {
props: { show: true },
global: {
stubs: {
Dialog: DialogStub,
Checkbox: true,
},
mocks: {
$t: key => key,
},
},
});
describe('NewImportDialog', () => {
beforeEach(() => {
DataImportsAPI.validateSource.mockResolvedValue({
data: { valid: true, totals: {} },
});
DataImportsAPI.create.mockResolvedValue({ data: { id: 42 } });
});
afterEach(() => {
vi.clearAllMocks();
});
it('validates and creates a Freshdesk import with its domain', async () => {
const wrapper = mountDialog();
await wrapper.find('select').setValue('freshdesk');
await nextTick();
const domainInput = wrapper.find(
'input[placeholder="DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN_PLACEHOLDER"]'
);
const apiKeyInput = wrapper.find(
'input[placeholder="DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY_PLACEHOLDER"]'
);
await domainInput.setValue('acme.freshdesk.com');
await apiKeyInput.setValue(' freshdesk-api-key ');
await apiKeyInput.trigger('blur');
await flushPromises();
expect(DataImportsAPI.validateSource).toHaveBeenCalledWith({
source_provider: 'freshdesk',
domain: 'acme.freshdesk.com',
access_token: 'freshdesk-api-key',
import_types: ['contacts', 'conversations'],
});
await wrapper.find('[data-test="confirm"]').trigger('click');
await flushPromises();
expect(DataImportsAPI.create).toHaveBeenCalledWith({
source_provider: 'freshdesk',
domain: 'acme.freshdesk.com',
access_token: 'freshdesk-api-key',
import_types: ['contacts', 'conversations'],
name: 'DATA_IMPORTS.DEFAULT_IMPORT_NAMES.FRESHDESK',
});
expect(wrapper.emitted('created')).toEqual([[42]]);
});
});

View File

@@ -0,0 +1,28 @@
import {
IMPORT_SOURCES,
importSourceConfigFor,
importSourceFor,
} from '../importSources';
describe('importSources', () => {
it('exposes Freshdesk as a domain-based integration source', () => {
expect(IMPORT_SOURCES.map(source => source.value)).toEqual([
'intercom',
'freshdesk',
]);
expect(importSourceConfigFor('freshdesk')).toMatchObject({
label: 'Freshdesk',
requiresDomain: true,
});
});
it('resolves persisted integrations and falls back for file imports', () => {
expect(importSourceFor({ source_provider: 'freshdesk' }).label).toBe(
'Freshdesk'
);
expect(importSourceFor({ source_provider: null })).toMatchObject({
value: 'file',
label: 'File import',
});
});
});

View File

@@ -2,6 +2,7 @@ import {
formatDate,
importedCount,
isActiveIntercomImport,
isActiveIntegrationImport,
statusDotClass,
} from '../importStatus';
@@ -32,12 +33,38 @@ describe('importStatus', () => {
});
});
describe('importedCount', () => {
it('sums Intercom imported stats', () => {
describe('isActiveIntegrationImport', () => {
it('treats matching Freshdesk and Intercom imports as active', () => {
expect(
importedCount({
isActiveIntegrationImport({
data_type: 'freshdesk',
source_provider: 'freshdesk',
status: 'pending',
})
).toBe(true);
expect(
isActiveIntegrationImport({
data_type: 'intercom',
source_provider: 'intercom',
status: 'processing',
})
).toBe(true);
expect(
isActiveIntegrationImport({
data_type: 'freshdesk',
source_provider: 'intercom',
status: 'processing',
})
).toBe(false);
});
});
describe('importedCount', () => {
it('sums integration imported stats', () => {
expect(
importedCount({
data_type: 'freshdesk',
source_provider: 'freshdesk',
processed_records: 20,
stats: {
contacts: { imported: 2 },

View File

@@ -0,0 +1,47 @@
class DataImports::BaseJob < ApplicationJob
queue_as :low
def fail_import!(error)
data_import = arguments.first
run_id = arguments.length > 1 ? arguments.last : nil
return if data_import.blank? || skip_import?(data_import, run_id)
importer_for(data_import, run_id).fail!(error)
end
private
def skip_import?(data_import, run_id = nil)
data_import.reload
data_import.abandoned? || data_import.failed? || data_import.completed? ||
data_import.completed_with_errors? || stale_import_run?(data_import, run_id)
end
def stale_import_run?(data_import, run_id)
active_run_id = data_import.active_import_run_id
active_run_id.present? && active_run_id != run_id
end
def importer_for(data_import, run_id = nil)
source_class(data_import).importer_class.new(data_import: data_import, run_id: run_id)
end
def fail_unexpected_error(importer, error)
raise error if source_class(arguments.first).client_error?(error)
importer&.fail!(error)
raise error
end
def contacts_page_job_class(data_import)
source_class(data_import).contacts_page_job_class
end
def conversations_page_job_class(data_import)
source_class(data_import).conversations_page_job_class
end
def source_class(data_import)
DataImports::Source.source_class(data_import.source_provider)
end
end

View File

@@ -0,0 +1,26 @@
module DataImports::ContactsPageJob
def perform(data_import, starting_after = nil, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return enqueue_conversations_or_finish(data_import, importer, run_id) if importer.contacts_completed?
result = importer.import_contacts_page(starting_after: starting_after)
return if skip_import?(data_import, run_id)
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
enqueue_conversations_or_finish(data_import, importer, run_id)
rescue StandardError => e
fail_unexpected_error(importer, e)
end
private
def enqueue_conversations_or_finish(data_import, importer, run_id)
if importer.import_conversations? && !importer.conversations_completed?
conversations_page_job_class(data_import).perform_later(data_import, importer.cursor_for('conversations'), run_id)
else
importer.finish!
end
end
end

View File

@@ -0,0 +1,16 @@
module DataImports::ConversationsPageJob
def perform(data_import, starting_after = nil, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return importer.finish! if importer.conversations_completed?
result = importer.import_conversations_page(starting_after: starting_after)
return if skip_import?(data_import, run_id)
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
importer.finish!
rescue StandardError => e
fail_unexpected_error(importer, e)
end
end

View File

@@ -0,0 +1,26 @@
class DataImports::Freshdesk::BaseJob < DataImports::BaseJob
DEFAULT_RATE_LIMIT_WAIT = 1.minute
retry_on DataImports::Freshdesk::Client::Error, wait: 1.minute, attempts: 3 do |job, error|
job.fail_import!(error)
end
retry_on DataImports::Freshdesk::Client::RateLimitError, wait: DEFAULT_RATE_LIMIT_WAIT, attempts: 5 do |job, error|
job.fail_import!(error)
end
discard_on CustomExceptions::DataImport::FreshdeskTicketLimitError
def retry_job(options = {})
error = options[:error]
options = options.merge(wait: rate_limit_wait(error)) if error.is_a?(DataImports::Freshdesk::Client::RateLimitError)
super(options)
end
private
def rate_limit_wait(error)
retry_after = error.retry_after.to_i
retry_after.positive? ? retry_after.seconds : DEFAULT_RATE_LIMIT_WAIT
end
end

View File

@@ -0,0 +1,3 @@
class DataImports::Freshdesk::ContactsPageJob < DataImports::Freshdesk::BaseJob
include DataImports::ContactsPageJob
end

View File

@@ -0,0 +1,3 @@
class DataImports::Freshdesk::ConversationsPageJob < DataImports::Freshdesk::BaseJob
include DataImports::ConversationsPageJob
end

View File

@@ -0,0 +1,3 @@
class DataImports::Freshdesk::ImportJob < DataImports::Freshdesk::BaseJob
include DataImports::ImportJob
end

View File

@@ -0,0 +1,24 @@
module DataImports::ImportJob
def perform(data_import, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return unless importer.start!
enqueue_next_stage(data_import, importer, run_id)
rescue StandardError => e
fail_unexpected_error(importer, e)
end
private
def enqueue_next_stage(data_import, importer, run_id)
if importer.import_contacts? && !importer.contacts_completed?
contacts_page_job_class(data_import).perform_later(data_import, importer.cursor_for('contacts'), run_id)
elsif importer.import_conversations? && !importer.conversations_completed?
conversations_page_job_class(data_import).perform_later(data_import, importer.cursor_for('conversations'), run_id)
else
importer.finish!
end
end
end

View File

@@ -1,6 +1,4 @@
class DataImports::Intercom::BaseJob < ApplicationJob
queue_as :low
class DataImports::Intercom::BaseJob < DataImports::BaseJob
retry_on DataImports::Intercom::Client::Error, wait: 1.minute, attempts: 3 do |job, error|
job.fail_import!(error)
end
@@ -8,36 +6,4 @@ class DataImports::Intercom::BaseJob < ApplicationJob
retry_on DataImports::Intercom::Client::RateLimitError, wait: 1.minute, attempts: 5 do |job, error|
job.fail_import!(error)
end
def fail_import!(error)
data_import = arguments.first
run_id = arguments.length > 1 ? arguments.last : nil
return if data_import.blank? || skip_import?(data_import, run_id)
DataImports::Intercom::Importer.new(data_import: data_import, run_id: run_id).fail!(error)
end
private
def skip_import?(data_import, run_id = nil)
data_import.reload
data_import.abandoned? || data_import.failed? || data_import.completed? ||
data_import.completed_with_errors? || stale_import_run?(data_import, run_id)
end
def stale_import_run?(data_import, run_id)
active_run_id = data_import.active_intercom_import_run_id
active_run_id.present? && active_run_id != run_id
end
def importer_for(data_import, run_id = nil)
DataImports::Intercom::Importer.new(data_import: data_import, run_id: run_id)
end
def fail_unexpected_error(importer, error)
raise error if error.is_a?(DataImports::Intercom::Client::Error)
importer&.fail!(error)
raise error
end
end

View File

@@ -1,26 +1,3 @@
class DataImports::Intercom::ContactsPageJob < DataImports::Intercom::BaseJob
def perform(data_import, starting_after = nil, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return enqueue_conversations_or_finish(data_import, importer, run_id) if importer.contacts_completed?
result = importer.import_contacts_page(starting_after: starting_after)
return if skip_import?(data_import, run_id)
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
enqueue_conversations_or_finish(data_import, importer, run_id)
rescue StandardError => e
fail_unexpected_error(importer, e)
end
private
def enqueue_conversations_or_finish(data_import, importer, run_id)
if importer.import_conversations? && !importer.conversations_completed?
DataImports::Intercom::ConversationsPageJob.perform_later(data_import, importer.cursor_for('conversations'), run_id)
else
importer.finish!
end
end
include DataImports::ContactsPageJob
end

View File

@@ -1,16 +1,3 @@
class DataImports::Intercom::ConversationsPageJob < DataImports::Intercom::BaseJob
def perform(data_import, starting_after = nil, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return importer.finish! if importer.conversations_completed?
result = importer.import_conversations_page(starting_after: starting_after)
return if skip_import?(data_import, run_id)
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
importer.finish!
rescue StandardError => e
fail_unexpected_error(importer, e)
end
include DataImports::ConversationsPageJob
end

View File

@@ -1,24 +1,3 @@
class DataImports::Intercom::ImportJob < DataImports::Intercom::BaseJob
def perform(data_import, run_id = nil)
return if skip_import?(data_import, run_id)
importer = importer_for(data_import, run_id)
return unless importer.start!
enqueue_next_stage(data_import, importer, run_id)
rescue StandardError => e
fail_unexpected_error(importer, e)
end
private
def enqueue_next_stage(data_import, importer, run_id)
if importer.import_contacts? && !importer.contacts_completed?
DataImports::Intercom::ContactsPageJob.perform_later(data_import, importer.cursor_for('contacts'), run_id)
elsif importer.import_conversations? && !importer.conversations_completed?
DataImports::Intercom::ConversationsPageJob.perform_later(data_import, importer.cursor_for('conversations'), run_id)
else
importer.finish!
end
end
include DataImports::ImportJob
end

View File

@@ -4,6 +4,15 @@ module OutOfOffisable
extend ActiveSupport::Concern
OFFISABLE_ATTRS = %w[day_of_week closed_all_day open_hour open_minutes close_hour close_minutes open_all_day].freeze
DEFAULT_WORKING_HOURS = [
{ day_of_week: 0, closed_all_day: true, open_all_day: false },
{ day_of_week: 1, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
{ day_of_week: 2, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
{ day_of_week: 3, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
{ day_of_week: 4, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
{ day_of_week: 5, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
{ day_of_week: 6, closed_all_day: true, open_all_day: false }
].map(&:freeze).freeze
included do
has_many :working_hours, dependent: :destroy_async
@@ -52,12 +61,6 @@ module OutOfOffisable
private
def create_default_working_hours
working_hours.create!(day_of_week: 0, closed_all_day: true, open_all_day: false)
working_hours.create!(day_of_week: 1, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
working_hours.create!(day_of_week: 2, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
working_hours.create!(day_of_week: 3, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
working_hours.create!(day_of_week: 4, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
working_hours.create!(day_of_week: 5, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
working_hours.create!(day_of_week: 6, closed_all_day: true, open_all_day: false)
DEFAULT_WORKING_HOURS.each { |attributes| working_hours.create!(attributes) }
end
end

View File

@@ -32,10 +32,12 @@
# index_data_imports_on_source_provider (source_provider)
#
class DataImport < ApplicationRecord
ACTIVE_IMPORT_RUN_ID_KEY = 'active_import_run_id'.freeze
ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY = 'active_intercom_import_run_id'.freeze
INTERCOM_STALLED_AFTER = 15.minutes
IMPORT_STALLED_AFTER = 15.minutes
INTERCOM_STALLED_AFTER = IMPORT_STALLED_AFTER
LEGACY_DATA_TYPES = ['contacts'].freeze
INTEGRATION_DATA_TYPES = ['intercom'].freeze
INTEGRATION_DATA_TYPES = %w[freshdesk intercom].freeze
IMPORT_TYPES = %w[contacts conversations].freeze
belongs_to :account
@@ -48,12 +50,16 @@ class DataImport < ApplicationRecord
has_many :import_errors, class_name: 'DataImportError', dependent: :destroy_async
validates :data_type, inclusion: { in: LEGACY_DATA_TYPES + INTEGRATION_DATA_TYPES, message: I18n.t('errors.data_import.data_type.invalid') }
validates :access_token, presence: true, on: :create, if: :intercom_import?
validates :access_token, presence: true, on: :create, if: :integration_import?
validate :validate_import_types
validate :validate_integration_provider
enum status: { pending: 0, processing: 1, completed: 2, failed: 3, completed_with_errors: 6, abandoned: 7 }
scope :active_intercom, -> { where(data_type: 'intercom', source_provider: 'intercom', status: [:pending, :processing]) }
scope :active_integrations, lambda {
where(data_type: INTEGRATION_DATA_TYPES, status: [:pending, :processing]).where('source_provider = data_type')
}
has_one_attached :import_file
has_one_attached :failed_records
@@ -68,38 +74,52 @@ class DataImport < ApplicationRecord
data_type == 'intercom' && source_provider == 'intercom'
end
def freshdesk_import?
data_type == 'freshdesk' && source_provider == 'freshdesk'
end
def integration_import?
INTEGRATION_DATA_TYPES.include?(data_type) && data_type == source_provider
end
def restartable?
failed? || abandoned?
end
def stalled?
intercom_import? && (pending? || processing?) && updated_at <= INTERCOM_STALLED_AFTER.ago
integration_import? && (pending? || processing?) && updated_at <= IMPORT_STALLED_AFTER.ago
end
def abandonable?
intercom_import? && (pending? || processing?)
integration_import? && (pending? || processing?)
end
def abandon!
self.class.transaction do
abandonable_import = self.class.lock.find_by(
id: id,
data_type: 'intercom',
source_provider: 'intercom',
status: [:pending, :processing]
)
active_imports = self.class.lock.where(id: id, data_type: INTEGRATION_DATA_TYPES, status: [:pending, :processing])
abandonable_import = active_imports.where('source_provider = data_type').first
abandonable_import&.update!(status: :abandoned, abandoned_at: Time.current)
end
reload
end
def active_intercom_import_run_id
source_metadata.to_h[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY]
active_import_run_id
end
def assign_active_intercom_import_run_id
self.source_metadata = source_metadata.to_h.merge(ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY => SecureRandom.uuid)
active_intercom_import_run_id
assign_active_import_run_id
end
def active_import_run_id
source_metadata.to_h[ACTIVE_IMPORT_RUN_ID_KEY] || source_metadata.to_h[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY]
end
def assign_active_import_run_id
run_id = SecureRandom.uuid
self.source_metadata = source_metadata.to_h.merge(ACTIVE_IMPORT_RUN_ID_KEY => run_id)
source_metadata[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY] = run_id if intercom_import?
run_id
end
private
@@ -119,4 +139,11 @@ class DataImport < ApplicationRecord
errors.add(:import_types, "contains unsupported values: #{invalid_types.join(', ')}")
end
def validate_integration_provider
return unless INTEGRATION_DATA_TYPES.include?(data_type)
return if source_provider == data_type
errors.add(:source_provider, 'must match the integration data type')
end
end

View File

@@ -0,0 +1,65 @@
class DataImports::CreationService
def initialize(account:, initiated_by:, source_params:)
@account = account
@initiated_by = initiated_by
@source_params = source_params.symbolize_keys
@access_token = @source_params[:access_token].to_s.strip
@provider = @source_params[:source_provider].to_s
@source_class = DataImports::Source.source_class(@provider)
end
def perform
return if active_import?
totals = validate_source
@account.with_lock do
next if active_import?
@account.data_imports.new(attributes(totals)).tap do |data_import|
data_import.assign_active_import_run_id
data_import.save!
end
end
end
private
def validate_source
@source_class.credentials_validator(source_params: @source_params, import_types: import_types).perform
end
def attributes(totals)
{
name: @source_params[:name].presence || @source_class.default_import_name,
data_type: @provider,
source_type: 'api',
source_provider: @provider,
import_types: import_types,
initiated_by: @initiated_by,
access_token: @access_token,
source_metadata: @source_class.source_metadata(@source_params),
stats: initial_stats(totals)
}
end
def import_types
return DataImports::Importer::DEFAULT_IMPORT_TYPES unless @source_params.key?(:import_types)
Array(@source_params[:import_types]).compact_blank
end
def initial_stats(totals)
{
'contacts' => { 'imported' => 0, 'skipped' => 0 },
'conversations' => { 'imported' => 0, 'skipped' => 0 },
'messages' => { 'imported' => 0, 'skipped' => 0 },
'errors' => { 'count' => 0 }
}.tap do |stats|
totals.each { |type, total| stats[type]['total'] = total unless total.nil? }
end
end
def active_import?
@account.data_imports.active_integrations.exists?
end
end

View File

@@ -0,0 +1,160 @@
require 'uri'
class DataImports::Freshdesk::Client
class Error < StandardError
attr_reader :status, :body
def initialize(message, status: nil, body: nil)
super(message)
@status = status
@body = body
end
end
class AuthenticationError < Error; end
class RateLimitError < Error
attr_reader :retry_after
def initialize(message, retry_after: nil, **)
super(message, **)
@retry_after = retry_after
end
end
Page = Struct.new(:data, :next_page, keyword_init: true)
DEFAULT_PER_PAGE = 100
MAX_TICKET_PAGES = 300
EARLIEST_TICKET_TIMESTAMP = '1970-01-01T00:00:00Z'.freeze
FRESHDESK_DOMAIN_REGEX = /\A[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.freshdesk\.com\z/i
attr_reader :domain
def self.normalize_domain(value)
candidate = value.to_s.strip.downcase
candidate = "#{candidate}.freshdesk.com" unless candidate.include?('.')
candidate = URI.parse(candidate).host if candidate.start_with?('http://', 'https://')
raise ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.' unless candidate&.match?(FRESHDESK_DOMAIN_REGEX)
candidate
rescue URI::InvalidURIError
raise ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.'
end
def initialize(domain:, api_key:)
@domain = self.class.normalize_domain(domain)
@api_key = api_key.to_s.strip
end
def list_contacts(page: 1, per_page: DEFAULT_PER_PAGE)
get_page('/contacts', query: { page: page, per_page: per_page })
end
def retrieve_contact(id)
get("/contacts/#{id}")
end
def list_tickets(page: 1, per_page: DEFAULT_PER_PAGE)
get_page(
'/tickets',
query: {
page: page,
per_page: per_page,
updated_since: EARLIEST_TICKET_TIMESTAMP
}
)
end
def retrieve_ticket(id)
get("/tickets/#{id}", query: { include: 'requester' })
end
def list_conversations(ticket_id, page: 1, per_page: DEFAULT_PER_PAGE)
get_page("/tickets/#{ticket_id}/conversations", query: { page: page, per_page: per_page })
end
private
def get_page(path, query: {})
response = request(path, query: query)
Page.new(
data: parsed_body(response),
next_page: next_page(response.headers['link'], current_page: query[:page] || query['page'])
)
end
def get(path, query: {})
parsed_body(request(path, query: query))
end
def request(path, query: {})
response =
begin
HTTParty.get(
"https://#{@domain}/api/v2#{path}",
query: query,
basic_auth: { username: @api_key, password: 'X' },
headers: { 'Accept' => 'application/json', 'Content-Type' => 'application/json' },
timeout: 30
)
rescue StandardError => e
raise Error.new(
"Freshdesk API request failed before receiving a response: #{e.message}",
body: { transport_error_class: e.class.name }
)
end
parse_response(response)
response
end
def parse_response(response)
return if response.success?
body = parsed_body(response)
message = error_message(body, response)
case response.code
when 401, 403
raise AuthenticationError.new(message, status: response.code, body: body)
when 429
raise RateLimitError.new(message, status: response.code, body: body, retry_after: response.headers['retry-after'])
else
raise Error.new(message, status: response.code, body: body)
end
end
def parsed_body(response)
response.parsed_response.presence || (response.code == 204 ? {} : [])
rescue JSON::ParserError
{}
end
def error_message(body, response)
description = body['description'] if body.is_a?(Hash)
errors = body['errors'] if body.is_a?(Hash)
first_error = errors.first if errors.is_a?(Array)
first_error_message = first_error['message'] if first_error.is_a?(Hash)
first_error_message.presence || description.presence || "Freshdesk API request failed with status #{response.code}"
end
def next_page(link_header, current_page:)
next_link = link_header.to_s.split(',').find { |link| link.include?('rel="next"') }
return if next_link.blank?
url = next_link[/<([^>]+)>/, 1]
return if url.blank?
parse_next_page(url, current_page)
rescue ArgumentError, URI::InvalidURIError
nil
end
def parse_next_page(url, current_page)
uri = URI.parse(url)
page = URI.decode_www_form(uri.query.to_s).to_h['page']
same_domain = uri.host.blank? || uri.host.casecmp?(@domain)
positive_page = page&.match?(/\A[1-9]\d*\z/)
page.to_i if same_domain && positive_page && page.to_i > current_page.to_i
end
end

View File

@@ -0,0 +1,31 @@
class DataImports::Freshdesk::CredentialsValidator
def initialize(domain:, api_key:, import_types:)
@domain = domain
@api_key = api_key.to_s.strip
@import_types = Array(import_types).compact_blank
end
def perform
validate_parameters!
client.list_contacts(per_page: 1) if @import_types.include?('contacts')
client.list_tickets(per_page: 1) if @import_types.include?('conversations')
{}
end
private
def validate_parameters!
raise ArgumentError, 'Freshdesk domain is required.' if @domain.blank?
raise ArgumentError, 'Freshdesk API key is required.' if @api_key.blank?
raise ArgumentError, 'Select at least one data type to import.' if @import_types.blank?
invalid_types = @import_types - DataImport::IMPORT_TYPES
raise ArgumentError, "Unsupported import types: #{invalid_types.join(', ')}" if invalid_types.present?
DataImports::Freshdesk::Client.normalize_domain(@domain)
end
def client
@client ||= DataImports::Freshdesk::Client.new(domain: @domain, api_key: @api_key)
end
end

View File

@@ -0,0 +1,59 @@
class DataImports::Freshdesk::Importer < DataImports::Importer
InvalidMessagePayloadError = DataImports::Importer::InvalidMessagePayloadError
ALREADY_IMPORTED_ERROR_CODE = DataImports::Freshdesk::Source::ALREADY_IMPORTED_ERROR_CODE
SKIPPED_MESSAGE_ERROR_CODE = DataImports::Freshdesk::Source::SKIPPED_MESSAGE_ERROR_CODE
TRUNCATED_PARTS_ERROR_CODE = DataImports::Freshdesk::Source::TRUNCATED_PARTS_ERROR_CODE
def import_conversations_page(starting_after: cursor_for('conversations'))
response = conversations_page(starting_after)
return PageResult.new(next_cursor: cursor_for('conversations')) unless checkpoint_ticket_page(response)
update_stat_total('conversations', response['total_count']) if response['total_count'].present?
return PageResult.new(next_cursor: cursor_for('conversations')) unless import_conversation_summaries(response)
return PageResult.new(next_cursor: nil) if import_stopped?
reconcile_dirty_stats
verify_ticket_history_complete!(response)
next_cursor = response.dig('pages', 'next', 'starting_after')
next_cursor = update_cursor('conversations', next_cursor)
PageResult.new(next_cursor: next_cursor)
end
private
def checkpoint_ticket_page(response)
current_cursor = response.dig('pages', 'current', 'starting_after')
return true if current_cursor == cursor_for('conversations')
update_cursor('conversations', current_cursor) == current_cursor
end
def conversations_page(starting_after)
@source.list_conversations(
starting_after: cursor_for('conversations').presence || starting_after,
per_page: @source.conversations_per_page
)
end
def verify_ticket_history_complete!(response)
raise CustomExceptions::DataImport::FreshdeskTicketLimitError if response.dig('pages', 'limit_reached')
end
def import_conversation_summaries(response)
summaries = Array(response['data'] || response['conversations'])
checkpoints = Array(response.dig('pages', 'checkpoints'))
summaries.each_with_index do |conversation_summary, index|
break if import_stopped?
import_conversation_from_summary(conversation_summary)
break if import_stopped?
next if index == summaries.length - 1
checkpoint = checkpoints.fetch(index)
return false unless update_cursor('conversations', checkpoint) == checkpoint
end
true
end
end

View File

@@ -0,0 +1,10 @@
class DataImports::Freshdesk::MessageBatchBuilder < DataImports::MessageBatchBuilder
def initialize(data_import:, conversation:, source_conversation:)
super(
data_import: data_import,
conversation: conversation,
source_conversation: source_conversation,
provider: 'freshdesk'
)
end
end

View File

@@ -0,0 +1,51 @@
class DataImports::Freshdesk::Metadata
def contact_custom_attributes(contact)
{
freshdesk_contact_id: contact['id'],
freshdesk_unique_external_id: contact['external_id']
}.compact
end
def contact_source(contact)
{
contact_id: contact['id'],
unique_external_id: contact['external_id'],
raw_phone: contact['phone'],
other_emails: contact['other_emails'],
custom_fields: contact['custom_fields'],
tags: contact['tags']
}.compact
end
def conversation_source(conversation)
{
ticket_id: conversation['id'],
subject: conversation['subject'],
status: conversation['status'],
priority: conversation['priority'],
requester_id: conversation['requester_id'],
responder_id: conversation['responder_id'],
group_id: conversation['group_id'],
company_id: conversation['company_id'],
tags: conversation['tags'],
custom_fields: conversation['custom_fields'],
archived: conversation['archived']
}.compact
end
def message_source(part)
{
conversation_id: part['id'],
conversation_source: part['source'],
user_id: part.dig('author', 'id'),
incoming: part['incoming'],
private: part['private'],
from_email: part['from_email'],
to_emails: part['to_emails'],
cc_emails: part['cc_emails'],
bcc_emails: part['bcc_emails'],
attachments: part['attachments'],
deleted: part['deleted']
}.compact
end
end

View File

@@ -0,0 +1,138 @@
class DataImports::Freshdesk::Normalizer
WEB_CHAT_SOURCE = 15
def contact(contact)
{
'id' => contact['id']&.to_s,
'name' => contact['name'],
'email' => contact['email'],
'phone' => contact['mobile'].presence || contact['phone'],
'external_id' => contact['unique_external_id'].presence || contact['external_id'],
'created_at' => unix_timestamp(contact['created_at']),
'updated_at' => unix_timestamp(contact['updated_at']),
'last_seen_at' => unix_timestamp(contact['updated_at']),
'other_emails' => contact['other_emails'],
'custom_fields' => contact['custom_fields'].presence || contact['custom_field'],
'tags' => contact['tags']
}.compact
end
def ticket(ticket, conversations)
{
'id' => ticket['id'].to_s,
'created_at' => unix_timestamp(ticket['created_at']),
'updated_at' => unix_timestamp(ticket['updated_at']),
'source' => ticket_source(ticket),
'contacts' => { 'contacts' => conversation_contacts(ticket, conversations) },
'conversation_parts' => conversation_parts(conversations)
}.merge(ticket_metadata(ticket)).compact
end
private
def ticket_source(ticket)
source = {
'id' => ticket['id'].to_s,
'type' => DataImports::Freshdesk::SourceBucket.source_type(ticket['source']),
'subject' => ticket['subject'],
'body' => ticket['description'].presence || ticket['description_text'],
'attachments' => ticket['attachments'],
'author' => source_author(ticket)
}.compact
source.except!('subject', 'body') if ticket['source'].to_i == WEB_CHAT_SOURCE
source
end
def source_author(ticket)
requester(ticket).merge('type' => outgoing_source?(ticket) ? 'admin' : 'contact')
end
def outgoing_source?(ticket)
[10, 14].include?(ticket['source'].to_i)
end
def requester(ticket)
requester = ticket['requester'].is_a?(Hash) ? ticket['requester'] : {}
contact(requester.merge('id' => requester['id'] || ticket['requester_id']))
end
def conversation_parts(conversations)
sorted_conversations = conversations.each_with_index.sort_by do |conversation, index|
[conversation['created_at'].to_s, index]
end.map(&:first)
{
'conversation_parts' => sorted_conversations.map { |conversation| message(conversation) },
'total_count' => conversations.size
}
end
def conversation_contacts(ticket, conversations)
contacts = [requester(ticket)]
contacts.concat(conversations.filter_map { |conversation| conversation_contact(conversation) })
contacts.uniq { |contact_payload| contact_payload['id'].presence || contact_payload['email'].to_s.downcase.presence }
end
def conversation_contact(conversation)
return unless conversation['incoming']
return if conversation['user_id'].blank? && conversation['from_email'].blank?
{
'id' => conversation['user_id']&.to_s,
'name' => conversation['from_email'],
'email' => conversation['from_email']
}.compact
end
def ticket_metadata(ticket)
{
'subject' => ticket['subject'],
'status' => ticket['status'],
'priority' => ticket['priority'],
'requester_id' => ticket['requester_id'],
'responder_id' => ticket['responder_id'],
'group_id' => ticket['group_id'],
'company_id' => ticket['company_id'],
'tags' => ticket['tags'],
'custom_fields' => ticket['custom_fields'],
'archived' => ticket['archived']
}
end
def message(conversation)
message_attributes(conversation).merge(message_metadata(conversation)).compact
end
def message_attributes(conversation)
{
'id' => conversation['id'].to_s,
'part_type' => conversation['private'] ? 'note' : 'comment',
'body' => conversation['body'].presence || conversation['body_text'],
'attachments' => conversation['attachments'],
'created_at' => unix_timestamp(conversation['created_at']),
'updated_at' => unix_timestamp(conversation['updated_at']),
'author' => message_author(conversation)
}
end
def message_metadata(conversation)
conversation.slice(
'source', 'incoming', 'private', 'from_email', 'to_emails', 'cc_emails', 'bcc_emails', 'deleted'
)
end
def message_author(conversation)
{
'id' => conversation['user_id']&.to_s,
'type' => conversation['incoming'] ? 'contact' : 'admin',
'name' => conversation['from_email'],
'email' => conversation['from_email']
}.compact
end
def unix_timestamp(value)
return if value.blank?
Time.zone.parse(value.to_s).to_i
end
end

View File

@@ -0,0 +1,5 @@
class DataImports::Freshdesk::PlaceholderInboxBuilder < DataImports::PlaceholderInboxBuilder
def initialize(account:)
super(account: account, provider: 'freshdesk', source_bucket: DataImports::Freshdesk::SourceBucket)
end
end

View File

@@ -0,0 +1,196 @@
class DataImports::Freshdesk::Source
PROVIDER = 'freshdesk'.freeze
DISPLAY_NAME = 'Freshdesk'.freeze
CONTACTS_PER_PAGE = 100
CONVERSATIONS_PER_PAGE = 100
ALREADY_IMPORTED_ERROR_CODE = 'DataImports::Freshdesk::AlreadyImported'.freeze
SKIPPED_MESSAGE_ERROR_CODE = 'DataImports::Freshdesk::SkippedMessage'.freeze
TRUNCATED_PARTS_ERROR_CODE = 'DataImports::Freshdesk::TruncatedConversations'.freeze
attr_reader :provider, :display_name, :contacts_per_page, :conversations_per_page
def self.credentials_validator(source_params:, import_types:)
DataImports::Freshdesk::CredentialsValidator.new(
domain: source_params[:domain],
api_key: source_params[:access_token],
import_types: import_types
)
end
def self.source_metadata(source_params)
{ domain: DataImports::Freshdesk::Client.normalize_domain(source_params[:domain]) }
end
def self.default_import_name
'Freshdesk import'
end
def self.credential_name
'API key'
end
def self.import_job_class
DataImports::Freshdesk::ImportJob
end
def self.importer_class
DataImports::Freshdesk::Importer
end
def self.contacts_page_job_class
DataImports::Freshdesk::ContactsPageJob
end
def self.conversations_page_job_class
DataImports::Freshdesk::ConversationsPageJob
end
def self.client_error?(error)
error.is_a?(DataImports::Freshdesk::Client::Error)
end
def self.authentication_error?(error)
error.is_a?(DataImports::Freshdesk::Client::AuthenticationError)
end
def initialize(access_token:, source_metadata: {})
@provider = PROVIDER
@display_name = DISPLAY_NAME
@contacts_per_page = CONTACTS_PER_PAGE
@conversations_per_page = CONVERSATIONS_PER_PAGE
@client = DataImports::Freshdesk::Client.new(domain: source_metadata['domain'] || source_metadata[:domain], api_key: access_token)
@normalizer = DataImports::Freshdesk::Normalizer.new
@metadata = DataImports::Freshdesk::Metadata.new
end
def list_contacts(starting_after:, per_page:)
page = @client.list_contacts(page: starting_after.presence || 1, per_page: per_page)
paginated_response(page, Array(page.data).map { |contact| @normalizer.contact(contact) })
end
def list_conversations(starting_after:, per_page:)
page = DataImports::Freshdesk::TicketPage.new(client: @client, starting_after: starting_after, per_page: per_page)
summaries = page.chunk.map do |ticket|
{
'id' => ticket['id'].to_s,
'source' => { 'type' => DataImports::Freshdesk::SourceBucket.source_type(ticket['source']) }
}
end
{
'data' => summaries,
'pages' => {
'current' => { 'starting_after' => page.current_cursor },
'next' => page.next_cursor.present? ? { 'starting_after' => page.next_cursor } : nil,
'checkpoints' => page.checkpoints,
'limit_reached' => page.limit_reached?
}
}
end
def retrieve_conversation(id)
ticket = @client.retrieve_ticket(id)
conversations = all_conversations(id)
@normalizer.ticket(ticket, conversations)
end
def retrieve_contact(id)
@normalizer.contact(@client.retrieve_contact(id))
end
def client_error?(error)
error.is_a?(DataImports::Freshdesk::Client::Error)
end
def placeholder_inbox_builder(account:)
DataImports::Freshdesk::PlaceholderInboxBuilder.new(account: account)
end
def message_batch_builder(data_import:, conversation:, source_conversation:)
DataImports::Freshdesk::MessageBatchBuilder.new(
data_import: data_import,
conversation: conversation,
source_conversation: source_conversation
)
end
def activity_part?(_part)
false
end
def source_message_importable?(source)
DataImports::Freshdesk::MessageBatchBuilder.source_message_importable?(source)
end
def activity_content(_part)
nil
end
def already_imported_error_code
ALREADY_IMPORTED_ERROR_CODE
end
def skipped_message_error_code
SKIPPED_MESSAGE_ERROR_CODE
end
def truncated_parts_error_code
TRUNCATED_PARTS_ERROR_CODE
end
def skipped_message_reason
'blank_freshdesk_conversation'
end
def contact_custom_attributes(contact_payload)
@metadata.contact_custom_attributes(contact_payload)
end
def contact_source_metadata(contact_payload)
@metadata.contact_source(contact_payload)
end
def conversation_custom_attributes(conversation)
{
freshdesk_ticket_id: conversation['id'],
freshdesk_subject: conversation['subject'],
freshdesk_status: conversation['status'],
freshdesk_priority: conversation['priority']
}.compact
end
def conversation_source_metadata(conversation)
@metadata.conversation_source(conversation)
end
def message_source_metadata(part)
@metadata.message_source(part)
end
def timestamp_for(value)
return Time.current if value.blank?
value.is_a?(Numeric) || value.to_s.match?(/\A-?\d+(?:\.\d+)?\z/) ? Time.zone.at(value.to_f) : Time.zone.parse(value.to_s)
end
private
def paginated_response(page, records)
{
'data' => records,
'pages' => { 'next' => page.next_page.present? ? { 'starting_after' => page.next_page } : nil }
}
end
def all_conversations(ticket_id)
records = []
page_number = 1
loop do
page = @client.list_conversations(ticket_id, page: page_number, per_page: CONVERSATIONS_PER_PAGE)
records.concat(Array(page.data))
break if page.next_page.blank?
page_number = page.next_page
end
records
end
end

View File

@@ -0,0 +1,61 @@
class DataImports::Freshdesk::SourceBucket
SOURCE_TYPES = {
1 => 'email',
2 => 'portal',
3 => 'phone',
4 => 'forum',
5 => 'twitter',
6 => 'facebook',
7 => 'chat',
8 => 'mobihelp',
9 => 'feedback_widget',
10 => 'outbound_email',
11 => 'ecommerce',
12 => 'bot',
13 => 'whatsapp',
14 => 'chat_internal_task',
15 => 'web_chat',
16 => 'web_form',
17 => 'instagram_message',
18 => 'instagram_comment',
19 => 'facebook_message',
20 => 'facebook_comment',
21 => 'mobile_chat_sdk',
22 => 'sms'
}.freeze
BUCKETS = {
'email' => { key: 'email', name: 'Email' },
'outbound_email' => { key: 'email', name: 'Email' },
'phone' => { key: 'phone', name: 'Phone' },
'forum' => { key: 'forum', name: 'Forum' },
'twitter' => { key: 'twitter', name: 'Twitter' },
'facebook' => { key: 'facebook', name: 'Facebook' },
'portal' => { key: 'portal', name: 'Portal' },
'feedback_widget' => { key: 'portal', name: 'Portal' },
'chat' => { key: 'chat', name: 'Chat' },
'mobihelp' => { key: 'mobile', name: 'Mobile' },
'ecommerce' => { key: 'ecommerce', name: 'Ecommerce' },
'bot' => { key: 'bot', name: 'Bot' },
'whatsapp' => { key: 'whatsapp', name: 'WhatsApp' },
'chat_internal_task' => { key: 'internal_task', name: 'Internal task' },
'web_chat' => { key: 'chat', name: 'Chat' },
'web_form' => { key: 'portal', name: 'Portal' },
'instagram_message' => { key: 'instagram', name: 'Instagram' },
'instagram_comment' => { key: 'instagram', name: 'Instagram' },
'facebook_message' => { key: 'facebook', name: 'Facebook' },
'facebook_comment' => { key: 'facebook', name: 'Facebook' },
'mobile_chat_sdk' => { key: 'mobile', name: 'Mobile' },
'sms' => { key: 'sms', name: 'SMS' }
}.freeze
DEFAULT_BUCKET = { key: 'unknown', name: 'Unknown' }.freeze
def self.source_type(value)
SOURCE_TYPES[value.to_i] || 'unknown'
end
def self.for(source_type)
BUCKETS[source_type.to_s.downcase] || DEFAULT_BUCKET
end
end

View File

@@ -0,0 +1,62 @@
class DataImports::Freshdesk::TicketPage
TICKETS_PER_CHUNK = 10
attr_reader :current_cursor
def initialize(client:, starting_after:, per_page:)
@client = client
@per_page = per_page
@cursor = normalize_cursor(starting_after)
@tickets, next_page = ticket_page(per_page)
@page_limit_reached = page_limit_reached?
next_page = nil if @page_limit_reached
@current_cursor = @cursor.merge('tickets' => @tickets, 'next_page' => next_page)
end
def chunk
@chunk ||= @tickets.slice(@cursor['offset'], TICKETS_PER_CHUNK) || []
end
def checkpoints
@checkpoints ||= chunk.each_index.map { |index| cursor_after(index + 1) }
end
def next_cursor
return checkpoints.last if chunk.present?
cursor_after(0)
end
def limit_reached?
@page_limit_reached && next_cursor.nil?
end
private
def normalize_cursor(starting_after)
return { 'page' => starting_after.presence || 1, 'offset' => 0 } unless starting_after.is_a?(Hash)
cursor = starting_after.deep_stringify_keys.slice('page', 'offset', 'tickets', 'next_page')
cursor.merge('page' => cursor['page'] || 1, 'offset' => cursor['offset'].to_i)
end
def ticket_page(per_page)
return [Array(@cursor['tickets']), @cursor['next_page']] if @cursor.key?('tickets')
page = @client.list_tickets(page: @cursor['page'], per_page: per_page)
tickets = Array(page.data).map { |ticket| ticket.slice('id', 'source') }
[tickets, page.next_page]
end
def cursor_after(processed_count)
next_offset = @cursor['offset'] + processed_count
return current_cursor.merge('offset' => next_offset) if next_offset < @tickets.size
return { 'page' => current_cursor['next_page'], 'offset' => 0 } if current_cursor['next_page'].present?
nil
end
def page_limit_reached?
@cursor['page'].to_i >= DataImports::Freshdesk::Client::MAX_TICKET_PAGES && @tickets.size >= @per_page
end
end

File diff suppressed because it is too large Load Diff

View File

@@ -1,67 +1,2 @@
class DataImports::Intercom::CreationService
def initialize(account:, initiated_by:, source_params:)
@account = account
@initiated_by = initiated_by
@source_params = source_params.symbolize_keys
@access_token = @source_params[:access_token].to_s.strip
end
def perform
return if active_import?
totals = validate_source
@account.with_lock do
next if active_import?
@account.data_imports.new(attributes(totals)).tap do |data_import|
data_import.assign_active_intercom_import_run_id
data_import.save!
end
end
end
private
def validate_source
raise ArgumentError, 'Unsupported import source.' unless @source_params[:source_provider] == 'intercom'
DataImports::Intercom::CredentialsValidator.new(
access_token: @access_token,
import_types: import_types
).perform
end
def attributes(totals)
{
name: @source_params[:name].presence || 'Intercom import',
data_type: 'intercom',
source_type: 'api',
source_provider: 'intercom',
import_types: import_types,
initiated_by: @initiated_by,
access_token: @access_token,
stats: initial_stats(totals)
}
end
def import_types
return DataImports::Intercom::Importer::DEFAULT_IMPORT_TYPES unless @source_params.key?(:import_types)
Array(@source_params[:import_types]).compact_blank
end
def initial_stats(totals)
{
'contacts' => { 'imported' => 0, 'skipped' => 0 },
'conversations' => { 'imported' => 0, 'skipped' => 0 },
'messages' => { 'imported' => 0, 'skipped' => 0 },
'errors' => { 'count' => 0 }
}.tap do |stats|
totals.each { |type, total| stats[type]['total'] = total unless total.nil? }
end
end
def active_import?
@account.data_imports.active_intercom.exists?
end
class DataImports::Intercom::CreationService < DataImports::CreationService
end

File diff suppressed because it is too large Load Diff

View File

@@ -1,144 +1,5 @@
class DataImports::Intercom::MessageBatchBuilder
PROVIDER = 'intercom'.freeze
REGULAR_PART_TYPES = %w[comment note source].freeze
Entry = Struct.new(:source_id, :part, :position, :mapping, :message, :classification, keyword_init: true) do
def source?
part['part_type'] == 'source'
end
end
Batch = Struct.new(:items, keyword_init: true) do
def entries
items
end
def source_entries
items.select(&:source?)
end
def part_entries
items.reject(&:source?)
end
end
def self.activity_part?(part)
part_type = part['part_type'].to_s
part_type.present? && REGULAR_PART_TYPES.exclude?(part_type)
end
def self.source_message_importable?(source)
source['body'].present? || source['subject'].present? || source['attachments'].present?
end
class DataImports::Intercom::MessageBatchBuilder < DataImports::MessageBatchBuilder
def initialize(data_import:, conversation:, source_conversation:)
@data_import = data_import
@account = data_import.account
@conversation = conversation
@source_conversation = source_conversation
end
def perform(source_entries = unprepared_entries)
classify(source_entries)
end
def refresh(entries)
classify(entries.map do |entry|
{ source_id: entry.source_id, part: entry.part, position: entry.position }
end)
end
def unprepared_entries
ordered_source_entries.map.with_index { |entry, position| entry.merge(position: position) }
end
private
def classify(source_entries)
return Batch.new(items: []) if source_entries.empty?
mappings = message_mappings(source_entries)
messages = messages_for(source_entries, mappings)
Batch.new(items: source_entries.map.with_index do |source_entry, position|
build_entry(source_entry, source_entry.fetch(:position, position), mappings, messages)
end)
end
def ordered_source_entries
entries = []
source = @source_conversation['source'].to_h
if self.class.source_message_importable?(source)
entries << {
source_id: "conversation:#{source_conversation_id}:source:#{source['id'].presence || 'initial'}",
part: source.merge('part_type' => 'source', 'created_at' => @source_conversation['created_at'])
}
end
conversation_parts.each do |part|
entries << { source_id: "conversation:#{source_conversation_id}:part:#{part['id']}", part: part }
end
entries
end
def conversation_parts
Array(@source_conversation.dig('conversation_parts', 'conversation_parts'))
end
def source_conversation_id
@source_conversation['id'].presence || @source_conversation['external_id'].presence || @source_conversation['email'].presence
end
def message_mappings(source_entries)
DataImportMapping.where(
account: @account,
source_provider: PROVIDER,
source_object_type: 'message',
source_object_id: source_entries.pluck(:source_id)
).index_by(&:source_object_id)
end
def messages_for(source_entries, mappings)
mapped_message_ids = mappings.values.filter_map do |mapping|
mapping.chatwoot_record_id if mapping.chatwoot_record_type == 'Message'
end
chatwoot_source_ids = source_entries.map { |entry| "intercom:#{entry[:source_id]}" }
messages = Message.where(id: mapped_message_ids).or(
Message.where(conversation_id: @conversation.id, source_id: chatwoot_source_ids)
).to_a
{
by_id: messages.index_by(&:id),
by_source_id: messages.index_by(&:source_id)
}
end
def build_entry(source_entry, position, mappings, messages)
source_id = source_entry[:source_id]
mapping = mappings[source_id]
mapped_message = messages[:by_id][mapping.chatwoot_record_id] if mapping&.chatwoot_record_type == 'Message'
existing_message = messages[:by_source_id]["intercom:#{source_id}"]
Entry.new(
source_id: source_id,
part: source_entry[:part],
position: position,
mapping: mapping,
message: mapped_message || existing_message,
classification: classification_for(mapping, mapped_message, existing_message, source_entry[:part])
)
end
def classification_for(mapping, mapped_message, existing_message, part)
return existing_message.present? ? :existing_message : :new_message if mapping.blank?
return :repairable_stale_mapping unless mapping_handled?(mapping, mapped_message, part)
mapping.data_import_id == @data_import.id ? :current_import : :previous_import
end
def mapping_handled?(mapping, mapped_message, part)
return false if mapping.metadata['skipped'] && self.class.activity_part?(part)
mapping.metadata['skipped'] || mapped_message.present?
super(data_import: data_import, conversation: conversation, source_conversation: source_conversation, provider: 'intercom')
end
end

View File

@@ -1,41 +1,5 @@
class DataImports::Intercom::PlaceholderInboxBuilder
AGENT_REPLY_TIME_WINDOW_HOURS = 1
class DataImports::Intercom::PlaceholderInboxBuilder < DataImports::PlaceholderInboxBuilder
def initialize(account:)
@account = account
end
def inbox_for(source_type)
bucket = DataImports::Intercom::SourceBucket.for(source_type)
placeholder_inboxes[bucket[:key]] ||= create_placeholder_inbox(bucket)
end
private
def placeholder_inboxes
@placeholder_inboxes ||= @account.inboxes.includes(:channel).where(channel_type: 'Channel::Api').each_with_object({}) do |inbox, inboxes|
attrs = inbox.channel.additional_attributes || {}
next unless attrs['source_provider'] == 'intercom' && attrs['import_placeholder'] == true
inboxes[attrs['source_bucket']] = inbox
end
end
def create_placeholder_inbox(bucket)
channel = @account.api_channels.create!(
additional_attributes: {
source_provider: 'intercom',
source_bucket: bucket[:key],
import_placeholder: true,
agent_reply_time_window: AGENT_REPLY_TIME_WINDOW_HOURS
}
)
@account.inboxes.create!(
name: "Intercom Import - #{bucket[:name]}",
channel: channel,
enable_auto_assignment: false,
allow_messages_after_resolved: false
)
super(account: account, provider: 'intercom', source_bucket: DataImports::Intercom::SourceBucket)
end
end

View File

@@ -1,55 +1,2 @@
class DataImports::Intercom::RestartService
attr_reader :data_import
def initialize(account:, data_import:)
@account = account
@data_import = data_import
end
def perform
@account.with_lock do
@data_import.reload
next :render_show unless @data_import.restartable?
if (active_import = find_active_import)
@data_import = active_import
next :render_show
end
next :access_token_missing if @data_import.access_token.blank?
@data_import.assign_active_intercom_import_run_id
retained_skip_logs = @data_import.import_errors.where("details ->> 'kind' = ?", 'skipped')
@data_import.import_errors.where.not(id: retained_skip_logs.select(:id)).delete_all
@data_import.update!(restart_attributes(retained_skip_logs))
:enqueue
end
end
private
def find_active_import
@account.data_imports.active_intercom.first
end
def restart_attributes(retained_skip_logs)
{
status: :pending,
abandoned_at: nil,
completed_at: nil,
last_error_at: nil,
started_at: nil,
stats: restart_stats(retained_skip_logs)
}
end
def restart_stats(retained_skip_logs)
@data_import.stats.to_h.deep_dup.tap do |stats|
%w[contact conversation message].each do |object_type|
stats["#{object_type}s"] ||= {}
stats["#{object_type}s"]['skipped'] = retained_skip_logs.where(source_object_type: object_type).count
end
stats['errors'] = { 'count' => 0 }
end
end
class DataImports::Intercom::RestartService < DataImports::RestartService
end

View File

@@ -1,28 +1,2 @@
class DataImports::Intercom::RetryService
attr_reader :data_import
def initialize(account:, data_import:)
@account = account
@data_import = data_import
end
def perform
@account.with_lock do
@data_import.with_lock do
next :not_stalled unless @data_import.stalled?
next :active_import_exists if another_active_import?
next :access_token_missing if @data_import.access_token.blank?
@data_import.assign_active_intercom_import_run_id
@data_import.update!(status: :pending)
:enqueue
end
end
end
private
def another_active_import?
@account.data_imports.active_intercom.where.not(id: @data_import.id).exists?
end
class DataImports::Intercom::RetryService < DataImports::RetryService
end

View File

@@ -0,0 +1,181 @@
class DataImports::Intercom::Source
PROVIDER = 'intercom'.freeze
DISPLAY_NAME = 'Intercom'.freeze
CONTACTS_PER_PAGE = 50
CONVERSATIONS_PER_PAGE = 10
ALREADY_IMPORTED_ERROR_CODE = 'DataImports::Intercom::AlreadyImported'.freeze
SKIPPED_MESSAGE_ERROR_CODE = 'DataImports::Intercom::SkippedMessage'.freeze
TRUNCATED_PARTS_ERROR_CODE = 'DataImports::Intercom::TruncatedConversationParts'.freeze
attr_reader :provider, :display_name, :contacts_per_page, :conversations_per_page
def self.credentials_validator(source_params:, import_types:)
DataImports::Intercom::CredentialsValidator.new(
access_token: source_params[:access_token],
import_types: import_types
)
end
def self.source_metadata(_source_params)
{}
end
def self.default_import_name
'Intercom import'
end
def self.credential_name
'access key'
end
def self.import_job_class
DataImports::Intercom::ImportJob
end
def self.importer_class
DataImports::Intercom::Importer
end
def self.contacts_page_job_class
DataImports::Intercom::ContactsPageJob
end
def self.conversations_page_job_class
DataImports::Intercom::ConversationsPageJob
end
def self.client_error?(error)
error.is_a?(DataImports::Intercom::Client::Error)
end
def self.authentication_error?(error)
error.is_a?(DataImports::Intercom::Client::AuthenticationError)
end
def initialize(access_token:, source_metadata: {}) # rubocop:disable Lint/UnusedMethodArgument
@provider = PROVIDER
@display_name = DISPLAY_NAME
@contacts_per_page = CONTACTS_PER_PAGE
@conversations_per_page = CONVERSATIONS_PER_PAGE
@client = DataImports::Intercom::Client.new(access_token: access_token)
end
def list_contacts(starting_after:, per_page:)
@client.list_contacts(starting_after: starting_after, per_page: per_page)
end
def list_conversations(starting_after:, per_page:)
@client.list_conversations(starting_after: starting_after, per_page: per_page)
end
def retrieve_conversation(id)
@client.retrieve_conversation(id)
end
def retrieve_contact(id)
@client.retrieve_contact(id)
end
def client_error?(error)
error.is_a?(DataImports::Intercom::Client::Error)
end
def placeholder_inbox_builder(account:)
DataImports::Intercom::PlaceholderInboxBuilder.new(account: account)
end
def message_batch_builder(data_import:, conversation:, source_conversation:)
DataImports::Intercom::MessageBatchBuilder.new(
data_import: data_import,
conversation: conversation,
source_conversation: source_conversation
)
end
def activity_part?(part)
DataImports::Intercom::MessageBatchBuilder.activity_part?(part)
end
def source_message_importable?(source)
DataImports::Intercom::MessageBatchBuilder.source_message_importable?(source)
end
def activity_content(part)
DataImports::Intercom::ActivityContentBuilder.new(part).perform
end
def already_imported_error_code
ALREADY_IMPORTED_ERROR_CODE
end
def skipped_message_error_code
SKIPPED_MESSAGE_ERROR_CODE
end
def truncated_parts_error_code
TRUNCATED_PARTS_ERROR_CODE
end
def skipped_message_reason
'blank_or_unsupported_intercom_part'
end
def contact_custom_attributes(contact_payload)
{
intercom_contact_id: contact_payload['id'],
intercom_external_id: contact_payload['external_id']
}.compact
end
def contact_source_metadata(contact_payload)
{
contact_id: contact_payload['id'],
external_id: contact_payload['external_id'],
raw_phone: contact_payload['phone']
}.compact
end
def conversation_custom_attributes(conversation)
{ intercom_conversation_id: source_id_for(conversation) }
end
def conversation_source_metadata(conversation)
{
conversation_id: source_id_for(conversation),
delivered_as: conversation.dig('source', 'delivered_as'),
source_url: conversation.dig('source', 'url'),
admin_assignee_id: conversation['admin_assignee_id'],
team_assignee_id: conversation['team_assignee_id'],
state: conversation['state'],
open: conversation['open']
}.compact
end
def message_source_metadata(part)
{
part_id: part['id'],
part_type: part['part_type'],
author: part['author'],
assigned_to: part['assigned_to'],
state: part['state'],
tags: part['tags'],
event_details: part['event_details'],
app_package_code: part['app_package_code'],
metadata: part['metadata'],
attachments: part['attachments'],
redacted: part['redacted']
}.compact
end
def timestamp_for(value)
return Time.current if value.blank?
Time.zone.at(value.to_i)
end
private
def source_id_for(payload)
payload['id'].presence || payload['external_id'].presence || payload['email'].presence
end
end

View File

@@ -0,0 +1,144 @@
class DataImports::MessageBatchBuilder
REGULAR_PART_TYPES = %w[comment note source].freeze
Entry = Struct.new(:source_id, :part, :position, :mapping, :message, :classification, keyword_init: true) do
def source?
part['part_type'] == 'source'
end
end
Batch = Struct.new(:items, keyword_init: true) do
def entries
items
end
def source_entries
items.select(&:source?)
end
def part_entries
items.reject(&:source?)
end
end
def self.activity_part?(part)
part_type = part['part_type'].to_s
part_type.present? && REGULAR_PART_TYPES.exclude?(part_type)
end
def self.source_message_importable?(source)
source['body'].present? || source['subject'].present? || source['attachments'].present?
end
def initialize(data_import:, conversation:, source_conversation:, provider:)
@data_import = data_import
@account = data_import.account
@conversation = conversation
@source_conversation = source_conversation
@provider = provider
end
def perform(source_entries = unprepared_entries)
classify(source_entries)
end
def refresh(entries)
classify(entries.map do |entry|
{ source_id: entry.source_id, part: entry.part, position: entry.position }
end)
end
def unprepared_entries
ordered_source_entries.map.with_index { |entry, position| entry.merge(position: position) }
end
private
def classify(source_entries)
return Batch.new(items: []) if source_entries.empty?
mappings = message_mappings(source_entries)
messages = messages_for(source_entries, mappings)
Batch.new(items: source_entries.map.with_index do |source_entry, position|
build_entry(source_entry, source_entry.fetch(:position, position), mappings, messages)
end)
end
def ordered_source_entries
entries = []
source = @source_conversation['source'].to_h
if self.class.source_message_importable?(source)
entries << {
source_id: "conversation:#{source_conversation_id}:source:#{source['id'].presence || 'initial'}",
part: source.merge('part_type' => 'source', 'created_at' => @source_conversation['created_at'])
}
end
conversation_parts.each do |part|
entries << { source_id: "conversation:#{source_conversation_id}:part:#{part['id']}", part: part }
end
entries
end
def conversation_parts
Array(@source_conversation.dig('conversation_parts', 'conversation_parts'))
end
def source_conversation_id
@source_conversation['id'].presence || @source_conversation['external_id'].presence || @source_conversation['email'].presence
end
def message_mappings(source_entries)
DataImportMapping.where(
account: @account,
source_provider: @provider,
source_object_type: 'message',
source_object_id: source_entries.pluck(:source_id)
).index_by(&:source_object_id)
end
def messages_for(source_entries, mappings)
mapped_message_ids = mappings.values.filter_map do |mapping|
mapping.chatwoot_record_id if mapping.chatwoot_record_type == 'Message'
end
chatwoot_source_ids = source_entries.map { |entry| "#{@provider}:#{entry[:source_id]}" }
messages = Message.where(id: mapped_message_ids).or(
Message.where(conversation_id: @conversation.id, source_id: chatwoot_source_ids)
).to_a
{
by_id: messages.index_by(&:id),
by_source_id: messages.index_by(&:source_id)
}
end
def build_entry(source_entry, position, mappings, messages)
source_id = source_entry[:source_id]
mapping = mappings[source_id]
mapped_message = messages[:by_id][mapping.chatwoot_record_id] if mapping&.chatwoot_record_type == 'Message'
existing_message = messages[:by_source_id]["#{@provider}:#{source_id}"]
Entry.new(
source_id: source_id,
part: source_entry[:part],
position: position,
mapping: mapping,
message: mapped_message || existing_message,
classification: classification_for(mapping, mapped_message, existing_message, source_entry[:part])
)
end
def classification_for(mapping, mapped_message, existing_message, part)
return existing_message.present? ? :existing_message : :new_message if mapping.blank?
return :repairable_stale_mapping unless mapping_handled?(mapping, mapped_message, part)
mapping.data_import_id == @data_import.id ? :current_import : :previous_import
end
def mapping_handled?(mapping, mapped_message, part)
return false if mapping.metadata['skipped'] && self.class.activity_part?(part)
mapping.metadata['skipped'] || mapped_message.present?
end
end

View File

@@ -0,0 +1,65 @@
class DataImports::PlaceholderInboxBuilder
AGENT_REPLY_TIME_WINDOW_HOURS = 1
def initialize(account:, provider:, source_bucket:)
@account = account
@provider = provider
@source_bucket = source_bucket
end
def inbox_for(source_type)
bucket = @source_bucket.for(source_type)
placeholder_inboxes[bucket[:key]] ||= create_placeholder_inbox(bucket)
end
private
def placeholder_inboxes
@placeholder_inboxes ||= @account.inboxes.includes(:channel).where(channel_type: 'Channel::Api').each_with_object({}) do |inbox, inboxes|
attrs = inbox.channel.additional_attributes || {}
next unless attrs['source_provider'] == @provider && attrs['import_placeholder'] == true
inboxes[attrs['source_bucket']] = inbox
end
end
def create_placeholder_inbox(bucket)
Inbox.transaction do
channel = create_placeholder_channel(bucket)
now = Time.current
result = Inbox.insert_all!( # rubocop:disable Rails/SkipsModelValidations
[placeholder_inbox_attributes(channel, bucket, now)],
returning: %w[id]
)
Inbox.find(result.rows.first.first).tap { |inbox| create_default_working_hours(inbox) }
end
end
def create_placeholder_channel(bucket)
@account.api_channels.create!(
additional_attributes: {
source_provider: @provider,
source_bucket: bucket[:key],
import_placeholder: true,
agent_reply_time_window: AGENT_REPLY_TIME_WINDOW_HOURS
}
)
end
def placeholder_inbox_attributes(channel, bucket, timestamp)
{
account_id: @account.id,
channel_id: channel.id,
channel_type: channel.class.name,
name: "#{@provider.titleize} Import - #{bucket[:name]}",
enable_auto_assignment: false,
allow_messages_after_resolved: false,
created_at: timestamp,
updated_at: timestamp
}
end
def create_default_working_hours(inbox)
OutOfOffisable::DEFAULT_WORKING_HOURS.each { |attributes| inbox.working_hours.create!(attributes) }
end
end

View File

@@ -0,0 +1,56 @@
class DataImports::RestartService
attr_reader :data_import
def initialize(account:, data_import:)
@account = account
@data_import = data_import
end
def perform
@account.with_lock do
@data_import.reload
next :render_show unless @data_import.integration_import?
next :render_show unless @data_import.restartable?
if (active_import = find_active_import)
@data_import = active_import
next :render_show
end
next :access_token_missing if @data_import.access_token.blank?
@data_import.assign_active_import_run_id
retained_skip_logs = @data_import.import_errors.where("details ->> 'kind' = ?", 'skipped')
@data_import.import_errors.where.not(id: retained_skip_logs.select(:id)).delete_all
@data_import.update!(restart_attributes(retained_skip_logs))
:enqueue
end
end
private
def find_active_import
@account.data_imports.active_integrations.first
end
def restart_attributes(retained_skip_logs)
{
status: :pending,
abandoned_at: nil,
completed_at: nil,
last_error_at: nil,
started_at: nil,
stats: restart_stats(retained_skip_logs)
}
end
def restart_stats(retained_skip_logs)
@data_import.stats.to_h.deep_dup.tap do |stats|
%w[contact conversation message].each do |object_type|
stats["#{object_type}s"] ||= {}
stats["#{object_type}s"]['skipped'] = retained_skip_logs.where(source_object_type: object_type).count
end
stats['errors'] = { 'count' => 0 }
end
end
end

View File

@@ -0,0 +1,28 @@
class DataImports::RetryService
attr_reader :data_import
def initialize(account:, data_import:)
@account = account
@data_import = data_import
end
def perform
@account.with_lock do
@data_import.with_lock do
next :not_stalled unless @data_import.stalled?
next :active_import_exists if another_active_import?
next :access_token_missing if @data_import.access_token.blank?
@data_import.assign_active_import_run_id
@data_import.update!(status: :pending)
:enqueue
end
end
end
private
def another_active_import?
@account.data_imports.active_integrations.where.not(id: @data_import.id).exists?
end
end

View File

@@ -0,0 +1,21 @@
class DataImports::Source
PROVIDERS = {
'freshdesk' => 'DataImports::Freshdesk::Source',
'intercom' => 'DataImports::Intercom::Source'
}.freeze
def self.for(data_import)
source_class(data_import.source_provider).new(
access_token: data_import.access_token,
source_metadata: data_import.source_metadata
)
end
def self.source_class(provider)
PROVIDERS.fetch(provider.to_s) { raise ArgumentError, 'Unsupported import source.' }.constantize
end
def self.supported?(provider)
PROVIDERS.key?(provider.to_s)
end
end

View File

@@ -0,0 +1,13 @@
class CustomExceptions::DataImport::FreshdeskTicketLimitError < CustomExceptions::Base
MESSAGE = 'Freshdesk limits ticket listing to 300 pages. This import stopped after 30,000 tickets because the API cannot ' \
'confirm that the ticket history is complete. A Freshdesk admin Account Export is required for any remaining history, ' \
'but this importer does not support account exports yet.'.freeze
def initialize
super({})
end
def message
MESSAGE
end
end

View File

@@ -12,5 +12,15 @@ FactoryBot.define do
access_token { 'intercom-token' }
import_file { nil }
end
trait :freshdesk do
data_type { 'freshdesk' }
source_type { 'api' }
source_provider { 'freshdesk' }
import_types { %w[contacts conversations] }
access_token { 'freshdesk-api-key' }
source_metadata { { domain: 'acme.freshdesk.com' } }
import_file { nil }
end
end
end

View File

@@ -0,0 +1,22 @@
# Freshdesk fixture provenance
These fixtures contain no production credentials, signed attachment URLs, or
personal data. Their field shapes were derived from the Freshdesk API v2
documentation, sanitized examples in these public repositories, and a
sanitized Freshdesk trial-account Web Chat response captured during live API
validation:
- [`Aaronontheweb/freshdesk-cli` ticket with conversations](https://github.com/Aaronontheweb/freshdesk-cli/blob/c87b74043a1662646378cd3385ec0db9af6fa28c/tests/TestData/FreshdeskResponses/ticket_with_conversations.json)
- [`airbytehq/airbyte` Freshdesk expected records](https://github.com/airbytehq/airbyte/blob/0673d69418fe3874a6970d6113eec891db4a296b/airbyte-integrations/connectors/source-freshdesk/integration_tests/expected_records.jsonl)
- [`pbrane/freshdesk-api-client` conversation fixture](https://github.com/pbrane/freshdesk-api-client/blob/c6685af0bf2da889b4a89ccb6cfeb53e65eb5575/src/main/resources/freshdesk/documents/tac-case-notes/tacCase99Conversations.json)
- [`freshworks-developers/fw-attach` conversation event fixture](https://github.com/freshworks-developers/fw-attach/blob/45b7ea1ad6b9008b9edb73ecca349ee9d3b28a6d/server/test_data/support_ticket/onConversationCreate.json)
The fixture intentionally covers an incoming ticket description, a public
agent reply, a private note with attachment metadata, and a later customer
reply.
`web_chat_ticket_with_conversations.json` covers Freshdesk's current Web Chat
ticket source (`15`), structured conversation bodies, the generated greeting,
the initial customer message repeated in the ticket description, subsequent
replies, and real attachment metadata keys. IDs, timestamps, addresses, names,
content, and attachment URLs were replaced or removed.

View File

@@ -0,0 +1,20 @@
{
"id": 1001,
"active": true,
"email": "customer@example.com",
"name": "Customer Example",
"mobile": "+15551234567",
"phone": null,
"created_at": "2024-01-01T09:00:00Z",
"updated_at": "2024-01-04T12:30:00Z",
"other_emails": [
"billing@example.com"
],
"unique_external_id": "customer-1001",
"custom_fields": {
"support_plan": "growth"
},
"tags": [
"priority"
]
}

View File

@@ -0,0 +1,98 @@
{
"ticket": {
"id": 2001,
"subject": "Unable to finish checkout",
"description": "<p>The checkout button is not responding.</p>",
"description_text": "The checkout button is not responding.",
"status": 4,
"priority": 2,
"source": 1,
"requester_id": 1001,
"responder_id": 501,
"group_id": 42,
"company_id": 77,
"tags": [
"checkout"
],
"custom_fields": {
"product_area": "payments"
},
"created_at": "2024-01-02T10:00:00Z",
"updated_at": "2024-01-04T12:30:00Z",
"attachments": [],
"requester": {
"id": 1001,
"active": true,
"email": "customer@example.com",
"name": "Customer Example",
"mobile": "+15551234567",
"created_at": "2024-01-01T09:00:00Z",
"updated_at": "2024-01-04T12:30:00Z",
"unique_external_id": "customer-1001"
}
},
"conversations": [
{
"id": 3003,
"body": "<p>The issue still happens in a private window.</p>",
"body_text": "The issue still happens in a private window.",
"incoming": true,
"private": false,
"source": 0,
"user_id": 1001,
"from_email": "customer@example.com",
"to_emails": [
"support@example.com"
],
"cc_emails": [],
"bcc_emails": [],
"created_at": "2024-01-04T12:30:00Z",
"updated_at": "2024-01-04T12:30:00Z",
"attachments": [],
"deleted": false
},
{
"id": 3001,
"body": "<p>Could you try this in a private window?</p>",
"body_text": "Could you try this in a private window?",
"incoming": false,
"private": false,
"source": 0,
"user_id": 501,
"from_email": "agent@example.com",
"to_emails": [
"customer@example.com"
],
"cc_emails": [],
"bcc_emails": [],
"created_at": "2024-01-03T08:00:00Z",
"updated_at": "2024-01-03T08:00:00Z",
"attachments": [],
"deleted": false
},
{
"id": 3002,
"body": "<p>Payments team is investigating the browser logs.</p>",
"body_text": "Payments team is investigating the browser logs.",
"incoming": false,
"private": true,
"source": 2,
"user_id": 501,
"from_email": "agent@example.com",
"to_emails": [],
"cc_emails": [],
"bcc_emails": [],
"created_at": "2024-01-03T09:00:00Z",
"updated_at": "2024-01-03T09:00:00Z",
"attachments": [
{
"id": 4001,
"name": "browser-log.txt",
"content_type": "text/plain",
"size": 128
}
],
"deleted": false
}
]
}

View File

@@ -0,0 +1,183 @@
{
"ticket": {
"id": 2101,
"subject": "FD-IMPORT-WEBCHAT-002 — Brew temperature drops",
"description": "<div>Ticket Description: FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.</div>",
"description_text": "Ticket Description: FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.",
"status": 2,
"priority": 1,
"source": 15,
"source_info": 1,
"requester_id": 1101,
"responder_id": 510,
"group_id": null,
"company_id": null,
"tags": [],
"custom_fields": {},
"created_at": "2026-07-31T08:00:00Z",
"updated_at": "2026-07-31T08:10:00Z",
"attachments": [],
"requester": {
"id": 1101,
"email": null,
"name": "",
"mobile": null,
"phone": null
}
},
"conversations": [
{
"id": 3101,
"body": "<div>Hello! How can we help you today?</div>",
"body_text": "Hello! How can we help you today?",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "Hello! How can we help you today?"
}
}
]
},
"incoming": false,
"private": false,
"source": 15,
"category": 5,
"user_id": null,
"from_email": null,
"created_at": "2026-07-31T08:00:00Z",
"updated_at": "2026-07-31T08:00:00Z",
"attachments": []
},
{
"id": 3102,
"body": "<div>FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.</div>",
"body_text": "FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups."
}
}
]
},
"incoming": true,
"private": false,
"source": 20,
"category": 1,
"user_id": 1101,
"from_email": null,
"created_at": "2026-07-31T08:00:01Z",
"updated_at": "2026-07-31T08:00:01Z",
"attachments": []
},
{
"id": 3103,
"body": "<div>The display reads 82°C instead of 93°C.</div>",
"body_text": "The display reads 82°C instead of 93°C.",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "The display reads 82°C instead of 93°C."
}
}
]
},
"incoming": true,
"private": false,
"source": 20,
"category": 1,
"user_id": 1101,
"from_email": null,
"created_at": "2026-07-31T08:01:00Z",
"updated_at": "2026-07-31T08:01:00Z",
"attachments": []
},
{
"id": 3104,
"body": "<div>Please run one rinse cycle, then check the next brew.</div>",
"body_text": "Please run one rinse cycle, then check the next brew.",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "Please run one rinse cycle, then check the next brew."
}
}
]
},
"incoming": false,
"private": false,
"source": 20,
"category": 3,
"user_id": 510,
"from_email": null,
"created_at": "2026-07-31T08:03:00Z",
"updated_at": "2026-07-31T08:03:00Z",
"attachments": []
},
{
"id": 3105,
"body": "<div>The next brew reached 89°C.</div>",
"body_text": "The next brew reached 89°C.",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "The next brew reached 89°C."
}
}
]
},
"incoming": true,
"private": false,
"source": 20,
"category": 1,
"user_id": 1101,
"from_email": null,
"created_at": "2026-07-31T08:05:00Z",
"updated_at": "2026-07-31T08:05:00Z",
"attachments": []
},
{
"id": 3106,
"body": "<div>Attached is a harmless text diagnostic.</div>",
"body_text": "Attached is a harmless text diagnostic.",
"structured_body": {
"body_contents": [
{
"type": "text",
"data": {
"content": "Attached is a harmless text diagnostic."
}
}
]
},
"incoming": false,
"private": false,
"source": 20,
"category": 3,
"user_id": 510,
"from_email": "support@example.com",
"created_at": "2026-07-31T08:10:00Z",
"updated_at": "2026-07-31T08:10:00Z",
"attachments": [
{
"id": 4101,
"content_type": "text/plain",
"size": 188,
"name": "freshdesk-validation-diagnostic.txt",
"created_at": "2026-07-31T08:10:00Z",
"updated_at": "2026-07-31T08:10:00Z"
}
]
}
]
}

View File

@@ -0,0 +1,96 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::ImportJob do
let(:account) { create(:account) }
let(:data_import) { create(:data_import, :freshdesk, account: account) }
let(:importer) { instance_double(DataImports::Freshdesk::Importer) }
let(:run_id) { 'freshdesk-run-1' }
before do
account.enable_features!('data_import')
data_import.update!(source_metadata: data_import.source_metadata.merge(DataImport::ACTIVE_IMPORT_RUN_ID_KEY => run_id))
allow(DataImports::Freshdesk::Importer).to receive(:new).with(data_import: data_import, run_id: run_id).and_return(importer)
end
describe DataImports::Freshdesk::BaseJob do
it 'checks rate limit retry before the generic client retry' do
handlers = described_class.rescue_handlers.map(&:first)
expect(handlers.index('DataImports::Freshdesk::Client::RateLimitError')).to be > handlers.index('DataImports::Freshdesk::Client::Error')
end
it 'schedules a rate-limit retry using the Freshdesk Retry-After header' do
error = DataImports::Freshdesk::Client::RateLimitError.new('Rate limited', retry_after: '42')
allow(importer).to receive(:start!).and_raise(error)
travel_to(Time.zone.parse('2026-07-30 12:00:00 UTC')) do
expect do
DataImports::Freshdesk::ImportJob.perform_now(data_import, run_id)
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).at(42.seconds.from_now)
end
end
it 'discards a terminal ticket-limit error after recording the failed import' do
error = CustomExceptions::DataImport::FreshdeskTicketLimitError.new
allow(importer).to receive_messages(conversations_completed?: false, fail!: true)
allow(importer).to receive(:import_conversations_page).with(starting_after: nil).and_raise(error)
expect do
DataImports::Freshdesk::ConversationsPageJob.perform_now(data_import, nil, run_id)
end.not_to have_enqueued_job
expect(importer).to have_received(:fail!).with(error)
end
end
describe DataImports::Freshdesk::ImportJob do
it 'starts the import and enqueues the first contacts page' do
allow(importer).to receive_messages(start!: true, import_contacts?: true, contacts_completed?: false, cursor_for: 2)
expect do
described_class.perform_now(data_import, run_id)
end.to have_enqueued_job(DataImports::Freshdesk::ContactsPageJob).with(data_import, 2, run_id).on_queue('low')
expect(importer).to have_received(:start!)
end
it 'skips stale import jobs from an earlier run' do
data_import.update!(source_metadata: data_import.source_metadata.merge(DataImport::ACTIVE_IMPORT_RUN_ID_KEY => 'new-run'))
expect(DataImports::Freshdesk::Importer).not_to receive(:new)
described_class.perform_now(data_import, 'old-run')
end
end
describe DataImports::Freshdesk::ContactsPageJob do
it 'hands off to tickets after the final contacts page' do
result = DataImports::Freshdesk::Importer::PageResult.new(next_cursor: nil)
allow(importer).to receive_messages(
contacts_completed?: false,
import_conversations?: true,
conversations_completed?: false
)
allow(importer).to receive(:import_contacts_page).with(starting_after: nil).and_return(result)
allow(importer).to receive(:cursor_for).with('conversations').and_return(1)
expect do
described_class.perform_now(data_import, nil, run_id)
end.to have_enqueued_job(DataImports::Freshdesk::ConversationsPageJob).with(data_import, 1, run_id)
end
end
describe DataImports::Freshdesk::ConversationsPageJob do
it 'finishes after the final tickets page' do
result = DataImports::Freshdesk::Importer::PageResult.new(next_cursor: nil)
allow(importer).to receive_messages(conversations_completed?: false, finish!: true)
allow(importer).to receive(:import_conversations_page).with(starting_after: nil).and_return(result)
expect do
described_class.perform_now(data_import, nil, run_id)
end.not_to have_enqueued_job
expect(importer).to have_received(:finish!)
end
end
end

View File

@@ -9,6 +9,13 @@ RSpec.describe DataImport do
it 'returns false for invalid data type' do
expect(build(:data_import, data_type: 'Xyc').valid?).to be false
end
it 'requires an integration provider to match its data type' do
data_import = build(:data_import, :freshdesk, source_provider: 'intercom')
expect(data_import).not_to be_valid
expect(data_import.errors[:source_provider]).to include('must match the integration data type')
end
end
describe 'access token encryption' do
@@ -80,6 +87,13 @@ RSpec.describe DataImport do
end
end
it 'identifies stalled Freshdesk imports' do
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :processing)
freshdesk_import.update!(updated_at: 15.minutes.ago)
expect(freshdesk_import.reload).to be_stalled
end
it 'does not identify recent or terminal Intercom imports as stalled', :aggregate_failures do
recent_import = create(:data_import, :intercom, account: account, status: :processing)
completed_import = create(:data_import, :intercom, account: account, status: :completed)

View File

@@ -4,10 +4,12 @@ RSpec.describe 'Data Imports API', type: :request do
let(:account) { create(:account) }
let(:admin) { create(:user, account: account, role: :administrator) }
let(:validator) { instance_double(DataImports::Intercom::CredentialsValidator, perform: { 'contacts' => 12, 'conversations' => 8 }) }
let(:freshdesk_validator) { instance_double(DataImports::Freshdesk::CredentialsValidator, perform: {}) }
before do
account.enable_features!('data_import')
allow(DataImports::Intercom::CredentialsValidator).to receive(:new).and_return(validator)
allow(DataImports::Freshdesk::CredentialsValidator).to receive(:new).and_return(freshdesk_validator)
end
describe 'POST /api/v1/accounts/:account_id/data_imports/validate_source' do
@@ -37,6 +39,49 @@ RSpec.describe 'Data Imports API', type: :request do
'message' => 'We could not validate this Intercom access key. Check the key and its permissions.'
)
end
it 'validates a Freshdesk domain and API key through its source adapter', :aggregate_failures do
post validate_source_api_v1_account_data_imports_url(account_id: account.id),
params: {
source_provider: 'freshdesk',
domain: 'acme.freshdesk.com',
access_token: 'freshdesk-api-key',
import_types: %w[contacts conversations]
},
headers: admin.create_new_auth_token,
as: :json
expect(response).to have_http_status(:ok)
expect(response.parsed_body).to eq('valid' => true, 'totals' => {})
expect(DataImports::Freshdesk::CredentialsValidator).to have_received(:new).with(
domain: 'acme.freshdesk.com',
api_key: 'freshdesk-api-key',
import_types: %w[contacts conversations]
)
end
it 'returns a safe Freshdesk authentication error' do
allow(freshdesk_validator).to receive(:perform).and_raise(
DataImports::Freshdesk::Client::AuthenticationError,
'provider response'
)
post validate_source_api_v1_account_data_imports_url(account_id: account.id),
params: {
source_provider: 'freshdesk',
domain: 'acme.freshdesk.com',
access_token: 'invalid',
import_types: %w[contacts]
},
headers: admin.create_new_auth_token,
as: :json
expect(response).to have_http_status(:unprocessable_entity)
expect(response.parsed_body).to eq(
'valid' => false,
'message' => 'We could not validate this Freshdesk API key. Check the key and its permissions.'
)
end
end
describe 'POST /api/v1/accounts/:account_id/data_imports' do
@@ -87,6 +132,36 @@ RSpec.describe 'Data Imports API', type: :request do
expect(response.parsed_body).not_to have_key('access_token')
end
it 'creates and enqueues a Freshdesk import', :aggregate_failures do
expect do
post api_v1_account_data_imports_url(account_id: account.id),
params: {
name: 'Freshdesk migration',
source_provider: 'freshdesk',
domain: 'https://ACME.freshdesk.com/support/home',
access_token: 'freshdesk-api-key',
import_types: %w[contacts conversations]
},
headers: admin.create_new_auth_token,
as: :json
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob)
expect(response).to have_http_status(:ok)
data_import = account.data_imports.last
expect(data_import).to have_attributes(
name: 'Freshdesk migration',
data_type: 'freshdesk',
source_type: 'api',
source_provider: 'freshdesk',
initiated_by_id: admin.id
)
expect(data_import.access_token).to eq('freshdesk-api-key')
expect(data_import.source_metadata).to include('domain' => 'acme.freshdesk.com')
expect(data_import.active_import_run_id).to be_present
expect(response.parsed_body['source_provider']).to eq('freshdesk')
expect(response.parsed_body).not_to have_key('access_token')
end
it 'rejects creation while another Intercom import is active' do
active_import = create(
:data_import, :intercom,
@@ -110,6 +185,26 @@ RSpec.describe 'Data Imports API', type: :request do
expect(active_import.reload).to be_processing
end
it 'rejects a Freshdesk import while another integration import is active' do
create(:data_import, :intercom, account: account, status: :processing)
expect do
post api_v1_account_data_imports_url(account_id: account.id),
params: {
source_provider: 'freshdesk',
domain: 'acme.freshdesk.com',
access_token: 'freshdesk-api-key',
import_types: %w[contacts conversations]
},
headers: admin.create_new_auth_token,
as: :json
end.not_to have_enqueued_job(DataImports::Freshdesk::ImportJob)
expect(response).to have_http_status(:unprocessable_entity)
expect(response.parsed_body['message']).to eq('Another data import is already in progress.')
expect(account.data_imports.where(source_provider: 'freshdesk')).to be_empty
end
it 'rejects unsupported import types instead of silently importing everything' do
allow(validator).to receive(:perform).and_raise(ArgumentError, 'Unsupported import types: companies')
@@ -207,6 +302,20 @@ RSpec.describe 'Data Imports API', type: :request do
expect(response.parsed_body['message']).to eq('The Intercom access key for this import is unavailable.')
expect(data_import.reload).to be_abandoned
end
it 'restarts an abandoned Freshdesk import with the Freshdesk job' do
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :abandoned, abandoned_at: 1.hour.ago)
expect do
post start_api_v1_account_data_import_url(account_id: account.id, id: freshdesk_import.id),
headers: admin.create_new_auth_token,
as: :json
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).with(freshdesk_import, a_kind_of(String))
expect(response).to have_http_status(:ok)
expect(freshdesk_import.reload).to be_pending
expect(freshdesk_import.active_import_run_id).to be_present
end
end
describe 'POST /api/v1/accounts/:account_id/data_imports/:id/retry' do
@@ -264,6 +373,20 @@ RSpec.describe 'Data Imports API', type: :request do
expect(response).to have_http_status(:unprocessable_entity)
expect(response.parsed_body['message']).to eq('Another Intercom import is already in progress.')
end
it 'resumes a stalled Freshdesk import with the Freshdesk job' do
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :processing, started_at: 2.hours.ago)
freshdesk_import.update!(updated_at: 16.minutes.ago)
expect do
post retry_api_v1_account_data_import_url(account_id: account.id, id: freshdesk_import.id),
headers: admin.create_new_auth_token,
as: :json
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).with(freshdesk_import, a_kind_of(String))
expect(response).to have_http_status(:ok)
expect(freshdesk_import.reload).to be_pending
end
end
describe 'POST /api/v1/accounts/:account_id/data_imports/:id/abandon' do

View File

@@ -0,0 +1,90 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::Client do
describe '.normalize_domain' do
it 'accepts a Freshdesk subdomain, hostname, or URL', :aggregate_failures do
expect(described_class.normalize_domain('acme')).to eq('acme.freshdesk.com')
expect(described_class.normalize_domain('ACME.FRESHDESK.COM')).to eq('acme.freshdesk.com')
expect(described_class.normalize_domain('https://acme.freshdesk.com/support/home')).to eq('acme.freshdesk.com')
end
it 'rejects hosts outside Freshdesk' do
expect do
described_class.normalize_domain('acme.example.com')
end.to raise_error(ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.')
end
end
describe '#list_tickets' do
it 'uses API key basic authentication, requests full history, and follows the next page link', :aggregate_failures do
response = instance_double(
HTTParty::Response,
success?: true,
code: 200,
parsed_response: [{ 'id' => 2001 }],
headers: {
'link' => '<https://acme.freshdesk.com/api/v2/tickets?page=3&per_page=100>; rel="next"'
}
)
allow(HTTParty).to receive(:get).and_return(response)
page = described_class.new(domain: 'acme', api_key: 'secret').list_tickets(page: 2)
expect(HTTParty).to have_received(:get).with(
'https://acme.freshdesk.com/api/v2/tickets',
query: {
page: 2,
per_page: 100,
updated_since: DataImports::Freshdesk::Client::EARLIEST_TICKET_TIMESTAMP
},
basic_auth: { username: 'secret', password: 'X' },
headers: { 'Accept' => 'application/json', 'Content-Type' => 'application/json' },
timeout: 30
)
expect(page.data).to eq([{ 'id' => 2001 }])
expect(page.next_page).to eq(3)
end
it 'accepts only forward pages from relative or same-domain links', :aggregate_failures do
client = described_class.new(domain: 'acme', api_key: 'secret')
expect(client.send(:next_page, '</api/v2/tickets?page=3>; rel="next"', current_page: 2)).to eq(3)
expect(client.send(:next_page, '<https://acme.freshdesk.com/api/v2/tickets?page=4>; rel="next"', current_page: 3)).to eq(4)
expect(client.send(:next_page, '<https://other.freshdesk.com/api/v2/tickets?page=4>; rel="next"', current_page: 3)).to be_nil
expect(client.send(:next_page, '</api/v2/tickets?page=abc>; rel="next"', current_page: 2)).to be_nil
expect(client.send(:next_page, '</api/v2/tickets?page=0>; rel="next"', current_page: 2)).to be_nil
expect(client.send(:next_page, '</api/v2/tickets?page=2>; rel="next"', current_page: 2)).to be_nil
expect(client.send(:next_page, '</api/v2/tickets?page=1>; rel="next"', current_page: 2)).to be_nil
expect(client.send(:next_page, '<%%%>; rel="next"', current_page: 2)).to be_nil
end
end
describe 'errors' do
it 'exposes Freshdesk retry timing on rate limits', :aggregate_failures do
response = instance_double(
HTTParty::Response,
success?: false,
code: 429,
parsed_response: { 'description' => 'Rate limit exceeded' },
headers: { 'retry-after' => '42' }
)
allow(HTTParty).to receive(:get).and_return(response)
expect { described_class.new(domain: 'acme', api_key: 'secret').list_contacts }.to raise_error do |error|
expect(error.class.name).to eq('DataImports::Freshdesk::Client::RateLimitError')
expect(error).to have_attributes(status: 429, retry_after: '42')
expect(error.message).to eq('Rate limit exceeded')
end
end
it 'wraps transport failures in a retryable client error', :aggregate_failures do
allow(HTTParty).to receive(:get).and_raise(SocketError, 'getaddrinfo failed')
expect { described_class.new(domain: 'acme', api_key: 'secret').list_contacts }.to raise_error do |error|
expect(error.class.name).to eq('DataImports::Freshdesk::Client::Error')
expect(error.message).to eq('Freshdesk API request failed before receiving a response: getaddrinfo failed')
expect(error.body).to include(transport_error_class: 'SocketError')
end
end
end
end

View File

@@ -0,0 +1,39 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::CredentialsValidator do
let(:client) { instance_double(DataImports::Freshdesk::Client) }
before do
allow(DataImports::Freshdesk::Client).to receive(:new).with(domain: 'acme', api_key: 'secret').and_return(client)
allow(client).to receive(:list_contacts)
allow(client).to receive(:list_tickets)
end
it 'validates only the selected Freshdesk resources and leaves totals undiscovered', :aggregate_failures do
totals = described_class.new(domain: 'acme', api_key: ' secret ', import_types: %w[conversations]).perform
expect(totals).to be_empty
expect(client).to have_received(:list_tickets).with(per_page: 1)
expect(client).not_to have_received(:list_contacts)
end
it 'validates contacts and tickets when both are selected', :aggregate_failures do
described_class.new(domain: 'acme', api_key: 'secret', import_types: %w[contacts conversations]).perform
expect(client).to have_received(:list_contacts).with(per_page: 1)
expect(client).to have_received(:list_tickets).with(per_page: 1)
end
it 'rejects missing or unsupported input before calling Freshdesk', :aggregate_failures do
expect do
described_class.new(domain: '', api_key: 'secret', import_types: %w[contacts]).perform
end.to raise_error(ArgumentError, 'Freshdesk domain is required.')
expect do
described_class.new(domain: 'acme', api_key: '', import_types: %w[contacts]).perform
end.to raise_error(ArgumentError, 'Freshdesk API key is required.')
expect do
described_class.new(domain: 'acme', api_key: 'secret', import_types: %w[companies]).perform
end.to raise_error(ArgumentError, 'Unsupported import types: companies')
expect(DataImports::Freshdesk::Client).not_to have_received(:new)
end
end

View File

@@ -0,0 +1,302 @@
require 'rails_helper'
require 'timeout'
RSpec.describe DataImports::Freshdesk::Importer do
let(:account) { create(:account) }
let(:data_import) { create(:data_import, :freshdesk, account: account) }
let(:client) { instance_double(DataImports::Freshdesk::Client) }
let(:contact_payload) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/contact.json').read)
end
let(:ticket_fixture) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
end
before do
account.enable_features!('data_import')
allow(DataImports::Freshdesk::Client).to receive(:new).with(
domain: 'acme.freshdesk.com',
api_key: 'freshdesk-api-key'
).and_return(client)
allow(client).to receive(:list_contacts).with(page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: [contact_payload], next_page: nil)
)
allow(client).to receive(:list_tickets).with(page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: [ticket_fixture.fetch('ticket')], next_page: nil)
)
allow(client).to receive(:retrieve_ticket).with('2001').and_return(ticket_fixture.fetch('ticket'))
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: ticket_fixture.fetch('conversations'), next_page: nil)
)
end
it 'imports Freshdesk contacts, tickets, replies, and private notes through the shared importer', :aggregate_failures do
described_class.new(data_import: data_import).perform
contact = account.contacts.find_by!(email: 'customer@example.com')
expect(contact).to have_attributes(
name: 'Customer Example',
phone_number: '+15551234567',
identifier: 'customer-1001'
)
expect(contact.custom_attributes).to include(
'freshdesk_contact_id' => '1001',
'freshdesk_unique_external_id' => 'customer-1001'
)
inbox = account.inboxes.find_by!(name: 'Freshdesk Import - Email')
expect(inbox.channel.additional_attributes).to include(
'source_provider' => 'freshdesk',
'source_bucket' => 'email',
'import_placeholder' => true
)
conversation = account.conversations.find_by!(identifier: 'freshdesk:2001')
expect(conversation).to have_attributes(
status: 'resolved',
inbox_id: inbox.id,
contact_id: contact.id
)
expect(conversation.custom_attributes).to include(
'freshdesk_ticket_id' => '2001',
'freshdesk_status' => 4,
'freshdesk_priority' => 2
)
messages = conversation.messages.order(:created_at, :id)
expect(messages.pluck(:content)).to eq(
[
"Unable to finish checkout\n\nThe checkout button is not responding.",
'Could you try this in a private window?',
"Payments team is investigating the browser logs.\n\n[Freshdesk attachment skipped: 1]",
'The issue still happens in a private window.'
]
)
expect(messages.map(&:message_type)).to eq(%w[incoming outgoing outgoing incoming])
expect(messages.pluck(:private)).to eq([false, false, true, false])
expect(messages.third.additional_attributes.dig('source', 'conversation_source')).to eq(2)
expect(messages.third.additional_attributes.dig('source', 'attachments', 0, 'name')).to eq('browser-log.txt')
expect(data_import.reload).to be_completed
expect(data_import.stats).to include(
'contacts' => include('imported' => 1, 'skipped' => 0),
'conversations' => include('imported' => 1, 'skipped' => 0),
'messages' => include('imported' => 4, 'skipped' => 0, 'total' => 4),
'errors' => { 'count' => 0 }
)
expect(data_import.processed_records).to eq(6)
expect(data_import.mappings.count).to eq(6)
end
it 'preserves the contact that authored an incoming reply' do
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
conversations.first.merge!('user_id' => 1002, 'from_email' => 'cc@example.com')
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: conversations, next_page: nil)
)
data_import.update!(import_types: ['conversations'])
described_class.new(data_import: data_import).perform
reply = account.conversations.find_by!(identifier: 'freshdesk:2001').messages.find_by!(content: 'The issue still happens in a private window.')
expect(reply.sender).to have_attributes(email: 'cc@example.com')
expect(reply.sender_id).not_to eq(reply.conversation.contact_id)
end
it 'preserves newer progress when a delayed worker persists a stale snapshot', :aggregate_failures do
data_import.update!(
status: :processing,
cursor: { 'contacts' => { 'starting_after' => 'cursor-1', 'completed' => false } },
stats: {
'contacts' => { 'total' => 4, 'imported' => 1, 'skipped' => 0 },
'conversations' => { 'imported' => 0, 'skipped' => 0 },
'messages' => { 'imported' => 0, 'skipped' => 0 },
'errors' => { 'count' => 0 }
},
processed_records: 1
)
data_import.items.create!(
source_provider: 'freshdesk', source_object_type: 'contact', source_object_id: '1001', status: :imported
)
delayed_importer = described_class.new(data_import: DataImport.find(data_import.id))
data_import.items.create!(
source_provider: 'freshdesk', source_object_type: 'contact', source_object_id: '1002', status: :imported
)
data_import.reload.update!(
cursor: { 'contacts' => { 'starting_after' => 'cursor-2', 'completed' => false } },
stats: data_import.stats.deep_merge('contacts' => { 'imported' => 2 }),
processed_records: 2
)
delayed_importer.send(:persist_stats)
expect(delayed_importer.send(:update_cursor, 'contacts', 'cursor-1')).to eq('cursor-2')
expect(data_import.reload.cursor.dig('contacts', 'starting_after')).to eq('cursor-2')
expect(data_import.stats.dig('contacts', 'imported')).to eq(2)
expect(data_import.processed_records).to eq(2)
end
it 'does not let an old import run overwrite its replacement cursor', :aggregate_failures do
data_import.update!(
status: :processing,
source_metadata: data_import.source_metadata.merge('active_import_run_id' => 'run-1'),
cursor: { 'conversations' => { 'starting_after' => { 'page' => 1, 'offset' => 0 }, 'completed' => false } }
)
delayed_importer = described_class.new(data_import: DataImport.find(data_import.id), run_id: 'run-1')
replacement_cursor = { 'page' => 1, 'offset' => 3 }
data_import.reload.update!(
source_metadata: data_import.source_metadata.merge('active_import_run_id' => 'run-2'),
cursor: { 'conversations' => { 'starting_after' => replacement_cursor, 'completed' => false } }
)
result = delayed_importer.send(:update_cursor, 'conversations', { 'page' => 1, 'offset' => 1 })
expect(result).to eq(replacement_cursor)
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to eq(replacement_cursor)
expect(delayed_importer.send(:import_stopped?)).to be(true)
end
it 'serializes overlapping workers importing the same conversation', :aggregate_failures do
started_requests = Queue.new
release_requests = Queue.new
allow(client).to receive(:retrieve_ticket).with('2001') do
started_requests << true
release_requests.pop
ticket_fixture.fetch('ticket')
end
importers = Array.new(2) { described_class.new(data_import: DataImport.find(data_import.id)) }
errors = Concurrent::Array.new
conversation_summary = { 'id' => '2001', 'source' => { 'type' => 'email' } }
threads = importers.map do |importer|
Thread.new do
importer.send(:import_conversation_from_summary, conversation_summary)
rescue StandardError => e
errors << e
end
end
begin
2.times { Timeout.timeout(5) { started_requests.pop } }
ensure
2.times { release_requests << true }
threads.each { |thread| thread.join(10) }
end
expect(threads).to all(satisfy { |thread| !thread.alive? })
expect(errors).to be_empty
expect(account.conversations.where(identifier: 'freshdesk:2001').count).to eq(1)
expect(account.messages.count).to eq(4)
expect(data_import.mappings.count).to eq(6)
expect(data_import.items.find_by!(source_object_type: 'conversation', source_object_id: '2001')).to have_attributes(
status: 'imported', attempt_count: 2
)
end
it 'recognizes a structured cursor after JSON persistence' do
importer = described_class.new(data_import: data_import)
importer.send(:update_cursor, 'conversations', { page: 1, offset: 2 })
data_import.reload
importer.send(:update_cursor, 'conversations', nil)
expect(data_import.reload.cursor['conversations']).to include('starting_after' => nil, 'completed' => true)
end
it 'resumes a rate-limited ticket chunk from the last persisted checkpoint', :aggregate_failures do
tickets = Array.new(3) { |index| { 'id' => index + 2001, 'source' => 1 } }
allow(client).to receive(:list_tickets).with(page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
)
importer = described_class.new(data_import: data_import)
processed_ticket_ids = []
rate_limited = true
allow(importer).to receive(:import_conversation_from_summary) do |summary|
processed_ticket_ids << summary['id']
if summary['id'] == '2003' && rate_limited
rate_limited = false
raise DataImports::Freshdesk::Client::RateLimitError.new('Rate limited', retry_after: '42')
end
end
expect do
importer.import_conversations_page(starting_after: { 'page' => 1, 'offset' => 0 })
end.to raise_error(DataImports::Freshdesk::Client::RateLimitError)
expect(processed_ticket_ids).to eq(%w[2001 2002 2003])
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to include('page' => 1, 'offset' => 2)
processed_ticket_ids.clear
result = importer.import_conversations_page(starting_after: { 'page' => 1, 'offset' => 0 })
expect(processed_ticket_ids).to eq(%w[2003])
expect(result).to be_done
expect(data_import.reload.cursor['conversations']).to include('starting_after' => nil, 'completed' => true)
expect(client).to have_received(:list_tickets).once
end
it 'stops a delayed worker when a newer per-ticket checkpoint is observed', :aggregate_failures do
current_cursor = { 'page' => 1, 'offset' => 0, 'tickets' => [], 'next_page' => 2 }
newer_cursor = current_cursor.merge('offset' => 3)
summaries = Array.new(3) { |index| { 'id' => (index + 2001).to_s } }
response = {
'data' => summaries,
'pages' => {
'current' => { 'starting_after' => current_cursor },
'next' => { 'starting_after' => { 'page' => 2, 'offset' => 0 } },
'checkpoints' => Array.new(3) { |index| current_cursor.merge('offset' => index + 1) }
}
}
importer = described_class.new(data_import: DataImport.find(data_import.id))
allow(importer).to receive(:conversations_page).and_return(response)
cursor_advanced = false
allow(importer).to receive(:persist_stats).and_wrap_original do |method, *args|
method.call(*args)
next if cursor_advanced
cursor_advanced = true
concurrent_import = DataImport.find(data_import.id)
concurrent_import.update!(
cursor: concurrent_import.cursor.to_h.deep_merge(
'conversations' => { 'starting_after' => newer_cursor, 'completed' => false }
)
)
end
result = importer.import_conversations_page(starting_after: current_cursor)
expect(account.conversations.where(identifier: 'freshdesk:2001').count).to eq(1)
expect(client).not_to have_received(:retrieve_ticket).with('2002')
expect(result.next_cursor).to eq(newer_cursor)
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to eq(newer_cursor)
end
it 'fails with an actionable error instead of completing at the REST ticket limit', :aggregate_failures do
tickets = Array.new(100) { |index| { 'id' => index + 1, 'source' => 1 } }
allow(client).to receive(:list_tickets).with(page: 300, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
)
importer = described_class.new(data_import: data_import)
processed_ticket_ids = []
allow(importer).to receive(:import_conversation_from_summary) { |summary| processed_ticket_ids << summary['id'] }
limit_error = nil
expect { importer.import_conversations_page(starting_after: { 'page' => 300, 'offset' => 90 }) }.to raise_error do |error|
expect(error.class.name).to eq('CustomExceptions::DataImport::FreshdeskTicketLimitError')
limit_error = error
end
importer.fail!(limit_error)
expect(processed_ticket_ids).to eq((91..100).map(&:to_s))
expect(data_import.reload).to be_failed
expect(data_import.cursor.dig('conversations', 'starting_after')).to include('page' => 300, 'offset' => 99)
expect(data_import.import_errors.last).to have_attributes(
error_code: 'CustomExceptions::DataImport::FreshdeskTicketLimitError',
message: limit_error.message
)
expect(data_import.import_errors.last.details).to include(
'kind' => 'run_error',
'source_provider' => 'freshdesk',
'error_class' => 'CustomExceptions::DataImport::FreshdeskTicketLimitError'
)
end
end

View File

@@ -0,0 +1,106 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::Normalizer do
let(:contact_payload) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/contact.json').read)
end
let(:ticket_fixture) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
end
let(:web_chat_fixture) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/web_chat_ticket_with_conversations.json').read)
end
it 'normalizes Freshdesk contacts into the shared importer contract', :aggregate_failures do
contact = described_class.new.contact(contact_payload)
expect(contact).to include(
'id' => '1001',
'name' => 'Customer Example',
'email' => 'customer@example.com',
'phone' => '+15551234567',
'external_id' => 'customer-1001',
'other_emails' => ['billing@example.com'],
'custom_fields' => { 'support_plan' => 'growth' },
'tags' => ['priority']
)
expect(contact['created_at']).to eq(Time.zone.parse('2024-01-01T09:00:00Z').to_i)
expect(contact['updated_at']).to eq(Time.zone.parse('2024-01-04T12:30:00Z').to_i)
end
it 'normalizes ticket descriptions, replies, and private notes in source order', :aggregate_failures do
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), ticket_fixture.fetch('conversations'))
parts = ticket.dig('conversation_parts', 'conversation_parts')
expect(ticket).to include(
'id' => '2001',
'status' => 4,
'priority' => 2,
'requester_id' => 1001
)
expect(ticket.dig('source', 'type')).to eq('email')
expect(ticket.dig('source', 'author', 'type')).to eq('contact')
expect(ticket.dig('contacts', 'contacts', 0, 'email')).to eq('customer@example.com')
expect(parts.pluck('id')).to eq(%w[3001 3002 3003])
expect(parts.pluck('part_type')).to eq(%w[comment note comment])
expect(parts.map { |part| part.dig('author', 'type') }).to eq(%w[admin admin contact])
expect(parts.second).to include('private' => true)
expect(parts.second['attachments'].first).to include('name' => 'browser-log.txt')
end
it 'preserves provider order for messages with equal timestamps' do
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
conversations[0]['created_at'] = conversations[1]['created_at']
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), conversations)
expect(ticket.dig('conversation_parts', 'conversation_parts').pluck('id')).to eq(%w[3003 3001 3002])
end
it 'includes incoming reply authors as conversation contacts' do
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
conversations.first.merge!('user_id' => 1002, 'from_email' => 'cc@example.com')
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), conversations)
expect(ticket.dig('contacts', 'contacts')).to include(
include('id' => '1001', 'email' => 'customer@example.com'),
include('id' => '1002', 'email' => 'cc@example.com')
)
expect(ticket.dig('conversation_parts', 'conversation_parts').last['author']).to include(
'id' => '1002', 'email' => 'cc@example.com'
)
end
it 'marks outbound email ticket descriptions as outgoing source messages' do
outbound_ticket = ticket_fixture.fetch('ticket').merge('source' => 10)
ticket = described_class.new.ticket(outbound_ticket, [])
expect(ticket.dig('source', 'type')).to eq('outbound_email')
expect(ticket.dig('source', 'author', 'type')).to eq('admin')
end
it 'uses Web Chat conversation events as the complete message history', :aggregate_failures do
ticket = described_class.new.ticket(web_chat_fixture.fetch('ticket'), web_chat_fixture.fetch('conversations'))
parts = ticket.dig('conversation_parts', 'conversation_parts')
expect(ticket).to include(
'id' => '2101',
'subject' => 'FD-IMPORT-WEBCHAT-002 — Brew temperature drops',
'status' => 2,
'priority' => 1
)
expect(ticket.dig('source', 'type')).to eq('web_chat')
expect(ticket.fetch('source')).not_to include('subject', 'body')
expect(DataImports::Freshdesk::MessageBatchBuilder.source_message_importable?(ticket.fetch('source'))).to be(false)
expect(parts.pluck('id')).to eq(%w[3101 3102 3103 3104 3105 3106])
expect(parts.map { |part| part.dig('author', 'type') }).to eq(%w[admin contact contact admin contact admin])
expect(parts.count { |part| part['body'].include?('FD-IMPORT-WEBCHAT-002') }).to eq(1)
expect(parts.last['attachments'].first).to include(
'name' => 'freshdesk-validation-diagnostic.txt',
'content_type' => 'text/plain',
'size' => 188
)
end
end

View File

@@ -0,0 +1,45 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::SourceBucket do
describe '.source_type' do
it 'maps documented and observed Freshdesk source identifiers', :aggregate_failures do
expect(described_class.source_type(1)).to eq('email')
expect(described_class.source_type(7)).to eq('chat')
expect(described_class.source_type(8)).to eq('mobihelp')
expect(described_class.source_type(11)).to eq('ecommerce')
expect(described_class.source_type(12)).to eq('bot')
expect(described_class.source_type(13)).to eq('whatsapp')
expect(described_class.source_type(14)).to eq('chat_internal_task')
expect(described_class.source_type(15)).to eq('web_chat')
expect(described_class.source_type(16)).to eq('web_form')
expect(described_class.source_type(17)).to eq('instagram_message')
expect(described_class.source_type(18)).to eq('instagram_comment')
expect(described_class.source_type(19)).to eq('facebook_message')
expect(described_class.source_type(20)).to eq('facebook_comment')
expect(described_class.source_type(21)).to eq('mobile_chat_sdk')
expect(described_class.source_type(22)).to eq('sms')
end
it 'uses unknown for an unsupported source identifier' do
expect(described_class.source_type(999)).to eq('unknown')
end
end
describe '.for' do
it 'groups equivalent sources and preserves distinct modern channels', :aggregate_failures do
expect(described_class.for('email')).to eq({ key: 'email', name: 'Email' })
expect(described_class.for('outbound_email')).to eq({ key: 'email', name: 'Email' })
expect(described_class.for('feedback_widget')).to eq({ key: 'portal', name: 'Portal' })
expect(described_class.for('whatsapp')).to eq({ key: 'whatsapp', name: 'WhatsApp' })
expect(described_class.for('chat_internal_task')).to eq({ key: 'internal_task', name: 'Internal task' })
expect(described_class.for('web_chat')).to eq({ key: 'chat', name: 'Chat' })
expect(described_class.for('web_form')).to eq({ key: 'portal', name: 'Portal' })
expect(described_class.for('instagram_message')).to eq({ key: 'instagram', name: 'Instagram' })
expect(described_class.for('instagram_comment')).to eq({ key: 'instagram', name: 'Instagram' })
expect(described_class.for('facebook_message')).to eq({ key: 'facebook', name: 'Facebook' })
expect(described_class.for('facebook_comment')).to eq({ key: 'facebook', name: 'Facebook' })
expect(described_class.for('mobile_chat_sdk')).to eq({ key: 'mobile', name: 'Mobile' })
expect(described_class.for('sms')).to eq({ key: 'sms', name: 'SMS' })
end
end
end

View File

@@ -0,0 +1,128 @@
require 'rails_helper'
RSpec.describe DataImports::Freshdesk::Source do
let(:client) { instance_double(DataImports::Freshdesk::Client) }
let(:ticket_fixture) do
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
end
let(:source) do
described_class.new(
access_token: 'freshdesk-api-key',
source_metadata: { domain: 'acme.freshdesk.com' }
)
end
before do
allow(DataImports::Freshdesk::Client).to receive(:new).with(
domain: 'acme.freshdesk.com',
api_key: 'freshdesk-api-key'
).and_return(client)
end
describe '.source_metadata' do
it 'stores only a normalized Freshdesk domain' do
expect(described_class.source_metadata(domain: 'https://ACME.freshdesk.com/support/home')).to eq(
domain: 'acme.freshdesk.com'
)
end
end
describe '#list_contacts' do
it 'normalizes records and translates numeric pages into shared cursors' do
allow(client).to receive(:list_contacts).with(page: 2, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(
data: [{ 'id' => 1001, 'email' => 'customer@example.com' }],
next_page: 3
)
)
response = source.list_contacts(starting_after: 2, per_page: 100)
expect(response.dig('data', 0)).to include('id' => '1001', 'email' => 'customer@example.com')
expect(response.dig('pages', 'next', 'starting_after')).to eq(3)
end
end
describe '#list_conversations' do
it 'returns a resumable ticket chunk with a cursor after every ticket', :aggregate_failures do
tickets = Array.new(25) { |index| { 'id' => index + 1, 'source' => 1 } }
allow(client).to receive(:list_tickets).with(page: 2, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: 3)
)
response = source.list_conversations(
starting_after: { 'page' => 2, 'offset' => 10 },
per_page: 100
)
expect(response['data'].pluck('id')).to eq((11..20).map(&:to_s))
expect(response.dig('pages', 'checkpoints').first).to include('page' => 2, 'offset' => 11, 'next_page' => 3)
expect(response.dig('pages', 'checkpoints').last).to include('page' => 2, 'offset' => 20, 'next_page' => 3)
expect(response.dig('pages', 'next', 'starting_after')).to include('page' => 2, 'offset' => 20, 'next_page' => 3)
end
it 'reuses the cached ticket page before advancing to the next Freshdesk page', :aggregate_failures do
tickets = Array.new(25) { |index| { 'id' => index + 1, 'source' => 1 } }
expect(client).to receive(:list_tickets).with(page: 2, per_page: 100).once.and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: 3)
)
first_chunk = source.list_conversations(
starting_after: { 'page' => 2, 'offset' => 10 },
per_page: 100
)
response = source.list_conversations(starting_after: first_chunk.dig('pages', 'next', 'starting_after'), per_page: 100)
expect(response['data'].pluck('id')).to eq((21..25).map(&:to_s))
expect(response.dig('pages', 'checkpoints').last).to eq('page' => 3, 'offset' => 0)
expect(response.dig('pages', 'next', 'starting_after')).to eq('page' => 3, 'offset' => 0)
end
it 'reports the REST ticket limit only after the final chunk of a full page 300', :aggregate_failures do
tickets = Array.new(100) { |index| { 'id' => index + 1, 'source' => 1 } }
expect(client).to receive(:list_tickets).with(page: 300, per_page: 100).once.and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
)
first_chunk = source.list_conversations(starting_after: { 'page' => 300, 'offset' => 0 }, per_page: 100)
final_cursor = first_chunk.dig('pages', 'current', 'starting_after').merge('offset' => 90)
final_chunk = source.list_conversations(starting_after: final_cursor, per_page: 100)
expect(first_chunk.dig('pages', 'limit_reached')).to be(false)
expect(final_chunk['data'].pluck('id')).to eq((91..100).map(&:to_s))
expect(final_chunk.dig('pages', 'next')).to be_nil
expect(final_chunk.dig('pages', 'limit_reached')).to be(true)
end
it 'completes a partial page 300 without reporting the ticket limit' do
tickets = Array.new(99) { |index| { 'id' => index + 1, 'source' => 1 } }
allow(client).to receive(:list_tickets).with(page: 300, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
)
response = source.list_conversations(starting_after: { 'page' => 300, 'offset' => 90 }, per_page: 100)
expect(response.dig('pages', 'next')).to be_nil
expect(response.dig('pages', 'limit_reached')).to be(false)
end
end
describe '#retrieve_conversation' do
it 'retrieves every conversation page and normalizes the ticket', :aggregate_failures do
conversations = ticket_fixture.fetch('conversations')
allow(client).to receive(:retrieve_ticket).with('2001').and_return(ticket_fixture.fetch('ticket'))
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: conversations.first(2), next_page: 2)
)
allow(client).to receive(:list_conversations).with('2001', page: 2, per_page: 100).and_return(
DataImports::Freshdesk::Client::Page.new(data: conversations.last(1), next_page: nil)
)
ticket = source.retrieve_conversation('2001')
expect(ticket['id']).to eq('2001')
expect(ticket.dig('conversation_parts', 'total_count')).to eq(3)
expect(ticket.dig('conversation_parts', 'conversation_parts').pluck('id')).to eq(%w[3001 3002 3003])
end
end
end

View File

@@ -29,5 +29,27 @@ RSpec.describe DataImports::Intercom::PlaceholderInboxBuilder do
expect(second_inbox).to eq(first_inbox)
expect(Inbox.where(account: account, channel_type: 'Channel::Api').count).to eq(1)
end
it 'creates editable default working hours without running inbox creation callbacks' do
inbox = described_class.new(account: account).inbox_for('email')
expect(inbox.working_hours.count).to eq(7)
inbox.update_working_hours(
[
{
'day_of_week' => 1,
'open_hour' => 10,
'open_minutes' => 0,
'close_hour' => 18,
'close_minutes' => 0,
'closed_all_day' => false,
'open_all_day' => false
}
]
)
expect(inbox.working_hours.find_by(day_of_week: 1)).to have_attributes(open_hour: 10, close_hour: 18)
end
end
end

View File

@@ -0,0 +1,26 @@
require 'rails_helper'
RSpec.describe DataImports::Source do
describe '.source_class' do
it 'resolves each supported integration adapter', :aggregate_failures do
expect(described_class.source_class('intercom')).to eq(DataImports::Intercom::Source)
expect(described_class.source_class('freshdesk')).to eq(DataImports::Freshdesk::Source)
expect(described_class.supported?('intercom')).to be(true)
expect(described_class.supported?('freshdesk')).to be(true)
end
it 'rejects unsupported providers' do
expect { described_class.source_class('zendesk') }.to raise_error(ArgumentError, 'Unsupported import source.')
end
end
describe '.for' do
it 'builds the configured adapter from persisted credentials' do
data_import = build(:data_import, :freshdesk)
source = described_class.for(data_import)
expect(source).to have_attributes(provider: 'freshdesk', display_name: 'Freshdesk')
end
end
end