feat(data-imports): add Freshdesk migration (1/3) (#15261)
## Description Adds Freshdesk as an integration import source so administrators can validate a Freshdesk domain and API key, then import contacts, tickets, public replies, customer replies, and private notes while tracking progress from Data Imports. The integration has now been validated against a live Freshdesk trial tenant with contacts, Web Chat and phone tickets, public replies, customer replies, a private note, pagination, requester expansion, and attachment metadata. That validation found and fixed the current Web Chat source mapping and prevented the ticket description from duplicating the initial Web Chat message. Related: #15116 ## Closes Closes [CW-7639](https://linear.app/chatwoot/issue/CW-7639/freshdesk-freshworks-migration) ## Type of change - [x] New feature (non-breaking change which adds functionality) ## What changed - Added a shared source adapter, importer, job, retry, restart, creation, and placeholder inbox contract used by Intercom and Freshdesk. - Added Freshdesk API authentication, contact and ticket pagination, requester expansion, conversation retrieval, normalization, channel grouping, and error handling. - Added current Freshdesk source identifiers through SMS, including Web Chat source `15`, and grouped equivalent sources into placeholder inboxes. - Used Web Chat conversation events as the complete message history so the generated ticket description does not duplicate the initial customer message. - Preserved Freshdesk ticket subjects in source metadata and added a sanitized live-derived Web Chat fixture with structured bodies and attachment metadata. - Added Freshdesk selection, domain and API key validation, and provider-neutral import status handling in the Data Imports UI. ## How to test 1. Enable the data_import feature for an account and open Settings > Data > New import. 2. Select Freshdesk and enter a Freshdesk domain and API key. 3. Select contacts and/or conversations, validate the credentials, and start the import. 4. Confirm progress is displayed and imported tickets appear as resolved conversations in Freshdesk placeholder inboxes with public replies and private notes preserved. 5. Verify Web Chat tickets appear in the Chat placeholder inbox and the initial customer message is imported once. 6. Verify an abandoned import can be restarted and a stalled import can be retried. ## Current scope - **Product decision:** Attachment binaries are intentionally not imported in the current migration scope. Attachment metadata is preserved and messages include a skipped-attachment marker. - Adaptive Retry-After scheduling and handling the 30,000-ticket listing ceiling are covered by stacked follow-up PRs. ## Checklist: - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented on my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] Any dependent changes have been merged and published in downstream modules
This commit is contained in:
@@ -19,14 +19,12 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
|
||||
end
|
||||
|
||||
def validate_source
|
||||
totals = validate_intercom_source
|
||||
totals = source_class.credentials_validator(source_params: permitted_params.to_h, import_types: import_types).perform
|
||||
render json: { valid: true, totals: totals }
|
||||
rescue DataImports::Intercom::Client::AuthenticationError
|
||||
render_source_validation_error('We could not validate this Intercom access key. Check the key and its permissions.')
|
||||
rescue DataImports::Intercom::Client::Error
|
||||
render_source_validation_error('Intercom could not be reached. Please try again.')
|
||||
rescue ArgumentError => e
|
||||
render_source_validation_error(e.message)
|
||||
rescue StandardError => e
|
||||
render_source_client_error(e)
|
||||
end
|
||||
|
||||
def create
|
||||
@@ -36,44 +34,42 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
|
||||
return
|
||||
end
|
||||
|
||||
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id)
|
||||
enqueue_import(@data_import)
|
||||
render_show
|
||||
rescue DataImports::Intercom::Client::AuthenticationError
|
||||
render_source_validation_error('We could not validate this Intercom access key. Check the key and its permissions.')
|
||||
rescue DataImports::Intercom::Client::Error
|
||||
render_source_validation_error('Intercom could not be reached. Please try again.')
|
||||
rescue ArgumentError => e
|
||||
render_source_validation_error(e.message)
|
||||
rescue StandardError => e
|
||||
render_source_client_error(e)
|
||||
end
|
||||
|
||||
def start
|
||||
restart_service = DataImports::Intercom::RestartService.new(account: Current.account, data_import: @data_import)
|
||||
restart_service = DataImports::RestartService.new(account: Current.account, data_import: @data_import)
|
||||
restart_result = restart_service.perform
|
||||
@data_import = restart_service.data_import
|
||||
if restart_result == :access_token_missing
|
||||
render json: { message: 'The Intercom access key for this import is unavailable.' }, status: :unprocessable_entity
|
||||
render json: { message: "The #{source_name} #{credential_name} for this import is unavailable." }, status: :unprocessable_entity
|
||||
return
|
||||
end
|
||||
|
||||
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id) if restart_result == :enqueue
|
||||
enqueue_import(@data_import) if restart_result == :enqueue
|
||||
render_show
|
||||
end
|
||||
|
||||
def retry_import
|
||||
retry_service = DataImports::Intercom::RetryService.new(account: Current.account, data_import: @data_import)
|
||||
retry_service = DataImports::RetryService.new(account: Current.account, data_import: @data_import)
|
||||
retry_result = retry_service.perform
|
||||
@data_import = retry_service.data_import
|
||||
|
||||
case retry_result
|
||||
when :enqueue
|
||||
DataImports::Intercom::ImportJob.perform_later(@data_import, @data_import.active_intercom_import_run_id)
|
||||
enqueue_import(@data_import)
|
||||
render_show
|
||||
when :not_stalled
|
||||
render json: { message: 'This Intercom import is no longer stalled.' }, status: :unprocessable_entity
|
||||
render json: { message: "This #{source_name} import is no longer stalled." }, status: :unprocessable_entity
|
||||
when :active_import_exists
|
||||
render json: { message: 'Another Intercom import is already in progress.' }, status: :unprocessable_entity
|
||||
render json: { message: "Another #{source_name} import is already in progress." }, status: :unprocessable_entity
|
||||
when :access_token_missing
|
||||
render json: { message: 'The Intercom access key for this import is unavailable.' }, status: :unprocessable_entity
|
||||
render json: { message: "The #{source_name} #{credential_name} for this import is unavailable." }, status: :unprocessable_entity
|
||||
end
|
||||
end
|
||||
|
||||
@@ -113,11 +109,11 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
|
||||
end
|
||||
|
||||
def permitted_params
|
||||
params.permit(:name, :source_provider, :access_token, import_types: [])
|
||||
params.permit(:name, :source_provider, :access_token, :domain, import_types: [])
|
||||
end
|
||||
|
||||
def creation_service
|
||||
DataImports::Intercom::CreationService.new(
|
||||
DataImports::CreationService.new(
|
||||
account: Current.account,
|
||||
initiated_by: Current.user,
|
||||
source_params: permitted_params.to_h
|
||||
@@ -125,24 +121,50 @@ class Api::V1::Accounts::DataImportsController < Api::V1::Accounts::BaseControll
|
||||
end
|
||||
|
||||
def import_types
|
||||
return DataImports::Intercom::Importer::DEFAULT_IMPORT_TYPES unless permitted_params.key?(:import_types)
|
||||
return DataImports::Importer::DEFAULT_IMPORT_TYPES unless permitted_params.key?(:import_types)
|
||||
|
||||
Array(permitted_params[:import_types]).compact_blank
|
||||
end
|
||||
|
||||
def validate_intercom_source
|
||||
raise ArgumentError, 'Unsupported import source.' unless permitted_params[:source_provider] == 'intercom'
|
||||
def source_class
|
||||
provider = @data_import&.source_provider || permitted_params[:source_provider]
|
||||
DataImports::Source.source_class(provider)
|
||||
end
|
||||
|
||||
DataImports::Intercom::CredentialsValidator.new(
|
||||
access_token: permitted_params[:access_token],
|
||||
import_types: import_types
|
||||
).perform
|
||||
def source_name
|
||||
return 'Integration' unless DataImports::Source.supported?(@data_import&.source_provider || permitted_params[:source_provider])
|
||||
|
||||
source_class::DISPLAY_NAME
|
||||
end
|
||||
|
||||
def credential_name
|
||||
return 'credential' unless DataImports::Source.supported?(@data_import&.source_provider || permitted_params[:source_provider])
|
||||
|
||||
source_class.credential_name
|
||||
end
|
||||
|
||||
def enqueue_import(data_import)
|
||||
DataImports::Source.source_class(data_import.source_provider).import_job_class.perform_later(
|
||||
data_import,
|
||||
data_import.active_import_run_id
|
||||
)
|
||||
end
|
||||
|
||||
def render_source_validation_error(message)
|
||||
render json: { valid: false, message: message }, status: :unprocessable_entity
|
||||
end
|
||||
|
||||
def render_source_client_error(error)
|
||||
raise error unless source_class.client_error?(error)
|
||||
|
||||
message = if source_class.authentication_error?(error)
|
||||
"We could not validate this #{source_name} #{credential_name}. Check the key and its permissions."
|
||||
else
|
||||
"#{source_name} could not be reached. Please try again."
|
||||
end
|
||||
render_source_validation_error(message)
|
||||
end
|
||||
|
||||
def render_show
|
||||
@import_errors_finder = DataImportErrorFinder.new(@data_import)
|
||||
@skip_logs_finder = DataImportSkipLogFinder.new(@data_import, params)
|
||||
|
||||
@@ -421,6 +421,10 @@
|
||||
"DESCRIPTION": "Bring your existing contacts and past conversations into this account from another support tool. Each import runs in the background, so you can keep working while it finishes, track its progress, and review anything that was skipped along the way.",
|
||||
"LOADING": "Fetching imports",
|
||||
"DEFAULT_IMPORT_NAME": "Intercom import",
|
||||
"DEFAULT_IMPORT_NAMES": {
|
||||
"INTERCOM": "Intercom import",
|
||||
"FRESHDESK": "Freshdesk import"
|
||||
},
|
||||
"TABS": {
|
||||
"IMPORT": "Import",
|
||||
"EXPORT": "Export"
|
||||
@@ -434,9 +438,15 @@
|
||||
"TITLE": "New import",
|
||||
"SOURCE": "Source",
|
||||
"NAME": "Import name",
|
||||
"NAME_PLACEHOLDER": "July Intercom migration",
|
||||
"NAME_PLACEHOLDER": "July support migration",
|
||||
"ACCESS_KEY": "Intercom access key",
|
||||
"ACCESS_KEY_PLACEHOLDER": "Paste your Intercom access key",
|
||||
"INTERCOM_ACCESS_KEY": "Intercom access key",
|
||||
"INTERCOM_ACCESS_KEY_PLACEHOLDER": "Paste your Intercom access key",
|
||||
"FRESHDESK_API_KEY": "Freshdesk API key",
|
||||
"FRESHDESK_API_KEY_PLACEHOLDER": "Paste your Freshdesk API key",
|
||||
"FRESHDESK_DOMAIN": "Freshdesk domain",
|
||||
"FRESHDESK_DOMAIN_PLACEHOLDER": "acme.freshdesk.com",
|
||||
"DATA_TYPES": "Data to import",
|
||||
"VALIDATING": "Validating access key...",
|
||||
"VALID_KEY": "Access key validated.",
|
||||
@@ -510,11 +520,11 @@
|
||||
}
|
||||
},
|
||||
"ALERTS": {
|
||||
"IMPORT_STARTED": "Intercom import has started.",
|
||||
"IMPORT_RETRIED": "Intercom import has been queued to resume.",
|
||||
"IMPORT_RETRY_FAILED": "Could not retry the Intercom import.",
|
||||
"IMPORT_ABANDONED": "Intercom import has been abandoned.",
|
||||
"IMPORT_FAILED": "Could not start the Intercom import."
|
||||
"IMPORT_STARTED": "Import has started.",
|
||||
"IMPORT_RETRIED": "Import has been queued to resume.",
|
||||
"IMPORT_RETRY_FAILED": "Could not retry the import.",
|
||||
"IMPORT_ABANDONED": "Import has been abandoned.",
|
||||
"IMPORT_FAILED": "Could not start the import."
|
||||
}
|
||||
},
|
||||
"CAPTAIN_SETTINGS": {
|
||||
|
||||
@@ -24,7 +24,7 @@ import {
|
||||
formatStatus,
|
||||
importedCount,
|
||||
isActiveImport,
|
||||
isActiveIntercomImport,
|
||||
isActiveIntegrationImport,
|
||||
statusDotClass,
|
||||
} from './importStatus';
|
||||
|
||||
@@ -53,8 +53,8 @@ const activeTabIndex = computed(() =>
|
||||
);
|
||||
|
||||
const hasActiveImport = computed(() => dataImports.value.some(isActiveImport));
|
||||
const hasActiveIntercomImport = computed(() =>
|
||||
dataImports.value.some(isActiveIntercomImport)
|
||||
const hasActiveIntegrationImport = computed(() =>
|
||||
dataImports.value.some(isActiveIntegrationImport)
|
||||
);
|
||||
|
||||
const dataImportRoute = dataImport => ({
|
||||
@@ -137,7 +137,7 @@ const openImport = dataImport => {
|
||||
};
|
||||
|
||||
const openImportDrawer = () => {
|
||||
if (!hasActiveIntercomImport.value) showImportDrawer.value = true;
|
||||
if (!hasActiveIntegrationImport.value) showImportDrawer.value = true;
|
||||
};
|
||||
|
||||
const onImportCreated = dataImportId => {
|
||||
@@ -227,9 +227,9 @@ onBeforeUnmount(() => {
|
||||
<Button
|
||||
size="sm"
|
||||
:label="$t('DATA_IMPORTS.TABLE.NEW_IMPORT')"
|
||||
:disabled="hasActiveIntercomImport"
|
||||
:disabled="hasActiveIntegrationImport"
|
||||
:title="
|
||||
hasActiveIntercomImport
|
||||
hasActiveIntegrationImport
|
||||
? $t('DATA_IMPORTS.DRAWER.ACTIVE_IMPORT')
|
||||
: undefined
|
||||
"
|
||||
@@ -371,7 +371,7 @@ onBeforeUnmount(() => {
|
||||
|
||||
<NewImportDialog
|
||||
:show="showImportDrawer"
|
||||
:has-active-import="hasActiveIntercomImport"
|
||||
:has-active-import="hasActiveIntegrationImport"
|
||||
@close="showImportDrawer = false"
|
||||
@created="onImportCreated"
|
||||
/>
|
||||
|
||||
@@ -8,7 +8,7 @@ import Checkbox from 'dashboard/components-next/checkbox/Checkbox.vue';
|
||||
import Input from 'dashboard/components-next/input/Input.vue';
|
||||
import Select from 'dashboard/components-next/select/Select.vue';
|
||||
import DataImportsAPI from 'dashboard/api/dataImports';
|
||||
import { IMPORT_SOURCES } from './importSources';
|
||||
import { IMPORT_SOURCES, importSourceConfigFor } from './importSources';
|
||||
|
||||
const props = defineProps({
|
||||
show: { type: Boolean, default: false },
|
||||
@@ -20,8 +20,17 @@ const emit = defineEmits(['close', 'created']);
|
||||
const { t } = useI18n();
|
||||
const dialogRef = ref(null);
|
||||
const sourceProvider = ref('intercom');
|
||||
const importName = ref(t('DATA_IMPORTS.DEFAULT_IMPORT_NAME'));
|
||||
const sourceConfig = computed(
|
||||
() => importSourceConfigFor(sourceProvider.value) || IMPORT_SOURCES[0]
|
||||
);
|
||||
const defaultImportName = computed(() =>
|
||||
sourceProvider.value === 'freshdesk'
|
||||
? t('DATA_IMPORTS.DEFAULT_IMPORT_NAMES.FRESHDESK')
|
||||
: t('DATA_IMPORTS.DEFAULT_IMPORT_NAMES.INTERCOM')
|
||||
);
|
||||
const importName = ref(defaultImportName.value);
|
||||
const accessToken = ref('');
|
||||
const domain = ref('');
|
||||
const selectedImportTypes = ref(['contacts', 'conversations']);
|
||||
const validationState = ref('idle');
|
||||
const validationMessage = ref('');
|
||||
@@ -34,6 +43,17 @@ const sourceOptions = computed(() =>
|
||||
IMPORT_SOURCES.map(({ value, label }) => ({ value, label }))
|
||||
);
|
||||
|
||||
const credentialPlaceholder = computed(() =>
|
||||
sourceProvider.value === 'freshdesk'
|
||||
? t('DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY_PLACEHOLDER')
|
||||
: t('DATA_IMPORTS.DRAWER.INTERCOM_ACCESS_KEY_PLACEHOLDER')
|
||||
);
|
||||
const credentialLabel = computed(() =>
|
||||
sourceProvider.value === 'freshdesk'
|
||||
? t('DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY')
|
||||
: t('DATA_IMPORTS.DRAWER.INTERCOM_ACCESS_KEY')
|
||||
);
|
||||
|
||||
const tokenMessageType = computed(() => {
|
||||
if (validationState.value === 'valid') return 'success';
|
||||
if (validationState.value === 'invalid') return 'error';
|
||||
@@ -51,9 +71,14 @@ const canCreate = computed(
|
||||
const validationPayload = () => ({
|
||||
source_provider: sourceProvider.value,
|
||||
access_token: accessToken.value.trim(),
|
||||
...(sourceConfig.value.requiresDomain ? { domain: domain.value.trim() } : {}),
|
||||
import_types: selectedImportTypes.value,
|
||||
});
|
||||
|
||||
const hasRequiredCredentials = () =>
|
||||
accessToken.value.trim() &&
|
||||
(!sourceConfig.value.requiresDomain || domain.value.trim());
|
||||
|
||||
const invalidateValidation = () => {
|
||||
validationRequestId += 1;
|
||||
validationState.value = 'idle';
|
||||
@@ -61,7 +86,7 @@ const invalidateValidation = () => {
|
||||
};
|
||||
|
||||
const validateSource = async () => {
|
||||
if (!accessToken.value.trim() || !selectedImportTypes.value.length) {
|
||||
if (!hasRequiredCredentials() || !selectedImportTypes.value.length) {
|
||||
invalidateValidation();
|
||||
return;
|
||||
}
|
||||
@@ -98,7 +123,7 @@ const createImport = async () => {
|
||||
try {
|
||||
const response = await DataImportsAPI.create({
|
||||
...validationPayload(),
|
||||
name: importName.value.trim() || t('DATA_IMPORTS.DEFAULT_IMPORT_NAME'),
|
||||
name: importName.value.trim() || defaultImportName.value,
|
||||
});
|
||||
useAlert(t('DATA_IMPORTS.ALERTS.IMPORT_STARTED'));
|
||||
emit('created', response.data.id);
|
||||
@@ -112,10 +137,18 @@ const createImport = async () => {
|
||||
};
|
||||
|
||||
watch(accessToken, invalidateValidation);
|
||||
watch(domain, invalidateValidation);
|
||||
|
||||
watch(sourceProvider, () => {
|
||||
importName.value = defaultImportName.value;
|
||||
accessToken.value = '';
|
||||
domain.value = '';
|
||||
invalidateValidation();
|
||||
});
|
||||
|
||||
watch(selectedImportTypes, () => {
|
||||
invalidateValidation();
|
||||
if (accessToken.value.trim() && selectedImportTypes.value.length) {
|
||||
if (hasRequiredCredentials() && selectedImportTypes.value.length) {
|
||||
validateSource();
|
||||
}
|
||||
});
|
||||
@@ -130,6 +163,7 @@ watch(
|
||||
|
||||
dialogRef.value?.close();
|
||||
accessToken.value = '';
|
||||
domain.value = '';
|
||||
validationState.value = 'idle';
|
||||
validationMessage.value = '';
|
||||
}
|
||||
@@ -164,12 +198,21 @@ watch(
|
||||
:placeholder="$t('DATA_IMPORTS.DRAWER.NAME_PLACEHOLDER')"
|
||||
/>
|
||||
|
||||
<Input
|
||||
v-if="sourceConfig.requiresDomain"
|
||||
v-model="domain"
|
||||
autocomplete="off"
|
||||
:label="$t('DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN')"
|
||||
:placeholder="$t('DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN_PLACEHOLDER')"
|
||||
@blur="validateSource"
|
||||
/>
|
||||
|
||||
<Input
|
||||
v-model="accessToken"
|
||||
type="password"
|
||||
autocomplete="off"
|
||||
:label="$t('DATA_IMPORTS.DRAWER.ACCESS_KEY')"
|
||||
:placeholder="$t('DATA_IMPORTS.DRAWER.ACCESS_KEY_PLACEHOLDER')"
|
||||
:label="credentialLabel"
|
||||
:placeholder="credentialPlaceholder"
|
||||
:message="validationMessage"
|
||||
:message-type="tokenMessageType"
|
||||
@blur="validateSource"
|
||||
|
||||
@@ -4,8 +4,17 @@ export const IMPORT_SOURCES = [
|
||||
label: 'Intercom',
|
||||
icon: '/dashboard/images/integrations/intercom.png',
|
||||
},
|
||||
{
|
||||
value: 'freshdesk',
|
||||
label: 'Freshdesk',
|
||||
iconClass: 'i-lucide-life-buoy',
|
||||
requiresDomain: true,
|
||||
},
|
||||
];
|
||||
|
||||
export const importSourceConfigFor = provider =>
|
||||
IMPORT_SOURCES.find(source => source.value === provider);
|
||||
|
||||
const DEFAULT_IMPORT_SOURCE = {
|
||||
value: 'file',
|
||||
label: 'File import',
|
||||
@@ -13,5 +22,4 @@ const DEFAULT_IMPORT_SOURCE = {
|
||||
};
|
||||
|
||||
export const importSourceFor = dataImport =>
|
||||
IMPORT_SOURCES.find(source => source.value === dataImport?.source_provider) ||
|
||||
DEFAULT_IMPORT_SOURCE;
|
||||
importSourceConfigFor(dataImport?.source_provider) || DEFAULT_IMPORT_SOURCE;
|
||||
|
||||
@@ -9,14 +9,21 @@ export const isIntercomImport = dataImport =>
|
||||
dataImport?.data_type === 'intercom' &&
|
||||
dataImport?.source_provider === 'intercom';
|
||||
|
||||
export const isIntegrationImport = dataImport =>
|
||||
['freshdesk', 'intercom'].includes(dataImport?.data_type) &&
|
||||
dataImport?.data_type === dataImport?.source_provider;
|
||||
|
||||
export const isActiveIntercomImport = dataImport =>
|
||||
isIntercomImport(dataImport) && isActiveImport(dataImport);
|
||||
|
||||
export const isActiveIntegrationImport = dataImport =>
|
||||
isIntegrationImport(dataImport) && isActiveImport(dataImport);
|
||||
|
||||
export const isAbandonableImport = dataImport =>
|
||||
isActiveIntercomImport(dataImport);
|
||||
isActiveIntegrationImport(dataImport);
|
||||
|
||||
export const importedCount = dataImport => {
|
||||
if (!isIntercomImport(dataImport)) {
|
||||
if (!isIntegrationImport(dataImport)) {
|
||||
return Number(dataImport?.processed_records || 0);
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
import { flushPromises, mount } from '@vue/test-utils';
|
||||
import { nextTick } from 'vue';
|
||||
import DataImportsAPI from 'dashboard/api/dataImports';
|
||||
import NewImportDialog from '../NewImportDialog.vue';
|
||||
|
||||
vi.mock('dashboard/api/dataImports', () => ({
|
||||
default: {
|
||||
validateSource: vi.fn(),
|
||||
create: vi.fn(),
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('dashboard/composables', () => ({
|
||||
useAlert: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock('vue-i18n', () => ({
|
||||
useI18n: () => ({ t: key => key }),
|
||||
}));
|
||||
|
||||
const DialogStub = {
|
||||
name: 'Dialog',
|
||||
props: {
|
||||
disableConfirmButton: { type: Boolean, default: false },
|
||||
},
|
||||
emits: ['close', 'confirm'],
|
||||
methods: {
|
||||
open() {},
|
||||
close() {},
|
||||
},
|
||||
template: `
|
||||
<section>
|
||||
<slot />
|
||||
<button
|
||||
data-test="confirm"
|
||||
:disabled="disableConfirmButton"
|
||||
@click="$emit('confirm')"
|
||||
/>
|
||||
</section>
|
||||
`,
|
||||
};
|
||||
|
||||
const mountDialog = () =>
|
||||
mount(NewImportDialog, {
|
||||
props: { show: true },
|
||||
global: {
|
||||
stubs: {
|
||||
Dialog: DialogStub,
|
||||
Checkbox: true,
|
||||
},
|
||||
mocks: {
|
||||
$t: key => key,
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
describe('NewImportDialog', () => {
|
||||
beforeEach(() => {
|
||||
DataImportsAPI.validateSource.mockResolvedValue({
|
||||
data: { valid: true, totals: {} },
|
||||
});
|
||||
DataImportsAPI.create.mockResolvedValue({ data: { id: 42 } });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
it('validates and creates a Freshdesk import with its domain', async () => {
|
||||
const wrapper = mountDialog();
|
||||
await wrapper.find('select').setValue('freshdesk');
|
||||
await nextTick();
|
||||
|
||||
const domainInput = wrapper.find(
|
||||
'input[placeholder="DATA_IMPORTS.DRAWER.FRESHDESK_DOMAIN_PLACEHOLDER"]'
|
||||
);
|
||||
const apiKeyInput = wrapper.find(
|
||||
'input[placeholder="DATA_IMPORTS.DRAWER.FRESHDESK_API_KEY_PLACEHOLDER"]'
|
||||
);
|
||||
await domainInput.setValue('acme.freshdesk.com');
|
||||
await apiKeyInput.setValue(' freshdesk-api-key ');
|
||||
await apiKeyInput.trigger('blur');
|
||||
await flushPromises();
|
||||
|
||||
expect(DataImportsAPI.validateSource).toHaveBeenCalledWith({
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'acme.freshdesk.com',
|
||||
access_token: 'freshdesk-api-key',
|
||||
import_types: ['contacts', 'conversations'],
|
||||
});
|
||||
|
||||
await wrapper.find('[data-test="confirm"]').trigger('click');
|
||||
await flushPromises();
|
||||
|
||||
expect(DataImportsAPI.create).toHaveBeenCalledWith({
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'acme.freshdesk.com',
|
||||
access_token: 'freshdesk-api-key',
|
||||
import_types: ['contacts', 'conversations'],
|
||||
name: 'DATA_IMPORTS.DEFAULT_IMPORT_NAMES.FRESHDESK',
|
||||
});
|
||||
expect(wrapper.emitted('created')).toEqual([[42]]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
import {
|
||||
IMPORT_SOURCES,
|
||||
importSourceConfigFor,
|
||||
importSourceFor,
|
||||
} from '../importSources';
|
||||
|
||||
describe('importSources', () => {
|
||||
it('exposes Freshdesk as a domain-based integration source', () => {
|
||||
expect(IMPORT_SOURCES.map(source => source.value)).toEqual([
|
||||
'intercom',
|
||||
'freshdesk',
|
||||
]);
|
||||
expect(importSourceConfigFor('freshdesk')).toMatchObject({
|
||||
label: 'Freshdesk',
|
||||
requiresDomain: true,
|
||||
});
|
||||
});
|
||||
|
||||
it('resolves persisted integrations and falls back for file imports', () => {
|
||||
expect(importSourceFor({ source_provider: 'freshdesk' }).label).toBe(
|
||||
'Freshdesk'
|
||||
);
|
||||
expect(importSourceFor({ source_provider: null })).toMatchObject({
|
||||
value: 'file',
|
||||
label: 'File import',
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -2,6 +2,7 @@ import {
|
||||
formatDate,
|
||||
importedCount,
|
||||
isActiveIntercomImport,
|
||||
isActiveIntegrationImport,
|
||||
statusDotClass,
|
||||
} from '../importStatus';
|
||||
|
||||
@@ -32,12 +33,38 @@ describe('importStatus', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('importedCount', () => {
|
||||
it('sums Intercom imported stats', () => {
|
||||
describe('isActiveIntegrationImport', () => {
|
||||
it('treats matching Freshdesk and Intercom imports as active', () => {
|
||||
expect(
|
||||
importedCount({
|
||||
isActiveIntegrationImport({
|
||||
data_type: 'freshdesk',
|
||||
source_provider: 'freshdesk',
|
||||
status: 'pending',
|
||||
})
|
||||
).toBe(true);
|
||||
expect(
|
||||
isActiveIntegrationImport({
|
||||
data_type: 'intercom',
|
||||
source_provider: 'intercom',
|
||||
status: 'processing',
|
||||
})
|
||||
).toBe(true);
|
||||
expect(
|
||||
isActiveIntegrationImport({
|
||||
data_type: 'freshdesk',
|
||||
source_provider: 'intercom',
|
||||
status: 'processing',
|
||||
})
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('importedCount', () => {
|
||||
it('sums integration imported stats', () => {
|
||||
expect(
|
||||
importedCount({
|
||||
data_type: 'freshdesk',
|
||||
source_provider: 'freshdesk',
|
||||
processed_records: 20,
|
||||
stats: {
|
||||
contacts: { imported: 2 },
|
||||
|
||||
47
app/jobs/data_imports/base_job.rb
Normal file
47
app/jobs/data_imports/base_job.rb
Normal file
@@ -0,0 +1,47 @@
|
||||
class DataImports::BaseJob < ApplicationJob
|
||||
queue_as :low
|
||||
|
||||
def fail_import!(error)
|
||||
data_import = arguments.first
|
||||
run_id = arguments.length > 1 ? arguments.last : nil
|
||||
return if data_import.blank? || skip_import?(data_import, run_id)
|
||||
|
||||
importer_for(data_import, run_id).fail!(error)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def skip_import?(data_import, run_id = nil)
|
||||
data_import.reload
|
||||
data_import.abandoned? || data_import.failed? || data_import.completed? ||
|
||||
data_import.completed_with_errors? || stale_import_run?(data_import, run_id)
|
||||
end
|
||||
|
||||
def stale_import_run?(data_import, run_id)
|
||||
active_run_id = data_import.active_import_run_id
|
||||
active_run_id.present? && active_run_id != run_id
|
||||
end
|
||||
|
||||
def importer_for(data_import, run_id = nil)
|
||||
source_class(data_import).importer_class.new(data_import: data_import, run_id: run_id)
|
||||
end
|
||||
|
||||
def fail_unexpected_error(importer, error)
|
||||
raise error if source_class(arguments.first).client_error?(error)
|
||||
|
||||
importer&.fail!(error)
|
||||
raise error
|
||||
end
|
||||
|
||||
def contacts_page_job_class(data_import)
|
||||
source_class(data_import).contacts_page_job_class
|
||||
end
|
||||
|
||||
def conversations_page_job_class(data_import)
|
||||
source_class(data_import).conversations_page_job_class
|
||||
end
|
||||
|
||||
def source_class(data_import)
|
||||
DataImports::Source.source_class(data_import.source_provider)
|
||||
end
|
||||
end
|
||||
26
app/jobs/data_imports/contacts_page_job.rb
Normal file
26
app/jobs/data_imports/contacts_page_job.rb
Normal file
@@ -0,0 +1,26 @@
|
||||
module DataImports::ContactsPageJob
|
||||
def perform(data_import, starting_after = nil, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return enqueue_conversations_or_finish(data_import, importer, run_id) if importer.contacts_completed?
|
||||
|
||||
result = importer.import_contacts_page(starting_after: starting_after)
|
||||
return if skip_import?(data_import, run_id)
|
||||
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
|
||||
|
||||
enqueue_conversations_or_finish(data_import, importer, run_id)
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def enqueue_conversations_or_finish(data_import, importer, run_id)
|
||||
if importer.import_conversations? && !importer.conversations_completed?
|
||||
conversations_page_job_class(data_import).perform_later(data_import, importer.cursor_for('conversations'), run_id)
|
||||
else
|
||||
importer.finish!
|
||||
end
|
||||
end
|
||||
end
|
||||
16
app/jobs/data_imports/conversations_page_job.rb
Normal file
16
app/jobs/data_imports/conversations_page_job.rb
Normal file
@@ -0,0 +1,16 @@
|
||||
module DataImports::ConversationsPageJob
|
||||
def perform(data_import, starting_after = nil, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return importer.finish! if importer.conversations_completed?
|
||||
|
||||
result = importer.import_conversations_page(starting_after: starting_after)
|
||||
return if skip_import?(data_import, run_id)
|
||||
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
|
||||
|
||||
importer.finish!
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
end
|
||||
26
app/jobs/data_imports/freshdesk/base_job.rb
Normal file
26
app/jobs/data_imports/freshdesk/base_job.rb
Normal file
@@ -0,0 +1,26 @@
|
||||
class DataImports::Freshdesk::BaseJob < DataImports::BaseJob
|
||||
DEFAULT_RATE_LIMIT_WAIT = 1.minute
|
||||
|
||||
retry_on DataImports::Freshdesk::Client::Error, wait: 1.minute, attempts: 3 do |job, error|
|
||||
job.fail_import!(error)
|
||||
end
|
||||
|
||||
retry_on DataImports::Freshdesk::Client::RateLimitError, wait: DEFAULT_RATE_LIMIT_WAIT, attempts: 5 do |job, error|
|
||||
job.fail_import!(error)
|
||||
end
|
||||
|
||||
discard_on CustomExceptions::DataImport::FreshdeskTicketLimitError
|
||||
|
||||
def retry_job(options = {})
|
||||
error = options[:error]
|
||||
options = options.merge(wait: rate_limit_wait(error)) if error.is_a?(DataImports::Freshdesk::Client::RateLimitError)
|
||||
super(options)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def rate_limit_wait(error)
|
||||
retry_after = error.retry_after.to_i
|
||||
retry_after.positive? ? retry_after.seconds : DEFAULT_RATE_LIMIT_WAIT
|
||||
end
|
||||
end
|
||||
3
app/jobs/data_imports/freshdesk/contacts_page_job.rb
Normal file
3
app/jobs/data_imports/freshdesk/contacts_page_job.rb
Normal file
@@ -0,0 +1,3 @@
|
||||
class DataImports::Freshdesk::ContactsPageJob < DataImports::Freshdesk::BaseJob
|
||||
include DataImports::ContactsPageJob
|
||||
end
|
||||
@@ -0,0 +1,3 @@
|
||||
class DataImports::Freshdesk::ConversationsPageJob < DataImports::Freshdesk::BaseJob
|
||||
include DataImports::ConversationsPageJob
|
||||
end
|
||||
3
app/jobs/data_imports/freshdesk/import_job.rb
Normal file
3
app/jobs/data_imports/freshdesk/import_job.rb
Normal file
@@ -0,0 +1,3 @@
|
||||
class DataImports::Freshdesk::ImportJob < DataImports::Freshdesk::BaseJob
|
||||
include DataImports::ImportJob
|
||||
end
|
||||
24
app/jobs/data_imports/import_job.rb
Normal file
24
app/jobs/data_imports/import_job.rb
Normal file
@@ -0,0 +1,24 @@
|
||||
module DataImports::ImportJob
|
||||
def perform(data_import, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return unless importer.start!
|
||||
|
||||
enqueue_next_stage(data_import, importer, run_id)
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def enqueue_next_stage(data_import, importer, run_id)
|
||||
if importer.import_contacts? && !importer.contacts_completed?
|
||||
contacts_page_job_class(data_import).perform_later(data_import, importer.cursor_for('contacts'), run_id)
|
||||
elsif importer.import_conversations? && !importer.conversations_completed?
|
||||
conversations_page_job_class(data_import).perform_later(data_import, importer.cursor_for('conversations'), run_id)
|
||||
else
|
||||
importer.finish!
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -1,6 +1,4 @@
|
||||
class DataImports::Intercom::BaseJob < ApplicationJob
|
||||
queue_as :low
|
||||
|
||||
class DataImports::Intercom::BaseJob < DataImports::BaseJob
|
||||
retry_on DataImports::Intercom::Client::Error, wait: 1.minute, attempts: 3 do |job, error|
|
||||
job.fail_import!(error)
|
||||
end
|
||||
@@ -8,36 +6,4 @@ class DataImports::Intercom::BaseJob < ApplicationJob
|
||||
retry_on DataImports::Intercom::Client::RateLimitError, wait: 1.minute, attempts: 5 do |job, error|
|
||||
job.fail_import!(error)
|
||||
end
|
||||
|
||||
def fail_import!(error)
|
||||
data_import = arguments.first
|
||||
run_id = arguments.length > 1 ? arguments.last : nil
|
||||
return if data_import.blank? || skip_import?(data_import, run_id)
|
||||
|
||||
DataImports::Intercom::Importer.new(data_import: data_import, run_id: run_id).fail!(error)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def skip_import?(data_import, run_id = nil)
|
||||
data_import.reload
|
||||
data_import.abandoned? || data_import.failed? || data_import.completed? ||
|
||||
data_import.completed_with_errors? || stale_import_run?(data_import, run_id)
|
||||
end
|
||||
|
||||
def stale_import_run?(data_import, run_id)
|
||||
active_run_id = data_import.active_intercom_import_run_id
|
||||
active_run_id.present? && active_run_id != run_id
|
||||
end
|
||||
|
||||
def importer_for(data_import, run_id = nil)
|
||||
DataImports::Intercom::Importer.new(data_import: data_import, run_id: run_id)
|
||||
end
|
||||
|
||||
def fail_unexpected_error(importer, error)
|
||||
raise error if error.is_a?(DataImports::Intercom::Client::Error)
|
||||
|
||||
importer&.fail!(error)
|
||||
raise error
|
||||
end
|
||||
end
|
||||
|
||||
@@ -1,26 +1,3 @@
|
||||
class DataImports::Intercom::ContactsPageJob < DataImports::Intercom::BaseJob
|
||||
def perform(data_import, starting_after = nil, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return enqueue_conversations_or_finish(data_import, importer, run_id) if importer.contacts_completed?
|
||||
|
||||
result = importer.import_contacts_page(starting_after: starting_after)
|
||||
return if skip_import?(data_import, run_id)
|
||||
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
|
||||
|
||||
enqueue_conversations_or_finish(data_import, importer, run_id)
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def enqueue_conversations_or_finish(data_import, importer, run_id)
|
||||
if importer.import_conversations? && !importer.conversations_completed?
|
||||
DataImports::Intercom::ConversationsPageJob.perform_later(data_import, importer.cursor_for('conversations'), run_id)
|
||||
else
|
||||
importer.finish!
|
||||
end
|
||||
end
|
||||
include DataImports::ContactsPageJob
|
||||
end
|
||||
|
||||
@@ -1,16 +1,3 @@
|
||||
class DataImports::Intercom::ConversationsPageJob < DataImports::Intercom::BaseJob
|
||||
def perform(data_import, starting_after = nil, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return importer.finish! if importer.conversations_completed?
|
||||
|
||||
result = importer.import_conversations_page(starting_after: starting_after)
|
||||
return if skip_import?(data_import, run_id)
|
||||
return self.class.perform_later(data_import, result.next_cursor, run_id) unless result.done?
|
||||
|
||||
importer.finish!
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
include DataImports::ConversationsPageJob
|
||||
end
|
||||
|
||||
@@ -1,24 +1,3 @@
|
||||
class DataImports::Intercom::ImportJob < DataImports::Intercom::BaseJob
|
||||
def perform(data_import, run_id = nil)
|
||||
return if skip_import?(data_import, run_id)
|
||||
|
||||
importer = importer_for(data_import, run_id)
|
||||
return unless importer.start!
|
||||
|
||||
enqueue_next_stage(data_import, importer, run_id)
|
||||
rescue StandardError => e
|
||||
fail_unexpected_error(importer, e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def enqueue_next_stage(data_import, importer, run_id)
|
||||
if importer.import_contacts? && !importer.contacts_completed?
|
||||
DataImports::Intercom::ContactsPageJob.perform_later(data_import, importer.cursor_for('contacts'), run_id)
|
||||
elsif importer.import_conversations? && !importer.conversations_completed?
|
||||
DataImports::Intercom::ConversationsPageJob.perform_later(data_import, importer.cursor_for('conversations'), run_id)
|
||||
else
|
||||
importer.finish!
|
||||
end
|
||||
end
|
||||
include DataImports::ImportJob
|
||||
end
|
||||
|
||||
@@ -4,6 +4,15 @@ module OutOfOffisable
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
OFFISABLE_ATTRS = %w[day_of_week closed_all_day open_hour open_minutes close_hour close_minutes open_all_day].freeze
|
||||
DEFAULT_WORKING_HOURS = [
|
||||
{ day_of_week: 0, closed_all_day: true, open_all_day: false },
|
||||
{ day_of_week: 1, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
|
||||
{ day_of_week: 2, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
|
||||
{ day_of_week: 3, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
|
||||
{ day_of_week: 4, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
|
||||
{ day_of_week: 5, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false },
|
||||
{ day_of_week: 6, closed_all_day: true, open_all_day: false }
|
||||
].map(&:freeze).freeze
|
||||
|
||||
included do
|
||||
has_many :working_hours, dependent: :destroy_async
|
||||
@@ -52,12 +61,6 @@ module OutOfOffisable
|
||||
private
|
||||
|
||||
def create_default_working_hours
|
||||
working_hours.create!(day_of_week: 0, closed_all_day: true, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 1, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 2, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 3, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 4, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 5, open_hour: 9, open_minutes: 0, close_hour: 17, close_minutes: 0, open_all_day: false)
|
||||
working_hours.create!(day_of_week: 6, closed_all_day: true, open_all_day: false)
|
||||
DEFAULT_WORKING_HOURS.each { |attributes| working_hours.create!(attributes) }
|
||||
end
|
||||
end
|
||||
|
||||
@@ -32,10 +32,12 @@
|
||||
# index_data_imports_on_source_provider (source_provider)
|
||||
#
|
||||
class DataImport < ApplicationRecord
|
||||
ACTIVE_IMPORT_RUN_ID_KEY = 'active_import_run_id'.freeze
|
||||
ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY = 'active_intercom_import_run_id'.freeze
|
||||
INTERCOM_STALLED_AFTER = 15.minutes
|
||||
IMPORT_STALLED_AFTER = 15.minutes
|
||||
INTERCOM_STALLED_AFTER = IMPORT_STALLED_AFTER
|
||||
LEGACY_DATA_TYPES = ['contacts'].freeze
|
||||
INTEGRATION_DATA_TYPES = ['intercom'].freeze
|
||||
INTEGRATION_DATA_TYPES = %w[freshdesk intercom].freeze
|
||||
IMPORT_TYPES = %w[contacts conversations].freeze
|
||||
|
||||
belongs_to :account
|
||||
@@ -48,12 +50,16 @@ class DataImport < ApplicationRecord
|
||||
has_many :import_errors, class_name: 'DataImportError', dependent: :destroy_async
|
||||
|
||||
validates :data_type, inclusion: { in: LEGACY_DATA_TYPES + INTEGRATION_DATA_TYPES, message: I18n.t('errors.data_import.data_type.invalid') }
|
||||
validates :access_token, presence: true, on: :create, if: :intercom_import?
|
||||
validates :access_token, presence: true, on: :create, if: :integration_import?
|
||||
validate :validate_import_types
|
||||
validate :validate_integration_provider
|
||||
|
||||
enum status: { pending: 0, processing: 1, completed: 2, failed: 3, completed_with_errors: 6, abandoned: 7 }
|
||||
|
||||
scope :active_intercom, -> { where(data_type: 'intercom', source_provider: 'intercom', status: [:pending, :processing]) }
|
||||
scope :active_integrations, lambda {
|
||||
where(data_type: INTEGRATION_DATA_TYPES, status: [:pending, :processing]).where('source_provider = data_type')
|
||||
}
|
||||
|
||||
has_one_attached :import_file
|
||||
has_one_attached :failed_records
|
||||
@@ -68,38 +74,52 @@ class DataImport < ApplicationRecord
|
||||
data_type == 'intercom' && source_provider == 'intercom'
|
||||
end
|
||||
|
||||
def freshdesk_import?
|
||||
data_type == 'freshdesk' && source_provider == 'freshdesk'
|
||||
end
|
||||
|
||||
def integration_import?
|
||||
INTEGRATION_DATA_TYPES.include?(data_type) && data_type == source_provider
|
||||
end
|
||||
|
||||
def restartable?
|
||||
failed? || abandoned?
|
||||
end
|
||||
|
||||
def stalled?
|
||||
intercom_import? && (pending? || processing?) && updated_at <= INTERCOM_STALLED_AFTER.ago
|
||||
integration_import? && (pending? || processing?) && updated_at <= IMPORT_STALLED_AFTER.ago
|
||||
end
|
||||
|
||||
def abandonable?
|
||||
intercom_import? && (pending? || processing?)
|
||||
integration_import? && (pending? || processing?)
|
||||
end
|
||||
|
||||
def abandon!
|
||||
self.class.transaction do
|
||||
abandonable_import = self.class.lock.find_by(
|
||||
id: id,
|
||||
data_type: 'intercom',
|
||||
source_provider: 'intercom',
|
||||
status: [:pending, :processing]
|
||||
)
|
||||
active_imports = self.class.lock.where(id: id, data_type: INTEGRATION_DATA_TYPES, status: [:pending, :processing])
|
||||
abandonable_import = active_imports.where('source_provider = data_type').first
|
||||
abandonable_import&.update!(status: :abandoned, abandoned_at: Time.current)
|
||||
end
|
||||
reload
|
||||
end
|
||||
|
||||
def active_intercom_import_run_id
|
||||
source_metadata.to_h[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY]
|
||||
active_import_run_id
|
||||
end
|
||||
|
||||
def assign_active_intercom_import_run_id
|
||||
self.source_metadata = source_metadata.to_h.merge(ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY => SecureRandom.uuid)
|
||||
active_intercom_import_run_id
|
||||
assign_active_import_run_id
|
||||
end
|
||||
|
||||
def active_import_run_id
|
||||
source_metadata.to_h[ACTIVE_IMPORT_RUN_ID_KEY] || source_metadata.to_h[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY]
|
||||
end
|
||||
|
||||
def assign_active_import_run_id
|
||||
run_id = SecureRandom.uuid
|
||||
self.source_metadata = source_metadata.to_h.merge(ACTIVE_IMPORT_RUN_ID_KEY => run_id)
|
||||
source_metadata[ACTIVE_INTERCOM_IMPORT_RUN_ID_KEY] = run_id if intercom_import?
|
||||
run_id
|
||||
end
|
||||
|
||||
private
|
||||
@@ -119,4 +139,11 @@ class DataImport < ApplicationRecord
|
||||
|
||||
errors.add(:import_types, "contains unsupported values: #{invalid_types.join(', ')}")
|
||||
end
|
||||
|
||||
def validate_integration_provider
|
||||
return unless INTEGRATION_DATA_TYPES.include?(data_type)
|
||||
return if source_provider == data_type
|
||||
|
||||
errors.add(:source_provider, 'must match the integration data type')
|
||||
end
|
||||
end
|
||||
|
||||
65
app/services/data_imports/creation_service.rb
Normal file
65
app/services/data_imports/creation_service.rb
Normal file
@@ -0,0 +1,65 @@
|
||||
class DataImports::CreationService
|
||||
def initialize(account:, initiated_by:, source_params:)
|
||||
@account = account
|
||||
@initiated_by = initiated_by
|
||||
@source_params = source_params.symbolize_keys
|
||||
@access_token = @source_params[:access_token].to_s.strip
|
||||
@provider = @source_params[:source_provider].to_s
|
||||
@source_class = DataImports::Source.source_class(@provider)
|
||||
end
|
||||
|
||||
def perform
|
||||
return if active_import?
|
||||
|
||||
totals = validate_source
|
||||
@account.with_lock do
|
||||
next if active_import?
|
||||
|
||||
@account.data_imports.new(attributes(totals)).tap do |data_import|
|
||||
data_import.assign_active_import_run_id
|
||||
data_import.save!
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def validate_source
|
||||
@source_class.credentials_validator(source_params: @source_params, import_types: import_types).perform
|
||||
end
|
||||
|
||||
def attributes(totals)
|
||||
{
|
||||
name: @source_params[:name].presence || @source_class.default_import_name,
|
||||
data_type: @provider,
|
||||
source_type: 'api',
|
||||
source_provider: @provider,
|
||||
import_types: import_types,
|
||||
initiated_by: @initiated_by,
|
||||
access_token: @access_token,
|
||||
source_metadata: @source_class.source_metadata(@source_params),
|
||||
stats: initial_stats(totals)
|
||||
}
|
||||
end
|
||||
|
||||
def import_types
|
||||
return DataImports::Importer::DEFAULT_IMPORT_TYPES unless @source_params.key?(:import_types)
|
||||
|
||||
Array(@source_params[:import_types]).compact_blank
|
||||
end
|
||||
|
||||
def initial_stats(totals)
|
||||
{
|
||||
'contacts' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'conversations' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'messages' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'errors' => { 'count' => 0 }
|
||||
}.tap do |stats|
|
||||
totals.each { |type, total| stats[type]['total'] = total unless total.nil? }
|
||||
end
|
||||
end
|
||||
|
||||
def active_import?
|
||||
@account.data_imports.active_integrations.exists?
|
||||
end
|
||||
end
|
||||
160
app/services/data_imports/freshdesk/client.rb
Normal file
160
app/services/data_imports/freshdesk/client.rb
Normal file
@@ -0,0 +1,160 @@
|
||||
require 'uri'
|
||||
|
||||
class DataImports::Freshdesk::Client
|
||||
class Error < StandardError
|
||||
attr_reader :status, :body
|
||||
|
||||
def initialize(message, status: nil, body: nil)
|
||||
super(message)
|
||||
@status = status
|
||||
@body = body
|
||||
end
|
||||
end
|
||||
|
||||
class AuthenticationError < Error; end
|
||||
|
||||
class RateLimitError < Error
|
||||
attr_reader :retry_after
|
||||
|
||||
def initialize(message, retry_after: nil, **)
|
||||
super(message, **)
|
||||
@retry_after = retry_after
|
||||
end
|
||||
end
|
||||
|
||||
Page = Struct.new(:data, :next_page, keyword_init: true)
|
||||
|
||||
DEFAULT_PER_PAGE = 100
|
||||
MAX_TICKET_PAGES = 300
|
||||
EARLIEST_TICKET_TIMESTAMP = '1970-01-01T00:00:00Z'.freeze
|
||||
FRESHDESK_DOMAIN_REGEX = /\A[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.freshdesk\.com\z/i
|
||||
|
||||
attr_reader :domain
|
||||
|
||||
def self.normalize_domain(value)
|
||||
candidate = value.to_s.strip.downcase
|
||||
candidate = "#{candidate}.freshdesk.com" unless candidate.include?('.')
|
||||
candidate = URI.parse(candidate).host if candidate.start_with?('http://', 'https://')
|
||||
raise ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.' unless candidate&.match?(FRESHDESK_DOMAIN_REGEX)
|
||||
|
||||
candidate
|
||||
rescue URI::InvalidURIError
|
||||
raise ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.'
|
||||
end
|
||||
|
||||
def initialize(domain:, api_key:)
|
||||
@domain = self.class.normalize_domain(domain)
|
||||
@api_key = api_key.to_s.strip
|
||||
end
|
||||
|
||||
def list_contacts(page: 1, per_page: DEFAULT_PER_PAGE)
|
||||
get_page('/contacts', query: { page: page, per_page: per_page })
|
||||
end
|
||||
|
||||
def retrieve_contact(id)
|
||||
get("/contacts/#{id}")
|
||||
end
|
||||
|
||||
def list_tickets(page: 1, per_page: DEFAULT_PER_PAGE)
|
||||
get_page(
|
||||
'/tickets',
|
||||
query: {
|
||||
page: page,
|
||||
per_page: per_page,
|
||||
updated_since: EARLIEST_TICKET_TIMESTAMP
|
||||
}
|
||||
)
|
||||
end
|
||||
|
||||
def retrieve_ticket(id)
|
||||
get("/tickets/#{id}", query: { include: 'requester' })
|
||||
end
|
||||
|
||||
def list_conversations(ticket_id, page: 1, per_page: DEFAULT_PER_PAGE)
|
||||
get_page("/tickets/#{ticket_id}/conversations", query: { page: page, per_page: per_page })
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def get_page(path, query: {})
|
||||
response = request(path, query: query)
|
||||
Page.new(
|
||||
data: parsed_body(response),
|
||||
next_page: next_page(response.headers['link'], current_page: query[:page] || query['page'])
|
||||
)
|
||||
end
|
||||
|
||||
def get(path, query: {})
|
||||
parsed_body(request(path, query: query))
|
||||
end
|
||||
|
||||
def request(path, query: {})
|
||||
response =
|
||||
begin
|
||||
HTTParty.get(
|
||||
"https://#{@domain}/api/v2#{path}",
|
||||
query: query,
|
||||
basic_auth: { username: @api_key, password: 'X' },
|
||||
headers: { 'Accept' => 'application/json', 'Content-Type' => 'application/json' },
|
||||
timeout: 30
|
||||
)
|
||||
rescue StandardError => e
|
||||
raise Error.new(
|
||||
"Freshdesk API request failed before receiving a response: #{e.message}",
|
||||
body: { transport_error_class: e.class.name }
|
||||
)
|
||||
end
|
||||
|
||||
parse_response(response)
|
||||
response
|
||||
end
|
||||
|
||||
def parse_response(response)
|
||||
return if response.success?
|
||||
|
||||
body = parsed_body(response)
|
||||
message = error_message(body, response)
|
||||
case response.code
|
||||
when 401, 403
|
||||
raise AuthenticationError.new(message, status: response.code, body: body)
|
||||
when 429
|
||||
raise RateLimitError.new(message, status: response.code, body: body, retry_after: response.headers['retry-after'])
|
||||
else
|
||||
raise Error.new(message, status: response.code, body: body)
|
||||
end
|
||||
end
|
||||
|
||||
def parsed_body(response)
|
||||
response.parsed_response.presence || (response.code == 204 ? {} : [])
|
||||
rescue JSON::ParserError
|
||||
{}
|
||||
end
|
||||
|
||||
def error_message(body, response)
|
||||
description = body['description'] if body.is_a?(Hash)
|
||||
errors = body['errors'] if body.is_a?(Hash)
|
||||
first_error = errors.first if errors.is_a?(Array)
|
||||
first_error_message = first_error['message'] if first_error.is_a?(Hash)
|
||||
first_error_message.presence || description.presence || "Freshdesk API request failed with status #{response.code}"
|
||||
end
|
||||
|
||||
def next_page(link_header, current_page:)
|
||||
next_link = link_header.to_s.split(',').find { |link| link.include?('rel="next"') }
|
||||
return if next_link.blank?
|
||||
|
||||
url = next_link[/<([^>]+)>/, 1]
|
||||
return if url.blank?
|
||||
|
||||
parse_next_page(url, current_page)
|
||||
rescue ArgumentError, URI::InvalidURIError
|
||||
nil
|
||||
end
|
||||
|
||||
def parse_next_page(url, current_page)
|
||||
uri = URI.parse(url)
|
||||
page = URI.decode_www_form(uri.query.to_s).to_h['page']
|
||||
same_domain = uri.host.blank? || uri.host.casecmp?(@domain)
|
||||
positive_page = page&.match?(/\A[1-9]\d*\z/)
|
||||
page.to_i if same_domain && positive_page && page.to_i > current_page.to_i
|
||||
end
|
||||
end
|
||||
31
app/services/data_imports/freshdesk/credentials_validator.rb
Normal file
31
app/services/data_imports/freshdesk/credentials_validator.rb
Normal file
@@ -0,0 +1,31 @@
|
||||
class DataImports::Freshdesk::CredentialsValidator
|
||||
def initialize(domain:, api_key:, import_types:)
|
||||
@domain = domain
|
||||
@api_key = api_key.to_s.strip
|
||||
@import_types = Array(import_types).compact_blank
|
||||
end
|
||||
|
||||
def perform
|
||||
validate_parameters!
|
||||
client.list_contacts(per_page: 1) if @import_types.include?('contacts')
|
||||
client.list_tickets(per_page: 1) if @import_types.include?('conversations')
|
||||
{}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def validate_parameters!
|
||||
raise ArgumentError, 'Freshdesk domain is required.' if @domain.blank?
|
||||
raise ArgumentError, 'Freshdesk API key is required.' if @api_key.blank?
|
||||
raise ArgumentError, 'Select at least one data type to import.' if @import_types.blank?
|
||||
|
||||
invalid_types = @import_types - DataImport::IMPORT_TYPES
|
||||
raise ArgumentError, "Unsupported import types: #{invalid_types.join(', ')}" if invalid_types.present?
|
||||
|
||||
DataImports::Freshdesk::Client.normalize_domain(@domain)
|
||||
end
|
||||
|
||||
def client
|
||||
@client ||= DataImports::Freshdesk::Client.new(domain: @domain, api_key: @api_key)
|
||||
end
|
||||
end
|
||||
59
app/services/data_imports/freshdesk/importer.rb
Normal file
59
app/services/data_imports/freshdesk/importer.rb
Normal file
@@ -0,0 +1,59 @@
|
||||
class DataImports::Freshdesk::Importer < DataImports::Importer
|
||||
InvalidMessagePayloadError = DataImports::Importer::InvalidMessagePayloadError
|
||||
ALREADY_IMPORTED_ERROR_CODE = DataImports::Freshdesk::Source::ALREADY_IMPORTED_ERROR_CODE
|
||||
SKIPPED_MESSAGE_ERROR_CODE = DataImports::Freshdesk::Source::SKIPPED_MESSAGE_ERROR_CODE
|
||||
TRUNCATED_PARTS_ERROR_CODE = DataImports::Freshdesk::Source::TRUNCATED_PARTS_ERROR_CODE
|
||||
|
||||
def import_conversations_page(starting_after: cursor_for('conversations'))
|
||||
response = conversations_page(starting_after)
|
||||
return PageResult.new(next_cursor: cursor_for('conversations')) unless checkpoint_ticket_page(response)
|
||||
|
||||
update_stat_total('conversations', response['total_count']) if response['total_count'].present?
|
||||
return PageResult.new(next_cursor: cursor_for('conversations')) unless import_conversation_summaries(response)
|
||||
return PageResult.new(next_cursor: nil) if import_stopped?
|
||||
|
||||
reconcile_dirty_stats
|
||||
verify_ticket_history_complete!(response)
|
||||
next_cursor = response.dig('pages', 'next', 'starting_after')
|
||||
next_cursor = update_cursor('conversations', next_cursor)
|
||||
PageResult.new(next_cursor: next_cursor)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def checkpoint_ticket_page(response)
|
||||
current_cursor = response.dig('pages', 'current', 'starting_after')
|
||||
return true if current_cursor == cursor_for('conversations')
|
||||
|
||||
update_cursor('conversations', current_cursor) == current_cursor
|
||||
end
|
||||
|
||||
def conversations_page(starting_after)
|
||||
@source.list_conversations(
|
||||
starting_after: cursor_for('conversations').presence || starting_after,
|
||||
per_page: @source.conversations_per_page
|
||||
)
|
||||
end
|
||||
|
||||
def verify_ticket_history_complete!(response)
|
||||
raise CustomExceptions::DataImport::FreshdeskTicketLimitError if response.dig('pages', 'limit_reached')
|
||||
end
|
||||
|
||||
def import_conversation_summaries(response)
|
||||
summaries = Array(response['data'] || response['conversations'])
|
||||
checkpoints = Array(response.dig('pages', 'checkpoints'))
|
||||
|
||||
summaries.each_with_index do |conversation_summary, index|
|
||||
break if import_stopped?
|
||||
|
||||
import_conversation_from_summary(conversation_summary)
|
||||
break if import_stopped?
|
||||
|
||||
next if index == summaries.length - 1
|
||||
|
||||
checkpoint = checkpoints.fetch(index)
|
||||
return false unless update_cursor('conversations', checkpoint) == checkpoint
|
||||
end
|
||||
true
|
||||
end
|
||||
end
|
||||
10
app/services/data_imports/freshdesk/message_batch_builder.rb
Normal file
10
app/services/data_imports/freshdesk/message_batch_builder.rb
Normal file
@@ -0,0 +1,10 @@
|
||||
class DataImports::Freshdesk::MessageBatchBuilder < DataImports::MessageBatchBuilder
|
||||
def initialize(data_import:, conversation:, source_conversation:)
|
||||
super(
|
||||
data_import: data_import,
|
||||
conversation: conversation,
|
||||
source_conversation: source_conversation,
|
||||
provider: 'freshdesk'
|
||||
)
|
||||
end
|
||||
end
|
||||
51
app/services/data_imports/freshdesk/metadata.rb
Normal file
51
app/services/data_imports/freshdesk/metadata.rb
Normal file
@@ -0,0 +1,51 @@
|
||||
class DataImports::Freshdesk::Metadata
|
||||
def contact_custom_attributes(contact)
|
||||
{
|
||||
freshdesk_contact_id: contact['id'],
|
||||
freshdesk_unique_external_id: contact['external_id']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def contact_source(contact)
|
||||
{
|
||||
contact_id: contact['id'],
|
||||
unique_external_id: contact['external_id'],
|
||||
raw_phone: contact['phone'],
|
||||
other_emails: contact['other_emails'],
|
||||
custom_fields: contact['custom_fields'],
|
||||
tags: contact['tags']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def conversation_source(conversation)
|
||||
{
|
||||
ticket_id: conversation['id'],
|
||||
subject: conversation['subject'],
|
||||
status: conversation['status'],
|
||||
priority: conversation['priority'],
|
||||
requester_id: conversation['requester_id'],
|
||||
responder_id: conversation['responder_id'],
|
||||
group_id: conversation['group_id'],
|
||||
company_id: conversation['company_id'],
|
||||
tags: conversation['tags'],
|
||||
custom_fields: conversation['custom_fields'],
|
||||
archived: conversation['archived']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def message_source(part)
|
||||
{
|
||||
conversation_id: part['id'],
|
||||
conversation_source: part['source'],
|
||||
user_id: part.dig('author', 'id'),
|
||||
incoming: part['incoming'],
|
||||
private: part['private'],
|
||||
from_email: part['from_email'],
|
||||
to_emails: part['to_emails'],
|
||||
cc_emails: part['cc_emails'],
|
||||
bcc_emails: part['bcc_emails'],
|
||||
attachments: part['attachments'],
|
||||
deleted: part['deleted']
|
||||
}.compact
|
||||
end
|
||||
end
|
||||
138
app/services/data_imports/freshdesk/normalizer.rb
Normal file
138
app/services/data_imports/freshdesk/normalizer.rb
Normal file
@@ -0,0 +1,138 @@
|
||||
class DataImports::Freshdesk::Normalizer
|
||||
WEB_CHAT_SOURCE = 15
|
||||
|
||||
def contact(contact)
|
||||
{
|
||||
'id' => contact['id']&.to_s,
|
||||
'name' => contact['name'],
|
||||
'email' => contact['email'],
|
||||
'phone' => contact['mobile'].presence || contact['phone'],
|
||||
'external_id' => contact['unique_external_id'].presence || contact['external_id'],
|
||||
'created_at' => unix_timestamp(contact['created_at']),
|
||||
'updated_at' => unix_timestamp(contact['updated_at']),
|
||||
'last_seen_at' => unix_timestamp(contact['updated_at']),
|
||||
'other_emails' => contact['other_emails'],
|
||||
'custom_fields' => contact['custom_fields'].presence || contact['custom_field'],
|
||||
'tags' => contact['tags']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def ticket(ticket, conversations)
|
||||
{
|
||||
'id' => ticket['id'].to_s,
|
||||
'created_at' => unix_timestamp(ticket['created_at']),
|
||||
'updated_at' => unix_timestamp(ticket['updated_at']),
|
||||
'source' => ticket_source(ticket),
|
||||
'contacts' => { 'contacts' => conversation_contacts(ticket, conversations) },
|
||||
'conversation_parts' => conversation_parts(conversations)
|
||||
}.merge(ticket_metadata(ticket)).compact
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def ticket_source(ticket)
|
||||
source = {
|
||||
'id' => ticket['id'].to_s,
|
||||
'type' => DataImports::Freshdesk::SourceBucket.source_type(ticket['source']),
|
||||
'subject' => ticket['subject'],
|
||||
'body' => ticket['description'].presence || ticket['description_text'],
|
||||
'attachments' => ticket['attachments'],
|
||||
'author' => source_author(ticket)
|
||||
}.compact
|
||||
|
||||
source.except!('subject', 'body') if ticket['source'].to_i == WEB_CHAT_SOURCE
|
||||
source
|
||||
end
|
||||
|
||||
def source_author(ticket)
|
||||
requester(ticket).merge('type' => outgoing_source?(ticket) ? 'admin' : 'contact')
|
||||
end
|
||||
|
||||
def outgoing_source?(ticket)
|
||||
[10, 14].include?(ticket['source'].to_i)
|
||||
end
|
||||
|
||||
def requester(ticket)
|
||||
requester = ticket['requester'].is_a?(Hash) ? ticket['requester'] : {}
|
||||
contact(requester.merge('id' => requester['id'] || ticket['requester_id']))
|
||||
end
|
||||
|
||||
def conversation_parts(conversations)
|
||||
sorted_conversations = conversations.each_with_index.sort_by do |conversation, index|
|
||||
[conversation['created_at'].to_s, index]
|
||||
end.map(&:first)
|
||||
{
|
||||
'conversation_parts' => sorted_conversations.map { |conversation| message(conversation) },
|
||||
'total_count' => conversations.size
|
||||
}
|
||||
end
|
||||
|
||||
def conversation_contacts(ticket, conversations)
|
||||
contacts = [requester(ticket)]
|
||||
contacts.concat(conversations.filter_map { |conversation| conversation_contact(conversation) })
|
||||
contacts.uniq { |contact_payload| contact_payload['id'].presence || contact_payload['email'].to_s.downcase.presence }
|
||||
end
|
||||
|
||||
def conversation_contact(conversation)
|
||||
return unless conversation['incoming']
|
||||
return if conversation['user_id'].blank? && conversation['from_email'].blank?
|
||||
|
||||
{
|
||||
'id' => conversation['user_id']&.to_s,
|
||||
'name' => conversation['from_email'],
|
||||
'email' => conversation['from_email']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def ticket_metadata(ticket)
|
||||
{
|
||||
'subject' => ticket['subject'],
|
||||
'status' => ticket['status'],
|
||||
'priority' => ticket['priority'],
|
||||
'requester_id' => ticket['requester_id'],
|
||||
'responder_id' => ticket['responder_id'],
|
||||
'group_id' => ticket['group_id'],
|
||||
'company_id' => ticket['company_id'],
|
||||
'tags' => ticket['tags'],
|
||||
'custom_fields' => ticket['custom_fields'],
|
||||
'archived' => ticket['archived']
|
||||
}
|
||||
end
|
||||
|
||||
def message(conversation)
|
||||
message_attributes(conversation).merge(message_metadata(conversation)).compact
|
||||
end
|
||||
|
||||
def message_attributes(conversation)
|
||||
{
|
||||
'id' => conversation['id'].to_s,
|
||||
'part_type' => conversation['private'] ? 'note' : 'comment',
|
||||
'body' => conversation['body'].presence || conversation['body_text'],
|
||||
'attachments' => conversation['attachments'],
|
||||
'created_at' => unix_timestamp(conversation['created_at']),
|
||||
'updated_at' => unix_timestamp(conversation['updated_at']),
|
||||
'author' => message_author(conversation)
|
||||
}
|
||||
end
|
||||
|
||||
def message_metadata(conversation)
|
||||
conversation.slice(
|
||||
'source', 'incoming', 'private', 'from_email', 'to_emails', 'cc_emails', 'bcc_emails', 'deleted'
|
||||
)
|
||||
end
|
||||
|
||||
def message_author(conversation)
|
||||
{
|
||||
'id' => conversation['user_id']&.to_s,
|
||||
'type' => conversation['incoming'] ? 'contact' : 'admin',
|
||||
'name' => conversation['from_email'],
|
||||
'email' => conversation['from_email']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def unix_timestamp(value)
|
||||
return if value.blank?
|
||||
|
||||
Time.zone.parse(value.to_s).to_i
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,5 @@
|
||||
class DataImports::Freshdesk::PlaceholderInboxBuilder < DataImports::PlaceholderInboxBuilder
|
||||
def initialize(account:)
|
||||
super(account: account, provider: 'freshdesk', source_bucket: DataImports::Freshdesk::SourceBucket)
|
||||
end
|
||||
end
|
||||
196
app/services/data_imports/freshdesk/source.rb
Normal file
196
app/services/data_imports/freshdesk/source.rb
Normal file
@@ -0,0 +1,196 @@
|
||||
class DataImports::Freshdesk::Source
|
||||
PROVIDER = 'freshdesk'.freeze
|
||||
DISPLAY_NAME = 'Freshdesk'.freeze
|
||||
CONTACTS_PER_PAGE = 100
|
||||
CONVERSATIONS_PER_PAGE = 100
|
||||
ALREADY_IMPORTED_ERROR_CODE = 'DataImports::Freshdesk::AlreadyImported'.freeze
|
||||
SKIPPED_MESSAGE_ERROR_CODE = 'DataImports::Freshdesk::SkippedMessage'.freeze
|
||||
TRUNCATED_PARTS_ERROR_CODE = 'DataImports::Freshdesk::TruncatedConversations'.freeze
|
||||
|
||||
attr_reader :provider, :display_name, :contacts_per_page, :conversations_per_page
|
||||
|
||||
def self.credentials_validator(source_params:, import_types:)
|
||||
DataImports::Freshdesk::CredentialsValidator.new(
|
||||
domain: source_params[:domain],
|
||||
api_key: source_params[:access_token],
|
||||
import_types: import_types
|
||||
)
|
||||
end
|
||||
|
||||
def self.source_metadata(source_params)
|
||||
{ domain: DataImports::Freshdesk::Client.normalize_domain(source_params[:domain]) }
|
||||
end
|
||||
|
||||
def self.default_import_name
|
||||
'Freshdesk import'
|
||||
end
|
||||
|
||||
def self.credential_name
|
||||
'API key'
|
||||
end
|
||||
|
||||
def self.import_job_class
|
||||
DataImports::Freshdesk::ImportJob
|
||||
end
|
||||
|
||||
def self.importer_class
|
||||
DataImports::Freshdesk::Importer
|
||||
end
|
||||
|
||||
def self.contacts_page_job_class
|
||||
DataImports::Freshdesk::ContactsPageJob
|
||||
end
|
||||
|
||||
def self.conversations_page_job_class
|
||||
DataImports::Freshdesk::ConversationsPageJob
|
||||
end
|
||||
|
||||
def self.client_error?(error)
|
||||
error.is_a?(DataImports::Freshdesk::Client::Error)
|
||||
end
|
||||
|
||||
def self.authentication_error?(error)
|
||||
error.is_a?(DataImports::Freshdesk::Client::AuthenticationError)
|
||||
end
|
||||
|
||||
def initialize(access_token:, source_metadata: {})
|
||||
@provider = PROVIDER
|
||||
@display_name = DISPLAY_NAME
|
||||
@contacts_per_page = CONTACTS_PER_PAGE
|
||||
@conversations_per_page = CONVERSATIONS_PER_PAGE
|
||||
@client = DataImports::Freshdesk::Client.new(domain: source_metadata['domain'] || source_metadata[:domain], api_key: access_token)
|
||||
@normalizer = DataImports::Freshdesk::Normalizer.new
|
||||
@metadata = DataImports::Freshdesk::Metadata.new
|
||||
end
|
||||
|
||||
def list_contacts(starting_after:, per_page:)
|
||||
page = @client.list_contacts(page: starting_after.presence || 1, per_page: per_page)
|
||||
paginated_response(page, Array(page.data).map { |contact| @normalizer.contact(contact) })
|
||||
end
|
||||
|
||||
def list_conversations(starting_after:, per_page:)
|
||||
page = DataImports::Freshdesk::TicketPage.new(client: @client, starting_after: starting_after, per_page: per_page)
|
||||
summaries = page.chunk.map do |ticket|
|
||||
{
|
||||
'id' => ticket['id'].to_s,
|
||||
'source' => { 'type' => DataImports::Freshdesk::SourceBucket.source_type(ticket['source']) }
|
||||
}
|
||||
end
|
||||
{
|
||||
'data' => summaries,
|
||||
'pages' => {
|
||||
'current' => { 'starting_after' => page.current_cursor },
|
||||
'next' => page.next_cursor.present? ? { 'starting_after' => page.next_cursor } : nil,
|
||||
'checkpoints' => page.checkpoints,
|
||||
'limit_reached' => page.limit_reached?
|
||||
}
|
||||
}
|
||||
end
|
||||
|
||||
def retrieve_conversation(id)
|
||||
ticket = @client.retrieve_ticket(id)
|
||||
conversations = all_conversations(id)
|
||||
@normalizer.ticket(ticket, conversations)
|
||||
end
|
||||
|
||||
def retrieve_contact(id)
|
||||
@normalizer.contact(@client.retrieve_contact(id))
|
||||
end
|
||||
|
||||
def client_error?(error)
|
||||
error.is_a?(DataImports::Freshdesk::Client::Error)
|
||||
end
|
||||
|
||||
def placeholder_inbox_builder(account:)
|
||||
DataImports::Freshdesk::PlaceholderInboxBuilder.new(account: account)
|
||||
end
|
||||
|
||||
def message_batch_builder(data_import:, conversation:, source_conversation:)
|
||||
DataImports::Freshdesk::MessageBatchBuilder.new(
|
||||
data_import: data_import,
|
||||
conversation: conversation,
|
||||
source_conversation: source_conversation
|
||||
)
|
||||
end
|
||||
|
||||
def activity_part?(_part)
|
||||
false
|
||||
end
|
||||
|
||||
def source_message_importable?(source)
|
||||
DataImports::Freshdesk::MessageBatchBuilder.source_message_importable?(source)
|
||||
end
|
||||
|
||||
def activity_content(_part)
|
||||
nil
|
||||
end
|
||||
|
||||
def already_imported_error_code
|
||||
ALREADY_IMPORTED_ERROR_CODE
|
||||
end
|
||||
|
||||
def skipped_message_error_code
|
||||
SKIPPED_MESSAGE_ERROR_CODE
|
||||
end
|
||||
|
||||
def truncated_parts_error_code
|
||||
TRUNCATED_PARTS_ERROR_CODE
|
||||
end
|
||||
|
||||
def skipped_message_reason
|
||||
'blank_freshdesk_conversation'
|
||||
end
|
||||
|
||||
def contact_custom_attributes(contact_payload)
|
||||
@metadata.contact_custom_attributes(contact_payload)
|
||||
end
|
||||
|
||||
def contact_source_metadata(contact_payload)
|
||||
@metadata.contact_source(contact_payload)
|
||||
end
|
||||
|
||||
def conversation_custom_attributes(conversation)
|
||||
{
|
||||
freshdesk_ticket_id: conversation['id'],
|
||||
freshdesk_subject: conversation['subject'],
|
||||
freshdesk_status: conversation['status'],
|
||||
freshdesk_priority: conversation['priority']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def conversation_source_metadata(conversation)
|
||||
@metadata.conversation_source(conversation)
|
||||
end
|
||||
|
||||
def message_source_metadata(part)
|
||||
@metadata.message_source(part)
|
||||
end
|
||||
|
||||
def timestamp_for(value)
|
||||
return Time.current if value.blank?
|
||||
|
||||
value.is_a?(Numeric) || value.to_s.match?(/\A-?\d+(?:\.\d+)?\z/) ? Time.zone.at(value.to_f) : Time.zone.parse(value.to_s)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def paginated_response(page, records)
|
||||
{
|
||||
'data' => records,
|
||||
'pages' => { 'next' => page.next_page.present? ? { 'starting_after' => page.next_page } : nil }
|
||||
}
|
||||
end
|
||||
|
||||
def all_conversations(ticket_id)
|
||||
records = []
|
||||
page_number = 1
|
||||
loop do
|
||||
page = @client.list_conversations(ticket_id, page: page_number, per_page: CONVERSATIONS_PER_PAGE)
|
||||
records.concat(Array(page.data))
|
||||
break if page.next_page.blank?
|
||||
|
||||
page_number = page.next_page
|
||||
end
|
||||
records
|
||||
end
|
||||
end
|
||||
61
app/services/data_imports/freshdesk/source_bucket.rb
Normal file
61
app/services/data_imports/freshdesk/source_bucket.rb
Normal file
@@ -0,0 +1,61 @@
|
||||
class DataImports::Freshdesk::SourceBucket
|
||||
SOURCE_TYPES = {
|
||||
1 => 'email',
|
||||
2 => 'portal',
|
||||
3 => 'phone',
|
||||
4 => 'forum',
|
||||
5 => 'twitter',
|
||||
6 => 'facebook',
|
||||
7 => 'chat',
|
||||
8 => 'mobihelp',
|
||||
9 => 'feedback_widget',
|
||||
10 => 'outbound_email',
|
||||
11 => 'ecommerce',
|
||||
12 => 'bot',
|
||||
13 => 'whatsapp',
|
||||
14 => 'chat_internal_task',
|
||||
15 => 'web_chat',
|
||||
16 => 'web_form',
|
||||
17 => 'instagram_message',
|
||||
18 => 'instagram_comment',
|
||||
19 => 'facebook_message',
|
||||
20 => 'facebook_comment',
|
||||
21 => 'mobile_chat_sdk',
|
||||
22 => 'sms'
|
||||
}.freeze
|
||||
|
||||
BUCKETS = {
|
||||
'email' => { key: 'email', name: 'Email' },
|
||||
'outbound_email' => { key: 'email', name: 'Email' },
|
||||
'phone' => { key: 'phone', name: 'Phone' },
|
||||
'forum' => { key: 'forum', name: 'Forum' },
|
||||
'twitter' => { key: 'twitter', name: 'Twitter' },
|
||||
'facebook' => { key: 'facebook', name: 'Facebook' },
|
||||
'portal' => { key: 'portal', name: 'Portal' },
|
||||
'feedback_widget' => { key: 'portal', name: 'Portal' },
|
||||
'chat' => { key: 'chat', name: 'Chat' },
|
||||
'mobihelp' => { key: 'mobile', name: 'Mobile' },
|
||||
'ecommerce' => { key: 'ecommerce', name: 'Ecommerce' },
|
||||
'bot' => { key: 'bot', name: 'Bot' },
|
||||
'whatsapp' => { key: 'whatsapp', name: 'WhatsApp' },
|
||||
'chat_internal_task' => { key: 'internal_task', name: 'Internal task' },
|
||||
'web_chat' => { key: 'chat', name: 'Chat' },
|
||||
'web_form' => { key: 'portal', name: 'Portal' },
|
||||
'instagram_message' => { key: 'instagram', name: 'Instagram' },
|
||||
'instagram_comment' => { key: 'instagram', name: 'Instagram' },
|
||||
'facebook_message' => { key: 'facebook', name: 'Facebook' },
|
||||
'facebook_comment' => { key: 'facebook', name: 'Facebook' },
|
||||
'mobile_chat_sdk' => { key: 'mobile', name: 'Mobile' },
|
||||
'sms' => { key: 'sms', name: 'SMS' }
|
||||
}.freeze
|
||||
|
||||
DEFAULT_BUCKET = { key: 'unknown', name: 'Unknown' }.freeze
|
||||
|
||||
def self.source_type(value)
|
||||
SOURCE_TYPES[value.to_i] || 'unknown'
|
||||
end
|
||||
|
||||
def self.for(source_type)
|
||||
BUCKETS[source_type.to_s.downcase] || DEFAULT_BUCKET
|
||||
end
|
||||
end
|
||||
62
app/services/data_imports/freshdesk/ticket_page.rb
Normal file
62
app/services/data_imports/freshdesk/ticket_page.rb
Normal file
@@ -0,0 +1,62 @@
|
||||
class DataImports::Freshdesk::TicketPage
|
||||
TICKETS_PER_CHUNK = 10
|
||||
|
||||
attr_reader :current_cursor
|
||||
|
||||
def initialize(client:, starting_after:, per_page:)
|
||||
@client = client
|
||||
@per_page = per_page
|
||||
@cursor = normalize_cursor(starting_after)
|
||||
@tickets, next_page = ticket_page(per_page)
|
||||
@page_limit_reached = page_limit_reached?
|
||||
next_page = nil if @page_limit_reached
|
||||
@current_cursor = @cursor.merge('tickets' => @tickets, 'next_page' => next_page)
|
||||
end
|
||||
|
||||
def chunk
|
||||
@chunk ||= @tickets.slice(@cursor['offset'], TICKETS_PER_CHUNK) || []
|
||||
end
|
||||
|
||||
def checkpoints
|
||||
@checkpoints ||= chunk.each_index.map { |index| cursor_after(index + 1) }
|
||||
end
|
||||
|
||||
def next_cursor
|
||||
return checkpoints.last if chunk.present?
|
||||
|
||||
cursor_after(0)
|
||||
end
|
||||
|
||||
def limit_reached?
|
||||
@page_limit_reached && next_cursor.nil?
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def normalize_cursor(starting_after)
|
||||
return { 'page' => starting_after.presence || 1, 'offset' => 0 } unless starting_after.is_a?(Hash)
|
||||
|
||||
cursor = starting_after.deep_stringify_keys.slice('page', 'offset', 'tickets', 'next_page')
|
||||
cursor.merge('page' => cursor['page'] || 1, 'offset' => cursor['offset'].to_i)
|
||||
end
|
||||
|
||||
def ticket_page(per_page)
|
||||
return [Array(@cursor['tickets']), @cursor['next_page']] if @cursor.key?('tickets')
|
||||
|
||||
page = @client.list_tickets(page: @cursor['page'], per_page: per_page)
|
||||
tickets = Array(page.data).map { |ticket| ticket.slice('id', 'source') }
|
||||
[tickets, page.next_page]
|
||||
end
|
||||
|
||||
def cursor_after(processed_count)
|
||||
next_offset = @cursor['offset'] + processed_count
|
||||
return current_cursor.merge('offset' => next_offset) if next_offset < @tickets.size
|
||||
return { 'page' => current_cursor['next_page'], 'offset' => 0 } if current_cursor['next_page'].present?
|
||||
|
||||
nil
|
||||
end
|
||||
|
||||
def page_limit_reached?
|
||||
@cursor['page'].to_i >= DataImports::Freshdesk::Client::MAX_TICKET_PAGES && @tickets.size >= @per_page
|
||||
end
|
||||
end
|
||||
1350
app/services/data_imports/importer.rb
Normal file
1350
app/services/data_imports/importer.rb
Normal file
File diff suppressed because it is too large
Load Diff
@@ -1,67 +1,2 @@
|
||||
class DataImports::Intercom::CreationService
|
||||
def initialize(account:, initiated_by:, source_params:)
|
||||
@account = account
|
||||
@initiated_by = initiated_by
|
||||
@source_params = source_params.symbolize_keys
|
||||
@access_token = @source_params[:access_token].to_s.strip
|
||||
end
|
||||
|
||||
def perform
|
||||
return if active_import?
|
||||
|
||||
totals = validate_source
|
||||
@account.with_lock do
|
||||
next if active_import?
|
||||
|
||||
@account.data_imports.new(attributes(totals)).tap do |data_import|
|
||||
data_import.assign_active_intercom_import_run_id
|
||||
data_import.save!
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def validate_source
|
||||
raise ArgumentError, 'Unsupported import source.' unless @source_params[:source_provider] == 'intercom'
|
||||
|
||||
DataImports::Intercom::CredentialsValidator.new(
|
||||
access_token: @access_token,
|
||||
import_types: import_types
|
||||
).perform
|
||||
end
|
||||
|
||||
def attributes(totals)
|
||||
{
|
||||
name: @source_params[:name].presence || 'Intercom import',
|
||||
data_type: 'intercom',
|
||||
source_type: 'api',
|
||||
source_provider: 'intercom',
|
||||
import_types: import_types,
|
||||
initiated_by: @initiated_by,
|
||||
access_token: @access_token,
|
||||
stats: initial_stats(totals)
|
||||
}
|
||||
end
|
||||
|
||||
def import_types
|
||||
return DataImports::Intercom::Importer::DEFAULT_IMPORT_TYPES unless @source_params.key?(:import_types)
|
||||
|
||||
Array(@source_params[:import_types]).compact_blank
|
||||
end
|
||||
|
||||
def initial_stats(totals)
|
||||
{
|
||||
'contacts' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'conversations' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'messages' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'errors' => { 'count' => 0 }
|
||||
}.tap do |stats|
|
||||
totals.each { |type, total| stats[type]['total'] = total unless total.nil? }
|
||||
end
|
||||
end
|
||||
|
||||
def active_import?
|
||||
@account.data_imports.active_intercom.exists?
|
||||
end
|
||||
class DataImports::Intercom::CreationService < DataImports::CreationService
|
||||
end
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,144 +1,5 @@
|
||||
class DataImports::Intercom::MessageBatchBuilder
|
||||
PROVIDER = 'intercom'.freeze
|
||||
REGULAR_PART_TYPES = %w[comment note source].freeze
|
||||
|
||||
Entry = Struct.new(:source_id, :part, :position, :mapping, :message, :classification, keyword_init: true) do
|
||||
def source?
|
||||
part['part_type'] == 'source'
|
||||
end
|
||||
end
|
||||
|
||||
Batch = Struct.new(:items, keyword_init: true) do
|
||||
def entries
|
||||
items
|
||||
end
|
||||
|
||||
def source_entries
|
||||
items.select(&:source?)
|
||||
end
|
||||
|
||||
def part_entries
|
||||
items.reject(&:source?)
|
||||
end
|
||||
end
|
||||
|
||||
def self.activity_part?(part)
|
||||
part_type = part['part_type'].to_s
|
||||
part_type.present? && REGULAR_PART_TYPES.exclude?(part_type)
|
||||
end
|
||||
|
||||
def self.source_message_importable?(source)
|
||||
source['body'].present? || source['subject'].present? || source['attachments'].present?
|
||||
end
|
||||
|
||||
class DataImports::Intercom::MessageBatchBuilder < DataImports::MessageBatchBuilder
|
||||
def initialize(data_import:, conversation:, source_conversation:)
|
||||
@data_import = data_import
|
||||
@account = data_import.account
|
||||
@conversation = conversation
|
||||
@source_conversation = source_conversation
|
||||
end
|
||||
|
||||
def perform(source_entries = unprepared_entries)
|
||||
classify(source_entries)
|
||||
end
|
||||
|
||||
def refresh(entries)
|
||||
classify(entries.map do |entry|
|
||||
{ source_id: entry.source_id, part: entry.part, position: entry.position }
|
||||
end)
|
||||
end
|
||||
|
||||
def unprepared_entries
|
||||
ordered_source_entries.map.with_index { |entry, position| entry.merge(position: position) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def classify(source_entries)
|
||||
return Batch.new(items: []) if source_entries.empty?
|
||||
|
||||
mappings = message_mappings(source_entries)
|
||||
messages = messages_for(source_entries, mappings)
|
||||
|
||||
Batch.new(items: source_entries.map.with_index do |source_entry, position|
|
||||
build_entry(source_entry, source_entry.fetch(:position, position), mappings, messages)
|
||||
end)
|
||||
end
|
||||
|
||||
def ordered_source_entries
|
||||
entries = []
|
||||
source = @source_conversation['source'].to_h
|
||||
if self.class.source_message_importable?(source)
|
||||
entries << {
|
||||
source_id: "conversation:#{source_conversation_id}:source:#{source['id'].presence || 'initial'}",
|
||||
part: source.merge('part_type' => 'source', 'created_at' => @source_conversation['created_at'])
|
||||
}
|
||||
end
|
||||
|
||||
conversation_parts.each do |part|
|
||||
entries << { source_id: "conversation:#{source_conversation_id}:part:#{part['id']}", part: part }
|
||||
end
|
||||
entries
|
||||
end
|
||||
|
||||
def conversation_parts
|
||||
Array(@source_conversation.dig('conversation_parts', 'conversation_parts'))
|
||||
end
|
||||
|
||||
def source_conversation_id
|
||||
@source_conversation['id'].presence || @source_conversation['external_id'].presence || @source_conversation['email'].presence
|
||||
end
|
||||
|
||||
def message_mappings(source_entries)
|
||||
DataImportMapping.where(
|
||||
account: @account,
|
||||
source_provider: PROVIDER,
|
||||
source_object_type: 'message',
|
||||
source_object_id: source_entries.pluck(:source_id)
|
||||
).index_by(&:source_object_id)
|
||||
end
|
||||
|
||||
def messages_for(source_entries, mappings)
|
||||
mapped_message_ids = mappings.values.filter_map do |mapping|
|
||||
mapping.chatwoot_record_id if mapping.chatwoot_record_type == 'Message'
|
||||
end
|
||||
chatwoot_source_ids = source_entries.map { |entry| "intercom:#{entry[:source_id]}" }
|
||||
messages = Message.where(id: mapped_message_ids).or(
|
||||
Message.where(conversation_id: @conversation.id, source_id: chatwoot_source_ids)
|
||||
).to_a
|
||||
|
||||
{
|
||||
by_id: messages.index_by(&:id),
|
||||
by_source_id: messages.index_by(&:source_id)
|
||||
}
|
||||
end
|
||||
|
||||
def build_entry(source_entry, position, mappings, messages)
|
||||
source_id = source_entry[:source_id]
|
||||
mapping = mappings[source_id]
|
||||
mapped_message = messages[:by_id][mapping.chatwoot_record_id] if mapping&.chatwoot_record_type == 'Message'
|
||||
existing_message = messages[:by_source_id]["intercom:#{source_id}"]
|
||||
|
||||
Entry.new(
|
||||
source_id: source_id,
|
||||
part: source_entry[:part],
|
||||
position: position,
|
||||
mapping: mapping,
|
||||
message: mapped_message || existing_message,
|
||||
classification: classification_for(mapping, mapped_message, existing_message, source_entry[:part])
|
||||
)
|
||||
end
|
||||
|
||||
def classification_for(mapping, mapped_message, existing_message, part)
|
||||
return existing_message.present? ? :existing_message : :new_message if mapping.blank?
|
||||
return :repairable_stale_mapping unless mapping_handled?(mapping, mapped_message, part)
|
||||
|
||||
mapping.data_import_id == @data_import.id ? :current_import : :previous_import
|
||||
end
|
||||
|
||||
def mapping_handled?(mapping, mapped_message, part)
|
||||
return false if mapping.metadata['skipped'] && self.class.activity_part?(part)
|
||||
|
||||
mapping.metadata['skipped'] || mapped_message.present?
|
||||
super(data_import: data_import, conversation: conversation, source_conversation: source_conversation, provider: 'intercom')
|
||||
end
|
||||
end
|
||||
|
||||
@@ -1,41 +1,5 @@
|
||||
class DataImports::Intercom::PlaceholderInboxBuilder
|
||||
AGENT_REPLY_TIME_WINDOW_HOURS = 1
|
||||
|
||||
class DataImports::Intercom::PlaceholderInboxBuilder < DataImports::PlaceholderInboxBuilder
|
||||
def initialize(account:)
|
||||
@account = account
|
||||
end
|
||||
|
||||
def inbox_for(source_type)
|
||||
bucket = DataImports::Intercom::SourceBucket.for(source_type)
|
||||
placeholder_inboxes[bucket[:key]] ||= create_placeholder_inbox(bucket)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def placeholder_inboxes
|
||||
@placeholder_inboxes ||= @account.inboxes.includes(:channel).where(channel_type: 'Channel::Api').each_with_object({}) do |inbox, inboxes|
|
||||
attrs = inbox.channel.additional_attributes || {}
|
||||
next unless attrs['source_provider'] == 'intercom' && attrs['import_placeholder'] == true
|
||||
|
||||
inboxes[attrs['source_bucket']] = inbox
|
||||
end
|
||||
end
|
||||
|
||||
def create_placeholder_inbox(bucket)
|
||||
channel = @account.api_channels.create!(
|
||||
additional_attributes: {
|
||||
source_provider: 'intercom',
|
||||
source_bucket: bucket[:key],
|
||||
import_placeholder: true,
|
||||
agent_reply_time_window: AGENT_REPLY_TIME_WINDOW_HOURS
|
||||
}
|
||||
)
|
||||
|
||||
@account.inboxes.create!(
|
||||
name: "Intercom Import - #{bucket[:name]}",
|
||||
channel: channel,
|
||||
enable_auto_assignment: false,
|
||||
allow_messages_after_resolved: false
|
||||
)
|
||||
super(account: account, provider: 'intercom', source_bucket: DataImports::Intercom::SourceBucket)
|
||||
end
|
||||
end
|
||||
|
||||
@@ -1,55 +1,2 @@
|
||||
class DataImports::Intercom::RestartService
|
||||
attr_reader :data_import
|
||||
|
||||
def initialize(account:, data_import:)
|
||||
@account = account
|
||||
@data_import = data_import
|
||||
end
|
||||
|
||||
def perform
|
||||
@account.with_lock do
|
||||
@data_import.reload
|
||||
next :render_show unless @data_import.restartable?
|
||||
|
||||
if (active_import = find_active_import)
|
||||
@data_import = active_import
|
||||
next :render_show
|
||||
end
|
||||
|
||||
next :access_token_missing if @data_import.access_token.blank?
|
||||
|
||||
@data_import.assign_active_intercom_import_run_id
|
||||
retained_skip_logs = @data_import.import_errors.where("details ->> 'kind' = ?", 'skipped')
|
||||
@data_import.import_errors.where.not(id: retained_skip_logs.select(:id)).delete_all
|
||||
@data_import.update!(restart_attributes(retained_skip_logs))
|
||||
:enqueue
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def find_active_import
|
||||
@account.data_imports.active_intercom.first
|
||||
end
|
||||
|
||||
def restart_attributes(retained_skip_logs)
|
||||
{
|
||||
status: :pending,
|
||||
abandoned_at: nil,
|
||||
completed_at: nil,
|
||||
last_error_at: nil,
|
||||
started_at: nil,
|
||||
stats: restart_stats(retained_skip_logs)
|
||||
}
|
||||
end
|
||||
|
||||
def restart_stats(retained_skip_logs)
|
||||
@data_import.stats.to_h.deep_dup.tap do |stats|
|
||||
%w[contact conversation message].each do |object_type|
|
||||
stats["#{object_type}s"] ||= {}
|
||||
stats["#{object_type}s"]['skipped'] = retained_skip_logs.where(source_object_type: object_type).count
|
||||
end
|
||||
stats['errors'] = { 'count' => 0 }
|
||||
end
|
||||
end
|
||||
class DataImports::Intercom::RestartService < DataImports::RestartService
|
||||
end
|
||||
|
||||
@@ -1,28 +1,2 @@
|
||||
class DataImports::Intercom::RetryService
|
||||
attr_reader :data_import
|
||||
|
||||
def initialize(account:, data_import:)
|
||||
@account = account
|
||||
@data_import = data_import
|
||||
end
|
||||
|
||||
def perform
|
||||
@account.with_lock do
|
||||
@data_import.with_lock do
|
||||
next :not_stalled unless @data_import.stalled?
|
||||
next :active_import_exists if another_active_import?
|
||||
next :access_token_missing if @data_import.access_token.blank?
|
||||
|
||||
@data_import.assign_active_intercom_import_run_id
|
||||
@data_import.update!(status: :pending)
|
||||
:enqueue
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def another_active_import?
|
||||
@account.data_imports.active_intercom.where.not(id: @data_import.id).exists?
|
||||
end
|
||||
class DataImports::Intercom::RetryService < DataImports::RetryService
|
||||
end
|
||||
|
||||
181
app/services/data_imports/intercom/source.rb
Normal file
181
app/services/data_imports/intercom/source.rb
Normal file
@@ -0,0 +1,181 @@
|
||||
class DataImports::Intercom::Source
|
||||
PROVIDER = 'intercom'.freeze
|
||||
DISPLAY_NAME = 'Intercom'.freeze
|
||||
CONTACTS_PER_PAGE = 50
|
||||
CONVERSATIONS_PER_PAGE = 10
|
||||
ALREADY_IMPORTED_ERROR_CODE = 'DataImports::Intercom::AlreadyImported'.freeze
|
||||
SKIPPED_MESSAGE_ERROR_CODE = 'DataImports::Intercom::SkippedMessage'.freeze
|
||||
TRUNCATED_PARTS_ERROR_CODE = 'DataImports::Intercom::TruncatedConversationParts'.freeze
|
||||
|
||||
attr_reader :provider, :display_name, :contacts_per_page, :conversations_per_page
|
||||
|
||||
def self.credentials_validator(source_params:, import_types:)
|
||||
DataImports::Intercom::CredentialsValidator.new(
|
||||
access_token: source_params[:access_token],
|
||||
import_types: import_types
|
||||
)
|
||||
end
|
||||
|
||||
def self.source_metadata(_source_params)
|
||||
{}
|
||||
end
|
||||
|
||||
def self.default_import_name
|
||||
'Intercom import'
|
||||
end
|
||||
|
||||
def self.credential_name
|
||||
'access key'
|
||||
end
|
||||
|
||||
def self.import_job_class
|
||||
DataImports::Intercom::ImportJob
|
||||
end
|
||||
|
||||
def self.importer_class
|
||||
DataImports::Intercom::Importer
|
||||
end
|
||||
|
||||
def self.contacts_page_job_class
|
||||
DataImports::Intercom::ContactsPageJob
|
||||
end
|
||||
|
||||
def self.conversations_page_job_class
|
||||
DataImports::Intercom::ConversationsPageJob
|
||||
end
|
||||
|
||||
def self.client_error?(error)
|
||||
error.is_a?(DataImports::Intercom::Client::Error)
|
||||
end
|
||||
|
||||
def self.authentication_error?(error)
|
||||
error.is_a?(DataImports::Intercom::Client::AuthenticationError)
|
||||
end
|
||||
|
||||
def initialize(access_token:, source_metadata: {}) # rubocop:disable Lint/UnusedMethodArgument
|
||||
@provider = PROVIDER
|
||||
@display_name = DISPLAY_NAME
|
||||
@contacts_per_page = CONTACTS_PER_PAGE
|
||||
@conversations_per_page = CONVERSATIONS_PER_PAGE
|
||||
@client = DataImports::Intercom::Client.new(access_token: access_token)
|
||||
end
|
||||
|
||||
def list_contacts(starting_after:, per_page:)
|
||||
@client.list_contacts(starting_after: starting_after, per_page: per_page)
|
||||
end
|
||||
|
||||
def list_conversations(starting_after:, per_page:)
|
||||
@client.list_conversations(starting_after: starting_after, per_page: per_page)
|
||||
end
|
||||
|
||||
def retrieve_conversation(id)
|
||||
@client.retrieve_conversation(id)
|
||||
end
|
||||
|
||||
def retrieve_contact(id)
|
||||
@client.retrieve_contact(id)
|
||||
end
|
||||
|
||||
def client_error?(error)
|
||||
error.is_a?(DataImports::Intercom::Client::Error)
|
||||
end
|
||||
|
||||
def placeholder_inbox_builder(account:)
|
||||
DataImports::Intercom::PlaceholderInboxBuilder.new(account: account)
|
||||
end
|
||||
|
||||
def message_batch_builder(data_import:, conversation:, source_conversation:)
|
||||
DataImports::Intercom::MessageBatchBuilder.new(
|
||||
data_import: data_import,
|
||||
conversation: conversation,
|
||||
source_conversation: source_conversation
|
||||
)
|
||||
end
|
||||
|
||||
def activity_part?(part)
|
||||
DataImports::Intercom::MessageBatchBuilder.activity_part?(part)
|
||||
end
|
||||
|
||||
def source_message_importable?(source)
|
||||
DataImports::Intercom::MessageBatchBuilder.source_message_importable?(source)
|
||||
end
|
||||
|
||||
def activity_content(part)
|
||||
DataImports::Intercom::ActivityContentBuilder.new(part).perform
|
||||
end
|
||||
|
||||
def already_imported_error_code
|
||||
ALREADY_IMPORTED_ERROR_CODE
|
||||
end
|
||||
|
||||
def skipped_message_error_code
|
||||
SKIPPED_MESSAGE_ERROR_CODE
|
||||
end
|
||||
|
||||
def truncated_parts_error_code
|
||||
TRUNCATED_PARTS_ERROR_CODE
|
||||
end
|
||||
|
||||
def skipped_message_reason
|
||||
'blank_or_unsupported_intercom_part'
|
||||
end
|
||||
|
||||
def contact_custom_attributes(contact_payload)
|
||||
{
|
||||
intercom_contact_id: contact_payload['id'],
|
||||
intercom_external_id: contact_payload['external_id']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def contact_source_metadata(contact_payload)
|
||||
{
|
||||
contact_id: contact_payload['id'],
|
||||
external_id: contact_payload['external_id'],
|
||||
raw_phone: contact_payload['phone']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def conversation_custom_attributes(conversation)
|
||||
{ intercom_conversation_id: source_id_for(conversation) }
|
||||
end
|
||||
|
||||
def conversation_source_metadata(conversation)
|
||||
{
|
||||
conversation_id: source_id_for(conversation),
|
||||
delivered_as: conversation.dig('source', 'delivered_as'),
|
||||
source_url: conversation.dig('source', 'url'),
|
||||
admin_assignee_id: conversation['admin_assignee_id'],
|
||||
team_assignee_id: conversation['team_assignee_id'],
|
||||
state: conversation['state'],
|
||||
open: conversation['open']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def message_source_metadata(part)
|
||||
{
|
||||
part_id: part['id'],
|
||||
part_type: part['part_type'],
|
||||
author: part['author'],
|
||||
assigned_to: part['assigned_to'],
|
||||
state: part['state'],
|
||||
tags: part['tags'],
|
||||
event_details: part['event_details'],
|
||||
app_package_code: part['app_package_code'],
|
||||
metadata: part['metadata'],
|
||||
attachments: part['attachments'],
|
||||
redacted: part['redacted']
|
||||
}.compact
|
||||
end
|
||||
|
||||
def timestamp_for(value)
|
||||
return Time.current if value.blank?
|
||||
|
||||
Time.zone.at(value.to_i)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def source_id_for(payload)
|
||||
payload['id'].presence || payload['external_id'].presence || payload['email'].presence
|
||||
end
|
||||
end
|
||||
144
app/services/data_imports/message_batch_builder.rb
Normal file
144
app/services/data_imports/message_batch_builder.rb
Normal file
@@ -0,0 +1,144 @@
|
||||
class DataImports::MessageBatchBuilder
|
||||
REGULAR_PART_TYPES = %w[comment note source].freeze
|
||||
|
||||
Entry = Struct.new(:source_id, :part, :position, :mapping, :message, :classification, keyword_init: true) do
|
||||
def source?
|
||||
part['part_type'] == 'source'
|
||||
end
|
||||
end
|
||||
|
||||
Batch = Struct.new(:items, keyword_init: true) do
|
||||
def entries
|
||||
items
|
||||
end
|
||||
|
||||
def source_entries
|
||||
items.select(&:source?)
|
||||
end
|
||||
|
||||
def part_entries
|
||||
items.reject(&:source?)
|
||||
end
|
||||
end
|
||||
|
||||
def self.activity_part?(part)
|
||||
part_type = part['part_type'].to_s
|
||||
part_type.present? && REGULAR_PART_TYPES.exclude?(part_type)
|
||||
end
|
||||
|
||||
def self.source_message_importable?(source)
|
||||
source['body'].present? || source['subject'].present? || source['attachments'].present?
|
||||
end
|
||||
|
||||
def initialize(data_import:, conversation:, source_conversation:, provider:)
|
||||
@data_import = data_import
|
||||
@account = data_import.account
|
||||
@conversation = conversation
|
||||
@source_conversation = source_conversation
|
||||
@provider = provider
|
||||
end
|
||||
|
||||
def perform(source_entries = unprepared_entries)
|
||||
classify(source_entries)
|
||||
end
|
||||
|
||||
def refresh(entries)
|
||||
classify(entries.map do |entry|
|
||||
{ source_id: entry.source_id, part: entry.part, position: entry.position }
|
||||
end)
|
||||
end
|
||||
|
||||
def unprepared_entries
|
||||
ordered_source_entries.map.with_index { |entry, position| entry.merge(position: position) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def classify(source_entries)
|
||||
return Batch.new(items: []) if source_entries.empty?
|
||||
|
||||
mappings = message_mappings(source_entries)
|
||||
messages = messages_for(source_entries, mappings)
|
||||
|
||||
Batch.new(items: source_entries.map.with_index do |source_entry, position|
|
||||
build_entry(source_entry, source_entry.fetch(:position, position), mappings, messages)
|
||||
end)
|
||||
end
|
||||
|
||||
def ordered_source_entries
|
||||
entries = []
|
||||
source = @source_conversation['source'].to_h
|
||||
if self.class.source_message_importable?(source)
|
||||
entries << {
|
||||
source_id: "conversation:#{source_conversation_id}:source:#{source['id'].presence || 'initial'}",
|
||||
part: source.merge('part_type' => 'source', 'created_at' => @source_conversation['created_at'])
|
||||
}
|
||||
end
|
||||
|
||||
conversation_parts.each do |part|
|
||||
entries << { source_id: "conversation:#{source_conversation_id}:part:#{part['id']}", part: part }
|
||||
end
|
||||
entries
|
||||
end
|
||||
|
||||
def conversation_parts
|
||||
Array(@source_conversation.dig('conversation_parts', 'conversation_parts'))
|
||||
end
|
||||
|
||||
def source_conversation_id
|
||||
@source_conversation['id'].presence || @source_conversation['external_id'].presence || @source_conversation['email'].presence
|
||||
end
|
||||
|
||||
def message_mappings(source_entries)
|
||||
DataImportMapping.where(
|
||||
account: @account,
|
||||
source_provider: @provider,
|
||||
source_object_type: 'message',
|
||||
source_object_id: source_entries.pluck(:source_id)
|
||||
).index_by(&:source_object_id)
|
||||
end
|
||||
|
||||
def messages_for(source_entries, mappings)
|
||||
mapped_message_ids = mappings.values.filter_map do |mapping|
|
||||
mapping.chatwoot_record_id if mapping.chatwoot_record_type == 'Message'
|
||||
end
|
||||
chatwoot_source_ids = source_entries.map { |entry| "#{@provider}:#{entry[:source_id]}" }
|
||||
messages = Message.where(id: mapped_message_ids).or(
|
||||
Message.where(conversation_id: @conversation.id, source_id: chatwoot_source_ids)
|
||||
).to_a
|
||||
|
||||
{
|
||||
by_id: messages.index_by(&:id),
|
||||
by_source_id: messages.index_by(&:source_id)
|
||||
}
|
||||
end
|
||||
|
||||
def build_entry(source_entry, position, mappings, messages)
|
||||
source_id = source_entry[:source_id]
|
||||
mapping = mappings[source_id]
|
||||
mapped_message = messages[:by_id][mapping.chatwoot_record_id] if mapping&.chatwoot_record_type == 'Message'
|
||||
existing_message = messages[:by_source_id]["#{@provider}:#{source_id}"]
|
||||
|
||||
Entry.new(
|
||||
source_id: source_id,
|
||||
part: source_entry[:part],
|
||||
position: position,
|
||||
mapping: mapping,
|
||||
message: mapped_message || existing_message,
|
||||
classification: classification_for(mapping, mapped_message, existing_message, source_entry[:part])
|
||||
)
|
||||
end
|
||||
|
||||
def classification_for(mapping, mapped_message, existing_message, part)
|
||||
return existing_message.present? ? :existing_message : :new_message if mapping.blank?
|
||||
return :repairable_stale_mapping unless mapping_handled?(mapping, mapped_message, part)
|
||||
|
||||
mapping.data_import_id == @data_import.id ? :current_import : :previous_import
|
||||
end
|
||||
|
||||
def mapping_handled?(mapping, mapped_message, part)
|
||||
return false if mapping.metadata['skipped'] && self.class.activity_part?(part)
|
||||
|
||||
mapping.metadata['skipped'] || mapped_message.present?
|
||||
end
|
||||
end
|
||||
65
app/services/data_imports/placeholder_inbox_builder.rb
Normal file
65
app/services/data_imports/placeholder_inbox_builder.rb
Normal file
@@ -0,0 +1,65 @@
|
||||
class DataImports::PlaceholderInboxBuilder
|
||||
AGENT_REPLY_TIME_WINDOW_HOURS = 1
|
||||
|
||||
def initialize(account:, provider:, source_bucket:)
|
||||
@account = account
|
||||
@provider = provider
|
||||
@source_bucket = source_bucket
|
||||
end
|
||||
|
||||
def inbox_for(source_type)
|
||||
bucket = @source_bucket.for(source_type)
|
||||
placeholder_inboxes[bucket[:key]] ||= create_placeholder_inbox(bucket)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def placeholder_inboxes
|
||||
@placeholder_inboxes ||= @account.inboxes.includes(:channel).where(channel_type: 'Channel::Api').each_with_object({}) do |inbox, inboxes|
|
||||
attrs = inbox.channel.additional_attributes || {}
|
||||
next unless attrs['source_provider'] == @provider && attrs['import_placeholder'] == true
|
||||
|
||||
inboxes[attrs['source_bucket']] = inbox
|
||||
end
|
||||
end
|
||||
|
||||
def create_placeholder_inbox(bucket)
|
||||
Inbox.transaction do
|
||||
channel = create_placeholder_channel(bucket)
|
||||
now = Time.current
|
||||
result = Inbox.insert_all!( # rubocop:disable Rails/SkipsModelValidations
|
||||
[placeholder_inbox_attributes(channel, bucket, now)],
|
||||
returning: %w[id]
|
||||
)
|
||||
Inbox.find(result.rows.first.first).tap { |inbox| create_default_working_hours(inbox) }
|
||||
end
|
||||
end
|
||||
|
||||
def create_placeholder_channel(bucket)
|
||||
@account.api_channels.create!(
|
||||
additional_attributes: {
|
||||
source_provider: @provider,
|
||||
source_bucket: bucket[:key],
|
||||
import_placeholder: true,
|
||||
agent_reply_time_window: AGENT_REPLY_TIME_WINDOW_HOURS
|
||||
}
|
||||
)
|
||||
end
|
||||
|
||||
def placeholder_inbox_attributes(channel, bucket, timestamp)
|
||||
{
|
||||
account_id: @account.id,
|
||||
channel_id: channel.id,
|
||||
channel_type: channel.class.name,
|
||||
name: "#{@provider.titleize} Import - #{bucket[:name]}",
|
||||
enable_auto_assignment: false,
|
||||
allow_messages_after_resolved: false,
|
||||
created_at: timestamp,
|
||||
updated_at: timestamp
|
||||
}
|
||||
end
|
||||
|
||||
def create_default_working_hours(inbox)
|
||||
OutOfOffisable::DEFAULT_WORKING_HOURS.each { |attributes| inbox.working_hours.create!(attributes) }
|
||||
end
|
||||
end
|
||||
56
app/services/data_imports/restart_service.rb
Normal file
56
app/services/data_imports/restart_service.rb
Normal file
@@ -0,0 +1,56 @@
|
||||
class DataImports::RestartService
|
||||
attr_reader :data_import
|
||||
|
||||
def initialize(account:, data_import:)
|
||||
@account = account
|
||||
@data_import = data_import
|
||||
end
|
||||
|
||||
def perform
|
||||
@account.with_lock do
|
||||
@data_import.reload
|
||||
next :render_show unless @data_import.integration_import?
|
||||
next :render_show unless @data_import.restartable?
|
||||
|
||||
if (active_import = find_active_import)
|
||||
@data_import = active_import
|
||||
next :render_show
|
||||
end
|
||||
|
||||
next :access_token_missing if @data_import.access_token.blank?
|
||||
|
||||
@data_import.assign_active_import_run_id
|
||||
retained_skip_logs = @data_import.import_errors.where("details ->> 'kind' = ?", 'skipped')
|
||||
@data_import.import_errors.where.not(id: retained_skip_logs.select(:id)).delete_all
|
||||
@data_import.update!(restart_attributes(retained_skip_logs))
|
||||
:enqueue
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def find_active_import
|
||||
@account.data_imports.active_integrations.first
|
||||
end
|
||||
|
||||
def restart_attributes(retained_skip_logs)
|
||||
{
|
||||
status: :pending,
|
||||
abandoned_at: nil,
|
||||
completed_at: nil,
|
||||
last_error_at: nil,
|
||||
started_at: nil,
|
||||
stats: restart_stats(retained_skip_logs)
|
||||
}
|
||||
end
|
||||
|
||||
def restart_stats(retained_skip_logs)
|
||||
@data_import.stats.to_h.deep_dup.tap do |stats|
|
||||
%w[contact conversation message].each do |object_type|
|
||||
stats["#{object_type}s"] ||= {}
|
||||
stats["#{object_type}s"]['skipped'] = retained_skip_logs.where(source_object_type: object_type).count
|
||||
end
|
||||
stats['errors'] = { 'count' => 0 }
|
||||
end
|
||||
end
|
||||
end
|
||||
28
app/services/data_imports/retry_service.rb
Normal file
28
app/services/data_imports/retry_service.rb
Normal file
@@ -0,0 +1,28 @@
|
||||
class DataImports::RetryService
|
||||
attr_reader :data_import
|
||||
|
||||
def initialize(account:, data_import:)
|
||||
@account = account
|
||||
@data_import = data_import
|
||||
end
|
||||
|
||||
def perform
|
||||
@account.with_lock do
|
||||
@data_import.with_lock do
|
||||
next :not_stalled unless @data_import.stalled?
|
||||
next :active_import_exists if another_active_import?
|
||||
next :access_token_missing if @data_import.access_token.blank?
|
||||
|
||||
@data_import.assign_active_import_run_id
|
||||
@data_import.update!(status: :pending)
|
||||
:enqueue
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def another_active_import?
|
||||
@account.data_imports.active_integrations.where.not(id: @data_import.id).exists?
|
||||
end
|
||||
end
|
||||
21
app/services/data_imports/source.rb
Normal file
21
app/services/data_imports/source.rb
Normal file
@@ -0,0 +1,21 @@
|
||||
class DataImports::Source
|
||||
PROVIDERS = {
|
||||
'freshdesk' => 'DataImports::Freshdesk::Source',
|
||||
'intercom' => 'DataImports::Intercom::Source'
|
||||
}.freeze
|
||||
|
||||
def self.for(data_import)
|
||||
source_class(data_import.source_provider).new(
|
||||
access_token: data_import.access_token,
|
||||
source_metadata: data_import.source_metadata
|
||||
)
|
||||
end
|
||||
|
||||
def self.source_class(provider)
|
||||
PROVIDERS.fetch(provider.to_s) { raise ArgumentError, 'Unsupported import source.' }.constantize
|
||||
end
|
||||
|
||||
def self.supported?(provider)
|
||||
PROVIDERS.key?(provider.to_s)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,13 @@
|
||||
class CustomExceptions::DataImport::FreshdeskTicketLimitError < CustomExceptions::Base
|
||||
MESSAGE = 'Freshdesk limits ticket listing to 300 pages. This import stopped after 30,000 tickets because the API cannot ' \
|
||||
'confirm that the ticket history is complete. A Freshdesk admin Account Export is required for any remaining history, ' \
|
||||
'but this importer does not support account exports yet.'.freeze
|
||||
|
||||
def initialize
|
||||
super({})
|
||||
end
|
||||
|
||||
def message
|
||||
MESSAGE
|
||||
end
|
||||
end
|
||||
@@ -12,5 +12,15 @@ FactoryBot.define do
|
||||
access_token { 'intercom-token' }
|
||||
import_file { nil }
|
||||
end
|
||||
|
||||
trait :freshdesk do
|
||||
data_type { 'freshdesk' }
|
||||
source_type { 'api' }
|
||||
source_provider { 'freshdesk' }
|
||||
import_types { %w[contacts conversations] }
|
||||
access_token { 'freshdesk-api-key' }
|
||||
source_metadata { { domain: 'acme.freshdesk.com' } }
|
||||
import_file { nil }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
22
spec/fixtures/data_import/freshdesk/README.md
vendored
Normal file
22
spec/fixtures/data_import/freshdesk/README.md
vendored
Normal file
@@ -0,0 +1,22 @@
|
||||
# Freshdesk fixture provenance
|
||||
|
||||
These fixtures contain no production credentials, signed attachment URLs, or
|
||||
personal data. Their field shapes were derived from the Freshdesk API v2
|
||||
documentation, sanitized examples in these public repositories, and a
|
||||
sanitized Freshdesk trial-account Web Chat response captured during live API
|
||||
validation:
|
||||
|
||||
- [`Aaronontheweb/freshdesk-cli` ticket with conversations](https://github.com/Aaronontheweb/freshdesk-cli/blob/c87b74043a1662646378cd3385ec0db9af6fa28c/tests/TestData/FreshdeskResponses/ticket_with_conversations.json)
|
||||
- [`airbytehq/airbyte` Freshdesk expected records](https://github.com/airbytehq/airbyte/blob/0673d69418fe3874a6970d6113eec891db4a296b/airbyte-integrations/connectors/source-freshdesk/integration_tests/expected_records.jsonl)
|
||||
- [`pbrane/freshdesk-api-client` conversation fixture](https://github.com/pbrane/freshdesk-api-client/blob/c6685af0bf2da889b4a89ccb6cfeb53e65eb5575/src/main/resources/freshdesk/documents/tac-case-notes/tacCase99Conversations.json)
|
||||
- [`freshworks-developers/fw-attach` conversation event fixture](https://github.com/freshworks-developers/fw-attach/blob/45b7ea1ad6b9008b9edb73ecca349ee9d3b28a6d/server/test_data/support_ticket/onConversationCreate.json)
|
||||
|
||||
The fixture intentionally covers an incoming ticket description, a public
|
||||
agent reply, a private note with attachment metadata, and a later customer
|
||||
reply.
|
||||
|
||||
`web_chat_ticket_with_conversations.json` covers Freshdesk's current Web Chat
|
||||
ticket source (`15`), structured conversation bodies, the generated greeting,
|
||||
the initial customer message repeated in the ticket description, subsequent
|
||||
replies, and real attachment metadata keys. IDs, timestamps, addresses, names,
|
||||
content, and attachment URLs were replaced or removed.
|
||||
20
spec/fixtures/data_import/freshdesk/contact.json
vendored
Normal file
20
spec/fixtures/data_import/freshdesk/contact.json
vendored
Normal file
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"id": 1001,
|
||||
"active": true,
|
||||
"email": "customer@example.com",
|
||||
"name": "Customer Example",
|
||||
"mobile": "+15551234567",
|
||||
"phone": null,
|
||||
"created_at": "2024-01-01T09:00:00Z",
|
||||
"updated_at": "2024-01-04T12:30:00Z",
|
||||
"other_emails": [
|
||||
"billing@example.com"
|
||||
],
|
||||
"unique_external_id": "customer-1001",
|
||||
"custom_fields": {
|
||||
"support_plan": "growth"
|
||||
},
|
||||
"tags": [
|
||||
"priority"
|
||||
]
|
||||
}
|
||||
98
spec/fixtures/data_import/freshdesk/ticket_with_conversations.json
vendored
Normal file
98
spec/fixtures/data_import/freshdesk/ticket_with_conversations.json
vendored
Normal file
@@ -0,0 +1,98 @@
|
||||
{
|
||||
"ticket": {
|
||||
"id": 2001,
|
||||
"subject": "Unable to finish checkout",
|
||||
"description": "<p>The checkout button is not responding.</p>",
|
||||
"description_text": "The checkout button is not responding.",
|
||||
"status": 4,
|
||||
"priority": 2,
|
||||
"source": 1,
|
||||
"requester_id": 1001,
|
||||
"responder_id": 501,
|
||||
"group_id": 42,
|
||||
"company_id": 77,
|
||||
"tags": [
|
||||
"checkout"
|
||||
],
|
||||
"custom_fields": {
|
||||
"product_area": "payments"
|
||||
},
|
||||
"created_at": "2024-01-02T10:00:00Z",
|
||||
"updated_at": "2024-01-04T12:30:00Z",
|
||||
"attachments": [],
|
||||
"requester": {
|
||||
"id": 1001,
|
||||
"active": true,
|
||||
"email": "customer@example.com",
|
||||
"name": "Customer Example",
|
||||
"mobile": "+15551234567",
|
||||
"created_at": "2024-01-01T09:00:00Z",
|
||||
"updated_at": "2024-01-04T12:30:00Z",
|
||||
"unique_external_id": "customer-1001"
|
||||
}
|
||||
},
|
||||
"conversations": [
|
||||
{
|
||||
"id": 3003,
|
||||
"body": "<p>The issue still happens in a private window.</p>",
|
||||
"body_text": "The issue still happens in a private window.",
|
||||
"incoming": true,
|
||||
"private": false,
|
||||
"source": 0,
|
||||
"user_id": 1001,
|
||||
"from_email": "customer@example.com",
|
||||
"to_emails": [
|
||||
"support@example.com"
|
||||
],
|
||||
"cc_emails": [],
|
||||
"bcc_emails": [],
|
||||
"created_at": "2024-01-04T12:30:00Z",
|
||||
"updated_at": "2024-01-04T12:30:00Z",
|
||||
"attachments": [],
|
||||
"deleted": false
|
||||
},
|
||||
{
|
||||
"id": 3001,
|
||||
"body": "<p>Could you try this in a private window?</p>",
|
||||
"body_text": "Could you try this in a private window?",
|
||||
"incoming": false,
|
||||
"private": false,
|
||||
"source": 0,
|
||||
"user_id": 501,
|
||||
"from_email": "agent@example.com",
|
||||
"to_emails": [
|
||||
"customer@example.com"
|
||||
],
|
||||
"cc_emails": [],
|
||||
"bcc_emails": [],
|
||||
"created_at": "2024-01-03T08:00:00Z",
|
||||
"updated_at": "2024-01-03T08:00:00Z",
|
||||
"attachments": [],
|
||||
"deleted": false
|
||||
},
|
||||
{
|
||||
"id": 3002,
|
||||
"body": "<p>Payments team is investigating the browser logs.</p>",
|
||||
"body_text": "Payments team is investigating the browser logs.",
|
||||
"incoming": false,
|
||||
"private": true,
|
||||
"source": 2,
|
||||
"user_id": 501,
|
||||
"from_email": "agent@example.com",
|
||||
"to_emails": [],
|
||||
"cc_emails": [],
|
||||
"bcc_emails": [],
|
||||
"created_at": "2024-01-03T09:00:00Z",
|
||||
"updated_at": "2024-01-03T09:00:00Z",
|
||||
"attachments": [
|
||||
{
|
||||
"id": 4001,
|
||||
"name": "browser-log.txt",
|
||||
"content_type": "text/plain",
|
||||
"size": 128
|
||||
}
|
||||
],
|
||||
"deleted": false
|
||||
}
|
||||
]
|
||||
}
|
||||
183
spec/fixtures/data_import/freshdesk/web_chat_ticket_with_conversations.json
vendored
Normal file
183
spec/fixtures/data_import/freshdesk/web_chat_ticket_with_conversations.json
vendored
Normal file
@@ -0,0 +1,183 @@
|
||||
{
|
||||
"ticket": {
|
||||
"id": 2101,
|
||||
"subject": "FD-IMPORT-WEBCHAT-002 — Brew temperature drops",
|
||||
"description": "<div>Ticket Description: FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.</div>",
|
||||
"description_text": "Ticket Description: FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.",
|
||||
"status": 2,
|
||||
"priority": 1,
|
||||
"source": 15,
|
||||
"source_info": 1,
|
||||
"requester_id": 1101,
|
||||
"responder_id": 510,
|
||||
"group_id": null,
|
||||
"company_id": null,
|
||||
"tags": [],
|
||||
"custom_fields": {},
|
||||
"created_at": "2026-07-31T08:00:00Z",
|
||||
"updated_at": "2026-07-31T08:10:00Z",
|
||||
"attachments": [],
|
||||
"requester": {
|
||||
"id": 1101,
|
||||
"email": null,
|
||||
"name": "",
|
||||
"mobile": null,
|
||||
"phone": null
|
||||
}
|
||||
},
|
||||
"conversations": [
|
||||
{
|
||||
"id": 3101,
|
||||
"body": "<div>Hello! How can we help you today?</div>",
|
||||
"body_text": "Hello! How can we help you today?",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "Hello! How can we help you today?"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": false,
|
||||
"private": false,
|
||||
"source": 15,
|
||||
"category": 5,
|
||||
"user_id": null,
|
||||
"from_email": null,
|
||||
"created_at": "2026-07-31T08:00:00Z",
|
||||
"updated_at": "2026-07-31T08:00:00Z",
|
||||
"attachments": []
|
||||
},
|
||||
{
|
||||
"id": 3102,
|
||||
"body": "<div>FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.</div>",
|
||||
"body_text": "FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups.",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "FD-IMPORT-WEBCHAT-002 — The brew temperature drops after three cups."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": true,
|
||||
"private": false,
|
||||
"source": 20,
|
||||
"category": 1,
|
||||
"user_id": 1101,
|
||||
"from_email": null,
|
||||
"created_at": "2026-07-31T08:00:01Z",
|
||||
"updated_at": "2026-07-31T08:00:01Z",
|
||||
"attachments": []
|
||||
},
|
||||
{
|
||||
"id": 3103,
|
||||
"body": "<div>The display reads 82°C instead of 93°C.</div>",
|
||||
"body_text": "The display reads 82°C instead of 93°C.",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "The display reads 82°C instead of 93°C."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": true,
|
||||
"private": false,
|
||||
"source": 20,
|
||||
"category": 1,
|
||||
"user_id": 1101,
|
||||
"from_email": null,
|
||||
"created_at": "2026-07-31T08:01:00Z",
|
||||
"updated_at": "2026-07-31T08:01:00Z",
|
||||
"attachments": []
|
||||
},
|
||||
{
|
||||
"id": 3104,
|
||||
"body": "<div>Please run one rinse cycle, then check the next brew.</div>",
|
||||
"body_text": "Please run one rinse cycle, then check the next brew.",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "Please run one rinse cycle, then check the next brew."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": false,
|
||||
"private": false,
|
||||
"source": 20,
|
||||
"category": 3,
|
||||
"user_id": 510,
|
||||
"from_email": null,
|
||||
"created_at": "2026-07-31T08:03:00Z",
|
||||
"updated_at": "2026-07-31T08:03:00Z",
|
||||
"attachments": []
|
||||
},
|
||||
{
|
||||
"id": 3105,
|
||||
"body": "<div>The next brew reached 89°C.</div>",
|
||||
"body_text": "The next brew reached 89°C.",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "The next brew reached 89°C."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": true,
|
||||
"private": false,
|
||||
"source": 20,
|
||||
"category": 1,
|
||||
"user_id": 1101,
|
||||
"from_email": null,
|
||||
"created_at": "2026-07-31T08:05:00Z",
|
||||
"updated_at": "2026-07-31T08:05:00Z",
|
||||
"attachments": []
|
||||
},
|
||||
{
|
||||
"id": 3106,
|
||||
"body": "<div>Attached is a harmless text diagnostic.</div>",
|
||||
"body_text": "Attached is a harmless text diagnostic.",
|
||||
"structured_body": {
|
||||
"body_contents": [
|
||||
{
|
||||
"type": "text",
|
||||
"data": {
|
||||
"content": "Attached is a harmless text diagnostic."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"incoming": false,
|
||||
"private": false,
|
||||
"source": 20,
|
||||
"category": 3,
|
||||
"user_id": 510,
|
||||
"from_email": "support@example.com",
|
||||
"created_at": "2026-07-31T08:10:00Z",
|
||||
"updated_at": "2026-07-31T08:10:00Z",
|
||||
"attachments": [
|
||||
{
|
||||
"id": 4101,
|
||||
"content_type": "text/plain",
|
||||
"size": 188,
|
||||
"name": "freshdesk-validation-diagnostic.txt",
|
||||
"created_at": "2026-07-31T08:10:00Z",
|
||||
"updated_at": "2026-07-31T08:10:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
96
spec/jobs/data_imports/freshdesk/import_jobs_spec.rb
Normal file
96
spec/jobs/data_imports/freshdesk/import_jobs_spec.rb
Normal file
@@ -0,0 +1,96 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::ImportJob do
|
||||
let(:account) { create(:account) }
|
||||
let(:data_import) { create(:data_import, :freshdesk, account: account) }
|
||||
let(:importer) { instance_double(DataImports::Freshdesk::Importer) }
|
||||
let(:run_id) { 'freshdesk-run-1' }
|
||||
|
||||
before do
|
||||
account.enable_features!('data_import')
|
||||
data_import.update!(source_metadata: data_import.source_metadata.merge(DataImport::ACTIVE_IMPORT_RUN_ID_KEY => run_id))
|
||||
allow(DataImports::Freshdesk::Importer).to receive(:new).with(data_import: data_import, run_id: run_id).and_return(importer)
|
||||
end
|
||||
|
||||
describe DataImports::Freshdesk::BaseJob do
|
||||
it 'checks rate limit retry before the generic client retry' do
|
||||
handlers = described_class.rescue_handlers.map(&:first)
|
||||
|
||||
expect(handlers.index('DataImports::Freshdesk::Client::RateLimitError')).to be > handlers.index('DataImports::Freshdesk::Client::Error')
|
||||
end
|
||||
|
||||
it 'schedules a rate-limit retry using the Freshdesk Retry-After header' do
|
||||
error = DataImports::Freshdesk::Client::RateLimitError.new('Rate limited', retry_after: '42')
|
||||
allow(importer).to receive(:start!).and_raise(error)
|
||||
|
||||
travel_to(Time.zone.parse('2026-07-30 12:00:00 UTC')) do
|
||||
expect do
|
||||
DataImports::Freshdesk::ImportJob.perform_now(data_import, run_id)
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).at(42.seconds.from_now)
|
||||
end
|
||||
end
|
||||
|
||||
it 'discards a terminal ticket-limit error after recording the failed import' do
|
||||
error = CustomExceptions::DataImport::FreshdeskTicketLimitError.new
|
||||
allow(importer).to receive_messages(conversations_completed?: false, fail!: true)
|
||||
allow(importer).to receive(:import_conversations_page).with(starting_after: nil).and_raise(error)
|
||||
|
||||
expect do
|
||||
DataImports::Freshdesk::ConversationsPageJob.perform_now(data_import, nil, run_id)
|
||||
end.not_to have_enqueued_job
|
||||
|
||||
expect(importer).to have_received(:fail!).with(error)
|
||||
end
|
||||
end
|
||||
|
||||
describe DataImports::Freshdesk::ImportJob do
|
||||
it 'starts the import and enqueues the first contacts page' do
|
||||
allow(importer).to receive_messages(start!: true, import_contacts?: true, contacts_completed?: false, cursor_for: 2)
|
||||
|
||||
expect do
|
||||
described_class.perform_now(data_import, run_id)
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ContactsPageJob).with(data_import, 2, run_id).on_queue('low')
|
||||
|
||||
expect(importer).to have_received(:start!)
|
||||
end
|
||||
|
||||
it 'skips stale import jobs from an earlier run' do
|
||||
data_import.update!(source_metadata: data_import.source_metadata.merge(DataImport::ACTIVE_IMPORT_RUN_ID_KEY => 'new-run'))
|
||||
|
||||
expect(DataImports::Freshdesk::Importer).not_to receive(:new)
|
||||
|
||||
described_class.perform_now(data_import, 'old-run')
|
||||
end
|
||||
end
|
||||
|
||||
describe DataImports::Freshdesk::ContactsPageJob do
|
||||
it 'hands off to tickets after the final contacts page' do
|
||||
result = DataImports::Freshdesk::Importer::PageResult.new(next_cursor: nil)
|
||||
allow(importer).to receive_messages(
|
||||
contacts_completed?: false,
|
||||
import_conversations?: true,
|
||||
conversations_completed?: false
|
||||
)
|
||||
allow(importer).to receive(:import_contacts_page).with(starting_after: nil).and_return(result)
|
||||
allow(importer).to receive(:cursor_for).with('conversations').and_return(1)
|
||||
|
||||
expect do
|
||||
described_class.perform_now(data_import, nil, run_id)
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ConversationsPageJob).with(data_import, 1, run_id)
|
||||
end
|
||||
end
|
||||
|
||||
describe DataImports::Freshdesk::ConversationsPageJob do
|
||||
it 'finishes after the final tickets page' do
|
||||
result = DataImports::Freshdesk::Importer::PageResult.new(next_cursor: nil)
|
||||
allow(importer).to receive_messages(conversations_completed?: false, finish!: true)
|
||||
allow(importer).to receive(:import_conversations_page).with(starting_after: nil).and_return(result)
|
||||
|
||||
expect do
|
||||
described_class.perform_now(data_import, nil, run_id)
|
||||
end.not_to have_enqueued_job
|
||||
|
||||
expect(importer).to have_received(:finish!)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -9,6 +9,13 @@ RSpec.describe DataImport do
|
||||
it 'returns false for invalid data type' do
|
||||
expect(build(:data_import, data_type: 'Xyc').valid?).to be false
|
||||
end
|
||||
|
||||
it 'requires an integration provider to match its data type' do
|
||||
data_import = build(:data_import, :freshdesk, source_provider: 'intercom')
|
||||
|
||||
expect(data_import).not_to be_valid
|
||||
expect(data_import.errors[:source_provider]).to include('must match the integration data type')
|
||||
end
|
||||
end
|
||||
|
||||
describe 'access token encryption' do
|
||||
@@ -80,6 +87,13 @@ RSpec.describe DataImport do
|
||||
end
|
||||
end
|
||||
|
||||
it 'identifies stalled Freshdesk imports' do
|
||||
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :processing)
|
||||
freshdesk_import.update!(updated_at: 15.minutes.ago)
|
||||
|
||||
expect(freshdesk_import.reload).to be_stalled
|
||||
end
|
||||
|
||||
it 'does not identify recent or terminal Intercom imports as stalled', :aggregate_failures do
|
||||
recent_import = create(:data_import, :intercom, account: account, status: :processing)
|
||||
completed_import = create(:data_import, :intercom, account: account, status: :completed)
|
||||
|
||||
@@ -4,10 +4,12 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
let(:account) { create(:account) }
|
||||
let(:admin) { create(:user, account: account, role: :administrator) }
|
||||
let(:validator) { instance_double(DataImports::Intercom::CredentialsValidator, perform: { 'contacts' => 12, 'conversations' => 8 }) }
|
||||
let(:freshdesk_validator) { instance_double(DataImports::Freshdesk::CredentialsValidator, perform: {}) }
|
||||
|
||||
before do
|
||||
account.enable_features!('data_import')
|
||||
allow(DataImports::Intercom::CredentialsValidator).to receive(:new).and_return(validator)
|
||||
allow(DataImports::Freshdesk::CredentialsValidator).to receive(:new).and_return(freshdesk_validator)
|
||||
end
|
||||
|
||||
describe 'POST /api/v1/accounts/:account_id/data_imports/validate_source' do
|
||||
@@ -37,6 +39,49 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
'message' => 'We could not validate this Intercom access key. Check the key and its permissions.'
|
||||
)
|
||||
end
|
||||
|
||||
it 'validates a Freshdesk domain and API key through its source adapter', :aggregate_failures do
|
||||
post validate_source_api_v1_account_data_imports_url(account_id: account.id),
|
||||
params: {
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'acme.freshdesk.com',
|
||||
access_token: 'freshdesk-api-key',
|
||||
import_types: %w[contacts conversations]
|
||||
},
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(response.parsed_body).to eq('valid' => true, 'totals' => {})
|
||||
expect(DataImports::Freshdesk::CredentialsValidator).to have_received(:new).with(
|
||||
domain: 'acme.freshdesk.com',
|
||||
api_key: 'freshdesk-api-key',
|
||||
import_types: %w[contacts conversations]
|
||||
)
|
||||
end
|
||||
|
||||
it 'returns a safe Freshdesk authentication error' do
|
||||
allow(freshdesk_validator).to receive(:perform).and_raise(
|
||||
DataImports::Freshdesk::Client::AuthenticationError,
|
||||
'provider response'
|
||||
)
|
||||
|
||||
post validate_source_api_v1_account_data_imports_url(account_id: account.id),
|
||||
params: {
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'acme.freshdesk.com',
|
||||
access_token: 'invalid',
|
||||
import_types: %w[contacts]
|
||||
},
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(response.parsed_body).to eq(
|
||||
'valid' => false,
|
||||
'message' => 'We could not validate this Freshdesk API key. Check the key and its permissions.'
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
describe 'POST /api/v1/accounts/:account_id/data_imports' do
|
||||
@@ -87,6 +132,36 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
expect(response.parsed_body).not_to have_key('access_token')
|
||||
end
|
||||
|
||||
it 'creates and enqueues a Freshdesk import', :aggregate_failures do
|
||||
expect do
|
||||
post api_v1_account_data_imports_url(account_id: account.id),
|
||||
params: {
|
||||
name: 'Freshdesk migration',
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'https://ACME.freshdesk.com/support/home',
|
||||
access_token: 'freshdesk-api-key',
|
||||
import_types: %w[contacts conversations]
|
||||
},
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob)
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
data_import = account.data_imports.last
|
||||
expect(data_import).to have_attributes(
|
||||
name: 'Freshdesk migration',
|
||||
data_type: 'freshdesk',
|
||||
source_type: 'api',
|
||||
source_provider: 'freshdesk',
|
||||
initiated_by_id: admin.id
|
||||
)
|
||||
expect(data_import.access_token).to eq('freshdesk-api-key')
|
||||
expect(data_import.source_metadata).to include('domain' => 'acme.freshdesk.com')
|
||||
expect(data_import.active_import_run_id).to be_present
|
||||
expect(response.parsed_body['source_provider']).to eq('freshdesk')
|
||||
expect(response.parsed_body).not_to have_key('access_token')
|
||||
end
|
||||
|
||||
it 'rejects creation while another Intercom import is active' do
|
||||
active_import = create(
|
||||
:data_import, :intercom,
|
||||
@@ -110,6 +185,26 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
expect(active_import.reload).to be_processing
|
||||
end
|
||||
|
||||
it 'rejects a Freshdesk import while another integration import is active' do
|
||||
create(:data_import, :intercom, account: account, status: :processing)
|
||||
|
||||
expect do
|
||||
post api_v1_account_data_imports_url(account_id: account.id),
|
||||
params: {
|
||||
source_provider: 'freshdesk',
|
||||
domain: 'acme.freshdesk.com',
|
||||
access_token: 'freshdesk-api-key',
|
||||
import_types: %w[contacts conversations]
|
||||
},
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
end.not_to have_enqueued_job(DataImports::Freshdesk::ImportJob)
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(response.parsed_body['message']).to eq('Another data import is already in progress.')
|
||||
expect(account.data_imports.where(source_provider: 'freshdesk')).to be_empty
|
||||
end
|
||||
|
||||
it 'rejects unsupported import types instead of silently importing everything' do
|
||||
allow(validator).to receive(:perform).and_raise(ArgumentError, 'Unsupported import types: companies')
|
||||
|
||||
@@ -207,6 +302,20 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
expect(response.parsed_body['message']).to eq('The Intercom access key for this import is unavailable.')
|
||||
expect(data_import.reload).to be_abandoned
|
||||
end
|
||||
|
||||
it 'restarts an abandoned Freshdesk import with the Freshdesk job' do
|
||||
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :abandoned, abandoned_at: 1.hour.ago)
|
||||
|
||||
expect do
|
||||
post start_api_v1_account_data_import_url(account_id: account.id, id: freshdesk_import.id),
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).with(freshdesk_import, a_kind_of(String))
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(freshdesk_import.reload).to be_pending
|
||||
expect(freshdesk_import.active_import_run_id).to be_present
|
||||
end
|
||||
end
|
||||
|
||||
describe 'POST /api/v1/accounts/:account_id/data_imports/:id/retry' do
|
||||
@@ -264,6 +373,20 @@ RSpec.describe 'Data Imports API', type: :request do
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(response.parsed_body['message']).to eq('Another Intercom import is already in progress.')
|
||||
end
|
||||
|
||||
it 'resumes a stalled Freshdesk import with the Freshdesk job' do
|
||||
freshdesk_import = create(:data_import, :freshdesk, account: account, status: :processing, started_at: 2.hours.ago)
|
||||
freshdesk_import.update!(updated_at: 16.minutes.ago)
|
||||
|
||||
expect do
|
||||
post retry_api_v1_account_data_import_url(account_id: account.id, id: freshdesk_import.id),
|
||||
headers: admin.create_new_auth_token,
|
||||
as: :json
|
||||
end.to have_enqueued_job(DataImports::Freshdesk::ImportJob).with(freshdesk_import, a_kind_of(String))
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(freshdesk_import.reload).to be_pending
|
||||
end
|
||||
end
|
||||
|
||||
describe 'POST /api/v1/accounts/:account_id/data_imports/:id/abandon' do
|
||||
|
||||
90
spec/services/data_imports/freshdesk/client_spec.rb
Normal file
90
spec/services/data_imports/freshdesk/client_spec.rb
Normal file
@@ -0,0 +1,90 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::Client do
|
||||
describe '.normalize_domain' do
|
||||
it 'accepts a Freshdesk subdomain, hostname, or URL', :aggregate_failures do
|
||||
expect(described_class.normalize_domain('acme')).to eq('acme.freshdesk.com')
|
||||
expect(described_class.normalize_domain('ACME.FRESHDESK.COM')).to eq('acme.freshdesk.com')
|
||||
expect(described_class.normalize_domain('https://acme.freshdesk.com/support/home')).to eq('acme.freshdesk.com')
|
||||
end
|
||||
|
||||
it 'rejects hosts outside Freshdesk' do
|
||||
expect do
|
||||
described_class.normalize_domain('acme.example.com')
|
||||
end.to raise_error(ArgumentError, 'Enter a valid Freshdesk domain, such as acme.freshdesk.com.')
|
||||
end
|
||||
end
|
||||
|
||||
describe '#list_tickets' do
|
||||
it 'uses API key basic authentication, requests full history, and follows the next page link', :aggregate_failures do
|
||||
response = instance_double(
|
||||
HTTParty::Response,
|
||||
success?: true,
|
||||
code: 200,
|
||||
parsed_response: [{ 'id' => 2001 }],
|
||||
headers: {
|
||||
'link' => '<https://acme.freshdesk.com/api/v2/tickets?page=3&per_page=100>; rel="next"'
|
||||
}
|
||||
)
|
||||
allow(HTTParty).to receive(:get).and_return(response)
|
||||
|
||||
page = described_class.new(domain: 'acme', api_key: 'secret').list_tickets(page: 2)
|
||||
|
||||
expect(HTTParty).to have_received(:get).with(
|
||||
'https://acme.freshdesk.com/api/v2/tickets',
|
||||
query: {
|
||||
page: 2,
|
||||
per_page: 100,
|
||||
updated_since: DataImports::Freshdesk::Client::EARLIEST_TICKET_TIMESTAMP
|
||||
},
|
||||
basic_auth: { username: 'secret', password: 'X' },
|
||||
headers: { 'Accept' => 'application/json', 'Content-Type' => 'application/json' },
|
||||
timeout: 30
|
||||
)
|
||||
expect(page.data).to eq([{ 'id' => 2001 }])
|
||||
expect(page.next_page).to eq(3)
|
||||
end
|
||||
|
||||
it 'accepts only forward pages from relative or same-domain links', :aggregate_failures do
|
||||
client = described_class.new(domain: 'acme', api_key: 'secret')
|
||||
|
||||
expect(client.send(:next_page, '</api/v2/tickets?page=3>; rel="next"', current_page: 2)).to eq(3)
|
||||
expect(client.send(:next_page, '<https://acme.freshdesk.com/api/v2/tickets?page=4>; rel="next"', current_page: 3)).to eq(4)
|
||||
expect(client.send(:next_page, '<https://other.freshdesk.com/api/v2/tickets?page=4>; rel="next"', current_page: 3)).to be_nil
|
||||
expect(client.send(:next_page, '</api/v2/tickets?page=abc>; rel="next"', current_page: 2)).to be_nil
|
||||
expect(client.send(:next_page, '</api/v2/tickets?page=0>; rel="next"', current_page: 2)).to be_nil
|
||||
expect(client.send(:next_page, '</api/v2/tickets?page=2>; rel="next"', current_page: 2)).to be_nil
|
||||
expect(client.send(:next_page, '</api/v2/tickets?page=1>; rel="next"', current_page: 2)).to be_nil
|
||||
expect(client.send(:next_page, '<%%%>; rel="next"', current_page: 2)).to be_nil
|
||||
end
|
||||
end
|
||||
|
||||
describe 'errors' do
|
||||
it 'exposes Freshdesk retry timing on rate limits', :aggregate_failures do
|
||||
response = instance_double(
|
||||
HTTParty::Response,
|
||||
success?: false,
|
||||
code: 429,
|
||||
parsed_response: { 'description' => 'Rate limit exceeded' },
|
||||
headers: { 'retry-after' => '42' }
|
||||
)
|
||||
allow(HTTParty).to receive(:get).and_return(response)
|
||||
|
||||
expect { described_class.new(domain: 'acme', api_key: 'secret').list_contacts }.to raise_error do |error|
|
||||
expect(error.class.name).to eq('DataImports::Freshdesk::Client::RateLimitError')
|
||||
expect(error).to have_attributes(status: 429, retry_after: '42')
|
||||
expect(error.message).to eq('Rate limit exceeded')
|
||||
end
|
||||
end
|
||||
|
||||
it 'wraps transport failures in a retryable client error', :aggregate_failures do
|
||||
allow(HTTParty).to receive(:get).and_raise(SocketError, 'getaddrinfo failed')
|
||||
|
||||
expect { described_class.new(domain: 'acme', api_key: 'secret').list_contacts }.to raise_error do |error|
|
||||
expect(error.class.name).to eq('DataImports::Freshdesk::Client::Error')
|
||||
expect(error.message).to eq('Freshdesk API request failed before receiving a response: getaddrinfo failed')
|
||||
expect(error.body).to include(transport_error_class: 'SocketError')
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,39 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::CredentialsValidator do
|
||||
let(:client) { instance_double(DataImports::Freshdesk::Client) }
|
||||
|
||||
before do
|
||||
allow(DataImports::Freshdesk::Client).to receive(:new).with(domain: 'acme', api_key: 'secret').and_return(client)
|
||||
allow(client).to receive(:list_contacts)
|
||||
allow(client).to receive(:list_tickets)
|
||||
end
|
||||
|
||||
it 'validates only the selected Freshdesk resources and leaves totals undiscovered', :aggregate_failures do
|
||||
totals = described_class.new(domain: 'acme', api_key: ' secret ', import_types: %w[conversations]).perform
|
||||
|
||||
expect(totals).to be_empty
|
||||
expect(client).to have_received(:list_tickets).with(per_page: 1)
|
||||
expect(client).not_to have_received(:list_contacts)
|
||||
end
|
||||
|
||||
it 'validates contacts and tickets when both are selected', :aggregate_failures do
|
||||
described_class.new(domain: 'acme', api_key: 'secret', import_types: %w[contacts conversations]).perform
|
||||
|
||||
expect(client).to have_received(:list_contacts).with(per_page: 1)
|
||||
expect(client).to have_received(:list_tickets).with(per_page: 1)
|
||||
end
|
||||
|
||||
it 'rejects missing or unsupported input before calling Freshdesk', :aggregate_failures do
|
||||
expect do
|
||||
described_class.new(domain: '', api_key: 'secret', import_types: %w[contacts]).perform
|
||||
end.to raise_error(ArgumentError, 'Freshdesk domain is required.')
|
||||
expect do
|
||||
described_class.new(domain: 'acme', api_key: '', import_types: %w[contacts]).perform
|
||||
end.to raise_error(ArgumentError, 'Freshdesk API key is required.')
|
||||
expect do
|
||||
described_class.new(domain: 'acme', api_key: 'secret', import_types: %w[companies]).perform
|
||||
end.to raise_error(ArgumentError, 'Unsupported import types: companies')
|
||||
expect(DataImports::Freshdesk::Client).not_to have_received(:new)
|
||||
end
|
||||
end
|
||||
302
spec/services/data_imports/freshdesk/importer_spec.rb
Normal file
302
spec/services/data_imports/freshdesk/importer_spec.rb
Normal file
@@ -0,0 +1,302 @@
|
||||
require 'rails_helper'
|
||||
require 'timeout'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::Importer do
|
||||
let(:account) { create(:account) }
|
||||
let(:data_import) { create(:data_import, :freshdesk, account: account) }
|
||||
let(:client) { instance_double(DataImports::Freshdesk::Client) }
|
||||
let(:contact_payload) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/contact.json').read)
|
||||
end
|
||||
let(:ticket_fixture) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
|
||||
end
|
||||
|
||||
before do
|
||||
account.enable_features!('data_import')
|
||||
allow(DataImports::Freshdesk::Client).to receive(:new).with(
|
||||
domain: 'acme.freshdesk.com',
|
||||
api_key: 'freshdesk-api-key'
|
||||
).and_return(client)
|
||||
allow(client).to receive(:list_contacts).with(page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: [contact_payload], next_page: nil)
|
||||
)
|
||||
allow(client).to receive(:list_tickets).with(page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: [ticket_fixture.fetch('ticket')], next_page: nil)
|
||||
)
|
||||
allow(client).to receive(:retrieve_ticket).with('2001').and_return(ticket_fixture.fetch('ticket'))
|
||||
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: ticket_fixture.fetch('conversations'), next_page: nil)
|
||||
)
|
||||
end
|
||||
|
||||
it 'imports Freshdesk contacts, tickets, replies, and private notes through the shared importer', :aggregate_failures do
|
||||
described_class.new(data_import: data_import).perform
|
||||
|
||||
contact = account.contacts.find_by!(email: 'customer@example.com')
|
||||
expect(contact).to have_attributes(
|
||||
name: 'Customer Example',
|
||||
phone_number: '+15551234567',
|
||||
identifier: 'customer-1001'
|
||||
)
|
||||
expect(contact.custom_attributes).to include(
|
||||
'freshdesk_contact_id' => '1001',
|
||||
'freshdesk_unique_external_id' => 'customer-1001'
|
||||
)
|
||||
|
||||
inbox = account.inboxes.find_by!(name: 'Freshdesk Import - Email')
|
||||
expect(inbox.channel.additional_attributes).to include(
|
||||
'source_provider' => 'freshdesk',
|
||||
'source_bucket' => 'email',
|
||||
'import_placeholder' => true
|
||||
)
|
||||
|
||||
conversation = account.conversations.find_by!(identifier: 'freshdesk:2001')
|
||||
expect(conversation).to have_attributes(
|
||||
status: 'resolved',
|
||||
inbox_id: inbox.id,
|
||||
contact_id: contact.id
|
||||
)
|
||||
expect(conversation.custom_attributes).to include(
|
||||
'freshdesk_ticket_id' => '2001',
|
||||
'freshdesk_status' => 4,
|
||||
'freshdesk_priority' => 2
|
||||
)
|
||||
|
||||
messages = conversation.messages.order(:created_at, :id)
|
||||
expect(messages.pluck(:content)).to eq(
|
||||
[
|
||||
"Unable to finish checkout\n\nThe checkout button is not responding.",
|
||||
'Could you try this in a private window?',
|
||||
"Payments team is investigating the browser logs.\n\n[Freshdesk attachment skipped: 1]",
|
||||
'The issue still happens in a private window.'
|
||||
]
|
||||
)
|
||||
expect(messages.map(&:message_type)).to eq(%w[incoming outgoing outgoing incoming])
|
||||
expect(messages.pluck(:private)).to eq([false, false, true, false])
|
||||
expect(messages.third.additional_attributes.dig('source', 'conversation_source')).to eq(2)
|
||||
expect(messages.third.additional_attributes.dig('source', 'attachments', 0, 'name')).to eq('browser-log.txt')
|
||||
|
||||
expect(data_import.reload).to be_completed
|
||||
expect(data_import.stats).to include(
|
||||
'contacts' => include('imported' => 1, 'skipped' => 0),
|
||||
'conversations' => include('imported' => 1, 'skipped' => 0),
|
||||
'messages' => include('imported' => 4, 'skipped' => 0, 'total' => 4),
|
||||
'errors' => { 'count' => 0 }
|
||||
)
|
||||
expect(data_import.processed_records).to eq(6)
|
||||
expect(data_import.mappings.count).to eq(6)
|
||||
end
|
||||
|
||||
it 'preserves the contact that authored an incoming reply' do
|
||||
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
|
||||
conversations.first.merge!('user_id' => 1002, 'from_email' => 'cc@example.com')
|
||||
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: conversations, next_page: nil)
|
||||
)
|
||||
data_import.update!(import_types: ['conversations'])
|
||||
|
||||
described_class.new(data_import: data_import).perform
|
||||
|
||||
reply = account.conversations.find_by!(identifier: 'freshdesk:2001').messages.find_by!(content: 'The issue still happens in a private window.')
|
||||
expect(reply.sender).to have_attributes(email: 'cc@example.com')
|
||||
expect(reply.sender_id).not_to eq(reply.conversation.contact_id)
|
||||
end
|
||||
|
||||
it 'preserves newer progress when a delayed worker persists a stale snapshot', :aggregate_failures do
|
||||
data_import.update!(
|
||||
status: :processing,
|
||||
cursor: { 'contacts' => { 'starting_after' => 'cursor-1', 'completed' => false } },
|
||||
stats: {
|
||||
'contacts' => { 'total' => 4, 'imported' => 1, 'skipped' => 0 },
|
||||
'conversations' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'messages' => { 'imported' => 0, 'skipped' => 0 },
|
||||
'errors' => { 'count' => 0 }
|
||||
},
|
||||
processed_records: 1
|
||||
)
|
||||
data_import.items.create!(
|
||||
source_provider: 'freshdesk', source_object_type: 'contact', source_object_id: '1001', status: :imported
|
||||
)
|
||||
delayed_importer = described_class.new(data_import: DataImport.find(data_import.id))
|
||||
|
||||
data_import.items.create!(
|
||||
source_provider: 'freshdesk', source_object_type: 'contact', source_object_id: '1002', status: :imported
|
||||
)
|
||||
data_import.reload.update!(
|
||||
cursor: { 'contacts' => { 'starting_after' => 'cursor-2', 'completed' => false } },
|
||||
stats: data_import.stats.deep_merge('contacts' => { 'imported' => 2 }),
|
||||
processed_records: 2
|
||||
)
|
||||
|
||||
delayed_importer.send(:persist_stats)
|
||||
expect(delayed_importer.send(:update_cursor, 'contacts', 'cursor-1')).to eq('cursor-2')
|
||||
|
||||
expect(data_import.reload.cursor.dig('contacts', 'starting_after')).to eq('cursor-2')
|
||||
expect(data_import.stats.dig('contacts', 'imported')).to eq(2)
|
||||
expect(data_import.processed_records).to eq(2)
|
||||
end
|
||||
|
||||
it 'does not let an old import run overwrite its replacement cursor', :aggregate_failures do
|
||||
data_import.update!(
|
||||
status: :processing,
|
||||
source_metadata: data_import.source_metadata.merge('active_import_run_id' => 'run-1'),
|
||||
cursor: { 'conversations' => { 'starting_after' => { 'page' => 1, 'offset' => 0 }, 'completed' => false } }
|
||||
)
|
||||
delayed_importer = described_class.new(data_import: DataImport.find(data_import.id), run_id: 'run-1')
|
||||
replacement_cursor = { 'page' => 1, 'offset' => 3 }
|
||||
data_import.reload.update!(
|
||||
source_metadata: data_import.source_metadata.merge('active_import_run_id' => 'run-2'),
|
||||
cursor: { 'conversations' => { 'starting_after' => replacement_cursor, 'completed' => false } }
|
||||
)
|
||||
|
||||
result = delayed_importer.send(:update_cursor, 'conversations', { 'page' => 1, 'offset' => 1 })
|
||||
|
||||
expect(result).to eq(replacement_cursor)
|
||||
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to eq(replacement_cursor)
|
||||
expect(delayed_importer.send(:import_stopped?)).to be(true)
|
||||
end
|
||||
|
||||
it 'serializes overlapping workers importing the same conversation', :aggregate_failures do
|
||||
started_requests = Queue.new
|
||||
release_requests = Queue.new
|
||||
allow(client).to receive(:retrieve_ticket).with('2001') do
|
||||
started_requests << true
|
||||
release_requests.pop
|
||||
ticket_fixture.fetch('ticket')
|
||||
end
|
||||
importers = Array.new(2) { described_class.new(data_import: DataImport.find(data_import.id)) }
|
||||
errors = Concurrent::Array.new
|
||||
conversation_summary = { 'id' => '2001', 'source' => { 'type' => 'email' } }
|
||||
threads = importers.map do |importer|
|
||||
Thread.new do
|
||||
importer.send(:import_conversation_from_summary, conversation_summary)
|
||||
rescue StandardError => e
|
||||
errors << e
|
||||
end
|
||||
end
|
||||
|
||||
begin
|
||||
2.times { Timeout.timeout(5) { started_requests.pop } }
|
||||
ensure
|
||||
2.times { release_requests << true }
|
||||
threads.each { |thread| thread.join(10) }
|
||||
end
|
||||
|
||||
expect(threads).to all(satisfy { |thread| !thread.alive? })
|
||||
expect(errors).to be_empty
|
||||
expect(account.conversations.where(identifier: 'freshdesk:2001').count).to eq(1)
|
||||
expect(account.messages.count).to eq(4)
|
||||
expect(data_import.mappings.count).to eq(6)
|
||||
expect(data_import.items.find_by!(source_object_type: 'conversation', source_object_id: '2001')).to have_attributes(
|
||||
status: 'imported', attempt_count: 2
|
||||
)
|
||||
end
|
||||
|
||||
it 'recognizes a structured cursor after JSON persistence' do
|
||||
importer = described_class.new(data_import: data_import)
|
||||
|
||||
importer.send(:update_cursor, 'conversations', { page: 1, offset: 2 })
|
||||
data_import.reload
|
||||
importer.send(:update_cursor, 'conversations', nil)
|
||||
|
||||
expect(data_import.reload.cursor['conversations']).to include('starting_after' => nil, 'completed' => true)
|
||||
end
|
||||
|
||||
it 'resumes a rate-limited ticket chunk from the last persisted checkpoint', :aggregate_failures do
|
||||
tickets = Array.new(3) { |index| { 'id' => index + 2001, 'source' => 1 } }
|
||||
allow(client).to receive(:list_tickets).with(page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
|
||||
)
|
||||
importer = described_class.new(data_import: data_import)
|
||||
processed_ticket_ids = []
|
||||
rate_limited = true
|
||||
allow(importer).to receive(:import_conversation_from_summary) do |summary|
|
||||
processed_ticket_ids << summary['id']
|
||||
if summary['id'] == '2003' && rate_limited
|
||||
rate_limited = false
|
||||
raise DataImports::Freshdesk::Client::RateLimitError.new('Rate limited', retry_after: '42')
|
||||
end
|
||||
end
|
||||
|
||||
expect do
|
||||
importer.import_conversations_page(starting_after: { 'page' => 1, 'offset' => 0 })
|
||||
end.to raise_error(DataImports::Freshdesk::Client::RateLimitError)
|
||||
expect(processed_ticket_ids).to eq(%w[2001 2002 2003])
|
||||
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to include('page' => 1, 'offset' => 2)
|
||||
|
||||
processed_ticket_ids.clear
|
||||
result = importer.import_conversations_page(starting_after: { 'page' => 1, 'offset' => 0 })
|
||||
|
||||
expect(processed_ticket_ids).to eq(%w[2003])
|
||||
expect(result).to be_done
|
||||
expect(data_import.reload.cursor['conversations']).to include('starting_after' => nil, 'completed' => true)
|
||||
expect(client).to have_received(:list_tickets).once
|
||||
end
|
||||
|
||||
it 'stops a delayed worker when a newer per-ticket checkpoint is observed', :aggregate_failures do
|
||||
current_cursor = { 'page' => 1, 'offset' => 0, 'tickets' => [], 'next_page' => 2 }
|
||||
newer_cursor = current_cursor.merge('offset' => 3)
|
||||
summaries = Array.new(3) { |index| { 'id' => (index + 2001).to_s } }
|
||||
response = {
|
||||
'data' => summaries,
|
||||
'pages' => {
|
||||
'current' => { 'starting_after' => current_cursor },
|
||||
'next' => { 'starting_after' => { 'page' => 2, 'offset' => 0 } },
|
||||
'checkpoints' => Array.new(3) { |index| current_cursor.merge('offset' => index + 1) }
|
||||
}
|
||||
}
|
||||
importer = described_class.new(data_import: DataImport.find(data_import.id))
|
||||
allow(importer).to receive(:conversations_page).and_return(response)
|
||||
cursor_advanced = false
|
||||
allow(importer).to receive(:persist_stats).and_wrap_original do |method, *args|
|
||||
method.call(*args)
|
||||
next if cursor_advanced
|
||||
|
||||
cursor_advanced = true
|
||||
concurrent_import = DataImport.find(data_import.id)
|
||||
concurrent_import.update!(
|
||||
cursor: concurrent_import.cursor.to_h.deep_merge(
|
||||
'conversations' => { 'starting_after' => newer_cursor, 'completed' => false }
|
||||
)
|
||||
)
|
||||
end
|
||||
|
||||
result = importer.import_conversations_page(starting_after: current_cursor)
|
||||
|
||||
expect(account.conversations.where(identifier: 'freshdesk:2001').count).to eq(1)
|
||||
expect(client).not_to have_received(:retrieve_ticket).with('2002')
|
||||
expect(result.next_cursor).to eq(newer_cursor)
|
||||
expect(data_import.reload.cursor.dig('conversations', 'starting_after')).to eq(newer_cursor)
|
||||
end
|
||||
|
||||
it 'fails with an actionable error instead of completing at the REST ticket limit', :aggregate_failures do
|
||||
tickets = Array.new(100) { |index| { 'id' => index + 1, 'source' => 1 } }
|
||||
allow(client).to receive(:list_tickets).with(page: 300, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
|
||||
)
|
||||
importer = described_class.new(data_import: data_import)
|
||||
processed_ticket_ids = []
|
||||
allow(importer).to receive(:import_conversation_from_summary) { |summary| processed_ticket_ids << summary['id'] }
|
||||
limit_error = nil
|
||||
|
||||
expect { importer.import_conversations_page(starting_after: { 'page' => 300, 'offset' => 90 }) }.to raise_error do |error|
|
||||
expect(error.class.name).to eq('CustomExceptions::DataImport::FreshdeskTicketLimitError')
|
||||
limit_error = error
|
||||
end
|
||||
importer.fail!(limit_error)
|
||||
|
||||
expect(processed_ticket_ids).to eq((91..100).map(&:to_s))
|
||||
expect(data_import.reload).to be_failed
|
||||
expect(data_import.cursor.dig('conversations', 'starting_after')).to include('page' => 300, 'offset' => 99)
|
||||
expect(data_import.import_errors.last).to have_attributes(
|
||||
error_code: 'CustomExceptions::DataImport::FreshdeskTicketLimitError',
|
||||
message: limit_error.message
|
||||
)
|
||||
expect(data_import.import_errors.last.details).to include(
|
||||
'kind' => 'run_error',
|
||||
'source_provider' => 'freshdesk',
|
||||
'error_class' => 'CustomExceptions::DataImport::FreshdeskTicketLimitError'
|
||||
)
|
||||
end
|
||||
end
|
||||
106
spec/services/data_imports/freshdesk/normalizer_spec.rb
Normal file
106
spec/services/data_imports/freshdesk/normalizer_spec.rb
Normal file
@@ -0,0 +1,106 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::Normalizer do
|
||||
let(:contact_payload) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/contact.json').read)
|
||||
end
|
||||
let(:ticket_fixture) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
|
||||
end
|
||||
let(:web_chat_fixture) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/web_chat_ticket_with_conversations.json').read)
|
||||
end
|
||||
|
||||
it 'normalizes Freshdesk contacts into the shared importer contract', :aggregate_failures do
|
||||
contact = described_class.new.contact(contact_payload)
|
||||
|
||||
expect(contact).to include(
|
||||
'id' => '1001',
|
||||
'name' => 'Customer Example',
|
||||
'email' => 'customer@example.com',
|
||||
'phone' => '+15551234567',
|
||||
'external_id' => 'customer-1001',
|
||||
'other_emails' => ['billing@example.com'],
|
||||
'custom_fields' => { 'support_plan' => 'growth' },
|
||||
'tags' => ['priority']
|
||||
)
|
||||
expect(contact['created_at']).to eq(Time.zone.parse('2024-01-01T09:00:00Z').to_i)
|
||||
expect(contact['updated_at']).to eq(Time.zone.parse('2024-01-04T12:30:00Z').to_i)
|
||||
end
|
||||
|
||||
it 'normalizes ticket descriptions, replies, and private notes in source order', :aggregate_failures do
|
||||
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), ticket_fixture.fetch('conversations'))
|
||||
parts = ticket.dig('conversation_parts', 'conversation_parts')
|
||||
|
||||
expect(ticket).to include(
|
||||
'id' => '2001',
|
||||
'status' => 4,
|
||||
'priority' => 2,
|
||||
'requester_id' => 1001
|
||||
)
|
||||
expect(ticket.dig('source', 'type')).to eq('email')
|
||||
expect(ticket.dig('source', 'author', 'type')).to eq('contact')
|
||||
expect(ticket.dig('contacts', 'contacts', 0, 'email')).to eq('customer@example.com')
|
||||
expect(parts.pluck('id')).to eq(%w[3001 3002 3003])
|
||||
expect(parts.pluck('part_type')).to eq(%w[comment note comment])
|
||||
expect(parts.map { |part| part.dig('author', 'type') }).to eq(%w[admin admin contact])
|
||||
expect(parts.second).to include('private' => true)
|
||||
expect(parts.second['attachments'].first).to include('name' => 'browser-log.txt')
|
||||
end
|
||||
|
||||
it 'preserves provider order for messages with equal timestamps' do
|
||||
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
|
||||
conversations[0]['created_at'] = conversations[1]['created_at']
|
||||
|
||||
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), conversations)
|
||||
|
||||
expect(ticket.dig('conversation_parts', 'conversation_parts').pluck('id')).to eq(%w[3003 3001 3002])
|
||||
end
|
||||
|
||||
it 'includes incoming reply authors as conversation contacts' do
|
||||
conversations = ticket_fixture.fetch('conversations').map(&:deep_dup)
|
||||
conversations.first.merge!('user_id' => 1002, 'from_email' => 'cc@example.com')
|
||||
|
||||
ticket = described_class.new.ticket(ticket_fixture.fetch('ticket'), conversations)
|
||||
|
||||
expect(ticket.dig('contacts', 'contacts')).to include(
|
||||
include('id' => '1001', 'email' => 'customer@example.com'),
|
||||
include('id' => '1002', 'email' => 'cc@example.com')
|
||||
)
|
||||
expect(ticket.dig('conversation_parts', 'conversation_parts').last['author']).to include(
|
||||
'id' => '1002', 'email' => 'cc@example.com'
|
||||
)
|
||||
end
|
||||
|
||||
it 'marks outbound email ticket descriptions as outgoing source messages' do
|
||||
outbound_ticket = ticket_fixture.fetch('ticket').merge('source' => 10)
|
||||
|
||||
ticket = described_class.new.ticket(outbound_ticket, [])
|
||||
|
||||
expect(ticket.dig('source', 'type')).to eq('outbound_email')
|
||||
expect(ticket.dig('source', 'author', 'type')).to eq('admin')
|
||||
end
|
||||
|
||||
it 'uses Web Chat conversation events as the complete message history', :aggregate_failures do
|
||||
ticket = described_class.new.ticket(web_chat_fixture.fetch('ticket'), web_chat_fixture.fetch('conversations'))
|
||||
parts = ticket.dig('conversation_parts', 'conversation_parts')
|
||||
|
||||
expect(ticket).to include(
|
||||
'id' => '2101',
|
||||
'subject' => 'FD-IMPORT-WEBCHAT-002 — Brew temperature drops',
|
||||
'status' => 2,
|
||||
'priority' => 1
|
||||
)
|
||||
expect(ticket.dig('source', 'type')).to eq('web_chat')
|
||||
expect(ticket.fetch('source')).not_to include('subject', 'body')
|
||||
expect(DataImports::Freshdesk::MessageBatchBuilder.source_message_importable?(ticket.fetch('source'))).to be(false)
|
||||
expect(parts.pluck('id')).to eq(%w[3101 3102 3103 3104 3105 3106])
|
||||
expect(parts.map { |part| part.dig('author', 'type') }).to eq(%w[admin contact contact admin contact admin])
|
||||
expect(parts.count { |part| part['body'].include?('FD-IMPORT-WEBCHAT-002') }).to eq(1)
|
||||
expect(parts.last['attachments'].first).to include(
|
||||
'name' => 'freshdesk-validation-diagnostic.txt',
|
||||
'content_type' => 'text/plain',
|
||||
'size' => 188
|
||||
)
|
||||
end
|
||||
end
|
||||
45
spec/services/data_imports/freshdesk/source_bucket_spec.rb
Normal file
45
spec/services/data_imports/freshdesk/source_bucket_spec.rb
Normal file
@@ -0,0 +1,45 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::SourceBucket do
|
||||
describe '.source_type' do
|
||||
it 'maps documented and observed Freshdesk source identifiers', :aggregate_failures do
|
||||
expect(described_class.source_type(1)).to eq('email')
|
||||
expect(described_class.source_type(7)).to eq('chat')
|
||||
expect(described_class.source_type(8)).to eq('mobihelp')
|
||||
expect(described_class.source_type(11)).to eq('ecommerce')
|
||||
expect(described_class.source_type(12)).to eq('bot')
|
||||
expect(described_class.source_type(13)).to eq('whatsapp')
|
||||
expect(described_class.source_type(14)).to eq('chat_internal_task')
|
||||
expect(described_class.source_type(15)).to eq('web_chat')
|
||||
expect(described_class.source_type(16)).to eq('web_form')
|
||||
expect(described_class.source_type(17)).to eq('instagram_message')
|
||||
expect(described_class.source_type(18)).to eq('instagram_comment')
|
||||
expect(described_class.source_type(19)).to eq('facebook_message')
|
||||
expect(described_class.source_type(20)).to eq('facebook_comment')
|
||||
expect(described_class.source_type(21)).to eq('mobile_chat_sdk')
|
||||
expect(described_class.source_type(22)).to eq('sms')
|
||||
end
|
||||
|
||||
it 'uses unknown for an unsupported source identifier' do
|
||||
expect(described_class.source_type(999)).to eq('unknown')
|
||||
end
|
||||
end
|
||||
|
||||
describe '.for' do
|
||||
it 'groups equivalent sources and preserves distinct modern channels', :aggregate_failures do
|
||||
expect(described_class.for('email')).to eq({ key: 'email', name: 'Email' })
|
||||
expect(described_class.for('outbound_email')).to eq({ key: 'email', name: 'Email' })
|
||||
expect(described_class.for('feedback_widget')).to eq({ key: 'portal', name: 'Portal' })
|
||||
expect(described_class.for('whatsapp')).to eq({ key: 'whatsapp', name: 'WhatsApp' })
|
||||
expect(described_class.for('chat_internal_task')).to eq({ key: 'internal_task', name: 'Internal task' })
|
||||
expect(described_class.for('web_chat')).to eq({ key: 'chat', name: 'Chat' })
|
||||
expect(described_class.for('web_form')).to eq({ key: 'portal', name: 'Portal' })
|
||||
expect(described_class.for('instagram_message')).to eq({ key: 'instagram', name: 'Instagram' })
|
||||
expect(described_class.for('instagram_comment')).to eq({ key: 'instagram', name: 'Instagram' })
|
||||
expect(described_class.for('facebook_message')).to eq({ key: 'facebook', name: 'Facebook' })
|
||||
expect(described_class.for('facebook_comment')).to eq({ key: 'facebook', name: 'Facebook' })
|
||||
expect(described_class.for('mobile_chat_sdk')).to eq({ key: 'mobile', name: 'Mobile' })
|
||||
expect(described_class.for('sms')).to eq({ key: 'sms', name: 'SMS' })
|
||||
end
|
||||
end
|
||||
end
|
||||
128
spec/services/data_imports/freshdesk/source_spec.rb
Normal file
128
spec/services/data_imports/freshdesk/source_spec.rb
Normal file
@@ -0,0 +1,128 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Freshdesk::Source do
|
||||
let(:client) { instance_double(DataImports::Freshdesk::Client) }
|
||||
let(:ticket_fixture) do
|
||||
JSON.parse(Rails.root.join('spec/fixtures/data_import/freshdesk/ticket_with_conversations.json').read)
|
||||
end
|
||||
let(:source) do
|
||||
described_class.new(
|
||||
access_token: 'freshdesk-api-key',
|
||||
source_metadata: { domain: 'acme.freshdesk.com' }
|
||||
)
|
||||
end
|
||||
|
||||
before do
|
||||
allow(DataImports::Freshdesk::Client).to receive(:new).with(
|
||||
domain: 'acme.freshdesk.com',
|
||||
api_key: 'freshdesk-api-key'
|
||||
).and_return(client)
|
||||
end
|
||||
|
||||
describe '.source_metadata' do
|
||||
it 'stores only a normalized Freshdesk domain' do
|
||||
expect(described_class.source_metadata(domain: 'https://ACME.freshdesk.com/support/home')).to eq(
|
||||
domain: 'acme.freshdesk.com'
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
describe '#list_contacts' do
|
||||
it 'normalizes records and translates numeric pages into shared cursors' do
|
||||
allow(client).to receive(:list_contacts).with(page: 2, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(
|
||||
data: [{ 'id' => 1001, 'email' => 'customer@example.com' }],
|
||||
next_page: 3
|
||||
)
|
||||
)
|
||||
|
||||
response = source.list_contacts(starting_after: 2, per_page: 100)
|
||||
|
||||
expect(response.dig('data', 0)).to include('id' => '1001', 'email' => 'customer@example.com')
|
||||
expect(response.dig('pages', 'next', 'starting_after')).to eq(3)
|
||||
end
|
||||
end
|
||||
|
||||
describe '#list_conversations' do
|
||||
it 'returns a resumable ticket chunk with a cursor after every ticket', :aggregate_failures do
|
||||
tickets = Array.new(25) { |index| { 'id' => index + 1, 'source' => 1 } }
|
||||
allow(client).to receive(:list_tickets).with(page: 2, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: 3)
|
||||
)
|
||||
|
||||
response = source.list_conversations(
|
||||
starting_after: { 'page' => 2, 'offset' => 10 },
|
||||
per_page: 100
|
||||
)
|
||||
|
||||
expect(response['data'].pluck('id')).to eq((11..20).map(&:to_s))
|
||||
expect(response.dig('pages', 'checkpoints').first).to include('page' => 2, 'offset' => 11, 'next_page' => 3)
|
||||
expect(response.dig('pages', 'checkpoints').last).to include('page' => 2, 'offset' => 20, 'next_page' => 3)
|
||||
expect(response.dig('pages', 'next', 'starting_after')).to include('page' => 2, 'offset' => 20, 'next_page' => 3)
|
||||
end
|
||||
|
||||
it 'reuses the cached ticket page before advancing to the next Freshdesk page', :aggregate_failures do
|
||||
tickets = Array.new(25) { |index| { 'id' => index + 1, 'source' => 1 } }
|
||||
expect(client).to receive(:list_tickets).with(page: 2, per_page: 100).once.and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: 3)
|
||||
)
|
||||
|
||||
first_chunk = source.list_conversations(
|
||||
starting_after: { 'page' => 2, 'offset' => 10 },
|
||||
per_page: 100
|
||||
)
|
||||
response = source.list_conversations(starting_after: first_chunk.dig('pages', 'next', 'starting_after'), per_page: 100)
|
||||
|
||||
expect(response['data'].pluck('id')).to eq((21..25).map(&:to_s))
|
||||
expect(response.dig('pages', 'checkpoints').last).to eq('page' => 3, 'offset' => 0)
|
||||
expect(response.dig('pages', 'next', 'starting_after')).to eq('page' => 3, 'offset' => 0)
|
||||
end
|
||||
|
||||
it 'reports the REST ticket limit only after the final chunk of a full page 300', :aggregate_failures do
|
||||
tickets = Array.new(100) { |index| { 'id' => index + 1, 'source' => 1 } }
|
||||
expect(client).to receive(:list_tickets).with(page: 300, per_page: 100).once.and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
|
||||
)
|
||||
|
||||
first_chunk = source.list_conversations(starting_after: { 'page' => 300, 'offset' => 0 }, per_page: 100)
|
||||
final_cursor = first_chunk.dig('pages', 'current', 'starting_after').merge('offset' => 90)
|
||||
final_chunk = source.list_conversations(starting_after: final_cursor, per_page: 100)
|
||||
|
||||
expect(first_chunk.dig('pages', 'limit_reached')).to be(false)
|
||||
expect(final_chunk['data'].pluck('id')).to eq((91..100).map(&:to_s))
|
||||
expect(final_chunk.dig('pages', 'next')).to be_nil
|
||||
expect(final_chunk.dig('pages', 'limit_reached')).to be(true)
|
||||
end
|
||||
|
||||
it 'completes a partial page 300 without reporting the ticket limit' do
|
||||
tickets = Array.new(99) { |index| { 'id' => index + 1, 'source' => 1 } }
|
||||
allow(client).to receive(:list_tickets).with(page: 300, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: tickets, next_page: nil)
|
||||
)
|
||||
|
||||
response = source.list_conversations(starting_after: { 'page' => 300, 'offset' => 90 }, per_page: 100)
|
||||
|
||||
expect(response.dig('pages', 'next')).to be_nil
|
||||
expect(response.dig('pages', 'limit_reached')).to be(false)
|
||||
end
|
||||
end
|
||||
|
||||
describe '#retrieve_conversation' do
|
||||
it 'retrieves every conversation page and normalizes the ticket', :aggregate_failures do
|
||||
conversations = ticket_fixture.fetch('conversations')
|
||||
allow(client).to receive(:retrieve_ticket).with('2001').and_return(ticket_fixture.fetch('ticket'))
|
||||
allow(client).to receive(:list_conversations).with('2001', page: 1, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: conversations.first(2), next_page: 2)
|
||||
)
|
||||
allow(client).to receive(:list_conversations).with('2001', page: 2, per_page: 100).and_return(
|
||||
DataImports::Freshdesk::Client::Page.new(data: conversations.last(1), next_page: nil)
|
||||
)
|
||||
|
||||
ticket = source.retrieve_conversation('2001')
|
||||
|
||||
expect(ticket['id']).to eq('2001')
|
||||
expect(ticket.dig('conversation_parts', 'total_count')).to eq(3)
|
||||
expect(ticket.dig('conversation_parts', 'conversation_parts').pluck('id')).to eq(%w[3001 3002 3003])
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -29,5 +29,27 @@ RSpec.describe DataImports::Intercom::PlaceholderInboxBuilder do
|
||||
expect(second_inbox).to eq(first_inbox)
|
||||
expect(Inbox.where(account: account, channel_type: 'Channel::Api').count).to eq(1)
|
||||
end
|
||||
|
||||
it 'creates editable default working hours without running inbox creation callbacks' do
|
||||
inbox = described_class.new(account: account).inbox_for('email')
|
||||
|
||||
expect(inbox.working_hours.count).to eq(7)
|
||||
|
||||
inbox.update_working_hours(
|
||||
[
|
||||
{
|
||||
'day_of_week' => 1,
|
||||
'open_hour' => 10,
|
||||
'open_minutes' => 0,
|
||||
'close_hour' => 18,
|
||||
'close_minutes' => 0,
|
||||
'closed_all_day' => false,
|
||||
'open_all_day' => false
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
expect(inbox.working_hours.find_by(day_of_week: 1)).to have_attributes(open_hour: 10, close_hour: 18)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
26
spec/services/data_imports/source_spec.rb
Normal file
26
spec/services/data_imports/source_spec.rb
Normal file
@@ -0,0 +1,26 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe DataImports::Source do
|
||||
describe '.source_class' do
|
||||
it 'resolves each supported integration adapter', :aggregate_failures do
|
||||
expect(described_class.source_class('intercom')).to eq(DataImports::Intercom::Source)
|
||||
expect(described_class.source_class('freshdesk')).to eq(DataImports::Freshdesk::Source)
|
||||
expect(described_class.supported?('intercom')).to be(true)
|
||||
expect(described_class.supported?('freshdesk')).to be(true)
|
||||
end
|
||||
|
||||
it 'rejects unsupported providers' do
|
||||
expect { described_class.source_class('zendesk') }.to raise_error(ArgumentError, 'Unsupported import source.')
|
||||
end
|
||||
end
|
||||
|
||||
describe '.for' do
|
||||
it 'builds the configured adapter from persisted credentials' do
|
||||
data_import = build(:data_import, :freshdesk)
|
||||
|
||||
source = described_class.for(data_import)
|
||||
|
||||
expect(source).to have_attributes(provider: 'freshdesk', display_name: 'Freshdesk')
|
||||
end
|
||||
end
|
||||
end
|
||||
Reference in New Issue
Block a user