AI Analysis and Content Strategy fixes. Enhanced Strategy Routes refactoring.

This commit is contained in:
ajaysi
2026-01-10 19:32:50 +05:30
parent 0b63ae7fc1
commit 8193cdba67
298 changed files with 45678 additions and 10952 deletions

View File

@@ -1,335 +1,492 @@
# Research Component Integration Guide
## Overview
**Date**: 2025-01-29
**Status**: Updated for Intent-Driven Research Architecture
The modular Research component has been implemented as a standalone, testable wizard that can be integrated into the blog writer or used independently. This document outlines the architecture, usage, and integration steps.
---
## Architecture
## 📋 Overview
### Backend Strategy Pattern
The Research component is a standalone, intent-driven research system that can be integrated into any part of the application. This guide explains how to integrate and use the Research component.
The research service now supports multiple research modes through a strategy pattern:
**Key Features**:
- Intent-driven research (AI infers user goals)
- Standalone and reusable
- 3-step wizard interface
- Provider optimization (Exa → Tavily → Google)
- Research persona integration
- Google Trends integration
```python
# Research modes
- Basic: Quick keyword-focused analysis
- Comprehensive: Full analysis with all components
- Targeted: Customizable components based on config
---
# Strategy implementation
backend/services/blog_writer/research/research_strategies.py
- ResearchStrategy (base class)
- BasicResearchStrategy
- ComprehensiveResearchStrategy
- TargetedResearchStrategy
## 🏗️ Architecture
### Intent-Driven Research Flow
```
User Input
UnifiedResearchAnalyzer (Single AI Call)
├── Intent Inference
├── Query Generation
└── Parameter Optimization
Research Execution (Exa → Tavily → Google)
IntentAwareAnalyzer
├── Result Analysis
└── Deliverable Extraction
IntentDrivenResearchResult
```
### Frontend Component Structure
### Component Structure
```
frontend/src/components/Research/
├── index.tsx # Main exports
├── ResearchWizard.tsx # Main wizard container
├── ResearchWizard.tsx # Main wizard orchestrator
├── steps/
│ ├── StepKeyword.tsx # Step 1: Keyword input
│ ├── StepOptions.tsx # Step 2: Mode selection
│ ├── StepProgress.tsx # Step 3: Progress display
│ └── StepResults.tsx # Step 4: Results display
│ ├── ResearchInput.tsx # Step 1: Input + Intent & Options
│ ├── StepProgress.tsx # Step 2: Progress/polling
│ ├── StepResults.tsx # Step 3: Results display
│ └── components/ # Sub-components
├── hooks/
│ ├── useResearchWizard.ts # Wizard state management
── useResearchExecution.ts # API calls and polling
├── types/
│ └── research.types.ts # TypeScript interfaces
└── utils/
└── researchUtils.ts # Utility functions
│ ├── useResearchWizard.ts # Wizard state management
── useResearchExecution.ts # API calls and polling
│ └── useIntentResearch.ts # Intent-driven research flow
└── types/
├── research.types.ts # Wizard state types
└── intent.types.ts # Intent-driven types
```
## Test Page
---
A dedicated test page is available at `/research-test` for testing the research wizard independently.
**Features:**
- Quick preset keywords for testing
- Debug panel with JSON export
- Performance metrics display
- Cache state visualization
## Usage
### Standalone Usage
```typescript
import { ResearchWizard } from '../components/Research';
<ResearchWizard
onComplete={(results) => {
console.log('Research complete:', results);
}}
onCancel={() => {
console.log('Cancelled');
}}
initialKeywords={['AI', 'marketing']}
initialIndustry="Technology"
/>
```
### Integration with Blog Writer
The component is designed to be easily integrated into the BlogWriter research phase:
**Current Implementation:**
- Uses CopilotKit sidebar for research input
- Displays results in `ResearchResults` component
- Manual fallback via `ManualResearchForm`
**Proposed Integration:**
Replace the CopilotKit/manual form with the wizard:
```typescript
// In BlogWriter.tsx
{currentPhase === 'research' && (
<ResearchWizard
onComplete={(results) => setResearch(results)}
onCancel={() => navigate('blog-writer')}
/>
)}
```
## Backend API Changes
### New Models
The `BlogResearchRequest` model now supports:
```python
class BlogResearchRequest(BaseModel):
keywords: List[str]
topic: Optional[str] = None
industry: Optional[str] = None
target_audience: Optional[str] = None
tone: Optional[str] = None
word_count_target: Optional[int] = 1500
persona: Optional[PersonaInfo] = None
research_mode: Optional[ResearchMode] = ResearchMode.BASIC # NEW
config: Optional[ResearchConfig] = None # NEW
```
### Backward Compatibility
The API remains backward compatible:
- If `research_mode` is not provided, defaults to `BASIC`
- If `config` is not provided, defaults to standard configuration
- Existing requests continue to work unchanged
## Research Modes
### Basic Mode
- Quick keyword analysis
- Primary & secondary keywords
- Current trends overview
- Top 5 content angles
- Key statistics
### Comprehensive Mode
- All basic features plus:
- Expert quotes & opinions
- Competitor analysis
- Market forecasts
- Best practices & case studies
- Content gaps identification
### Targeted Mode
- Selectable components:
- Statistics
- Expert quotes
- Competitors
- Trends
- Always includes: Keywords & content angles
## Configuration Options
### ResearchConfig Model
```python
class ResearchConfig(BaseModel):
mode: ResearchMode = ResearchMode.BASIC
date_range: Optional[DateRange] = None
source_types: List[SourceType] = []
max_sources: int = 10
include_statistics: bool = True
include_expert_quotes: bool = True
include_competitors: bool = True
include_trends: bool = True
```
### Date Range Options
- `last_week`
- `last_month`
- `last_3_months`
- `last_6_months`
- `last_year`
- `all_time`
### Source Types
- `web` - Web articles
- `academic` - Academic papers
- `news` - News articles
- `industry` - Industry reports
- `expert` - Expert opinions
## Caching
The research component uses the existing cache infrastructure:
- Cache keys include research mode
- Cache is shared across basic/comprehensive/targeted modes
- Cache invalidation handled automatically
## Testing
### Test the Wizard
1. Navigate to `/research-test`
2. Use quick presets or enter custom keywords
3. Select research mode
4. Monitor progress
5. Review results
6. Export JSON for analysis
### Integration Testing
To test integration with BlogWriter:
1. Start backend: `python start_alwrity_backend.py`
2. Navigate to `/blog-writer` (current implementation)
3. Or navigate to `/research-test` (new wizard)
4. Compare results and UI
## Migration Path
### Phase 1: Parallel Testing (Current)
- `/research-test` - New wizard available
- `/blog-writer` - Current implementation unchanged
- Users can test both
### Phase 2: Integration
1. Add wizard as option in BlogWriter
2. A/B test user preference
3. Monitor performance metrics
### Phase 3: Replacement (Optional)
1. Replace CopilotKit/manual form with wizard
2. Remove old implementation
3. Update documentation
## API Endpoints
All existing endpoints remain unchanged:
```
POST /api/blog/research/start
- Supports new research_mode and config parameters
- Backward compatible with existing requests
GET /api/blog/research/status/{task_id}
- No changes required
```
## Benefits
1. **Modularity**: Component works standalone
2. **Testability**: Dedicated test page for experimentation
3. **Backward Compatibility**: Existing functionality unchanged
4. **Progressive Enhancement**: Can add features incrementally
5. **Reusability**: Can be used in other parts of the app
## Future Enhancements
Potential future improvements:
1. **Multi-stage Research**: Sequential research with refinement
2. **Source Quality Validation**: Advanced credibility scoring
3. **Interactive Query Builder**: Dynamic search refinement
4. **Advanced Prompting**: Few-shot examples, reasoning chains
5. **Custom Strategy Plugins**: User-defined research strategies
## Troubleshooting
### Research Results Not Showing
Check:
1. Backend logs for API errors
2. Network tab for failed requests
3. Browser console for JavaScript errors
4. Verify user authentication
### Cache Issues
Clear cache:
```typescript
import { researchCache } from '../services/researchCache';
researchCache.clearCache();
```
### Type Errors
Ensure all imports are correct:
```typescript
import {
ResearchWizard,
useResearchWizard,
WizardState
} from '../components/Research';
import {
BlogResearchRequest,
BlogResearchResponse,
ResearchMode,
ResearchConfig
} from '../services/blogWriterApi';
```
## Examples
## 🔌 Integration
### Basic Integration
```typescript
import { ResearchWizard } from './components/Research';
import { BlogResearchResponse } from './services/blogWriterApi';
const MyComponent: React.FC = () => {
const [results, setResults] = useState<BlogResearchResponse | null>(null);
import { ResearchWizard } from '../components/Research';
function MyComponent() {
return (
<ResearchWizard
onComplete={(res) => setResults(res)}
onCancel={() => console.log('Cancelled')}
onComplete={(results) => {
console.log('Research complete:', results);
// Use results in your component
}}
onCancel={() => {
console.log('Research cancelled');
}}
/>
);
};
}
```
### Advanced Integration with Custom Config
### With Initial Data
```typescript
const request: BlogResearchRequest = {
keywords: ['AI', 'automation'],
industry: 'Technology',
research_mode: 'targeted',
config: {
mode: 'targeted',
include_statistics: true,
include_competitors: true,
include_trends: false,
<ResearchWizard
initialKeywords={['AI marketing tools']}
initialIndustry="Technology"
initialTargetAudience="Marketing professionals"
initialResearchMode="comprehensive"
initialConfig={{
provider: 'exa',
max_sources: 20,
}
};
include_statistics: true,
include_expert_quotes: true
}}
initialResults={savedResults} // For restoring saved projects
/>
```
## Support
### Blog Writer Integration
For issues or questions:
1. Check this documentation
2. Review test page examples
3. Inspect backend logs
4. Check frontend console
```typescript
import { BlogWriterAdapter } from '../components/Research/integrations/BlogWriterAdapter';
function BlogWriter() {
const [researchData, setResearchData] = useState(null);
return (
<>
<BlogWriterAdapter
onResearchComplete={(data) => {
setResearchData(data);
// Use research data for blog generation
}}
/>
{/* Rest of blog writer UI */}
</>
);
}
```
---
## 🔄 Research Flow
### Step 1: Research Input
**User provides**:
- Keywords/topic
- Industry (optional, pre-filled from persona)
- Target audience (optional, pre-filled from persona)
**Component triggers**:
- Intent analysis when user clicks "Intent & Options"
- Shows `IntentConfirmationPanel` with AI-inferred intent
### Step 2: Intent Confirmation
**User reviews**:
- Primary research question
- Generated research queries
- Optimized provider settings
- Google Trends keywords (if applicable)
**User can**:
- Edit primary question
- Toggle deliverables
- Select/edit queries
- Review provider settings
**Component executes**:
- Research with selected queries
- Shows progress
- Auto-navigates to results
### Step 3: Results Display
**Component shows**:
- Summary tab (AI-generated overview)
- Deliverables tab (statistics, quotes, case studies, trends)
- Sources tab (citations with credibility scores)
- Analysis tab (deep insights)
---
## 🔌 API Integration
### Intent Analysis Endpoint
```typescript
POST /api/research/intent/analyze
Request:
{
"keywords": "AI marketing tools",
"industry": "Technology",
"target_audience": "Marketing professionals"
}
Response:
{
"success": true,
"intent": {
"primary_question": "What are the latest AI-powered marketing automation tools?",
"research_goals": ["identify tools", "compare features", "analyze trends"],
"deliverables": ["statistics", "expert_quotes", "case_studies"],
"industry": "Technology",
"target_audience": "Marketing professionals"
},
"queries": [
{
"query": "AI marketing automation platforms 2025",
"provider": "exa",
"justification": "Exa is best for finding company/product information"
}
],
"optimized_config": {
"provider": "exa",
"exa_category": "company",
"provider_justification": "Exa excels at finding company and product information"
},
"trends_config": {
"keywords": ["AI marketing", "marketing automation"],
"enabled": true
}
}
```
### Intent-Driven Research Endpoint
```typescript
POST /api/research/intent/research
Request:
{
"intent": {...},
"queries": [...],
"config": {...}
}
Response:
{
"success": true,
"result": {
"summary": "Comprehensive overview...",
"deliverables": {
"statistics": [
{
"value": "85%",
"description": "of marketers use AI tools",
"citation": {...}
}
],
"expert_quotes": [...],
"case_studies": [...],
"trends": [...]
},
"sources": [...],
"analysis": "Deep insights based on intent..."
}
}
```
---
## 🎨 Customization
### Custom Styling
```typescript
import { ResearchWizard } from '../components/Research';
import { ThemeProvider, createTheme } from '@mui/material';
const customTheme = createTheme({
// Your custom theme
});
<ThemeProvider theme={customTheme}>
<ResearchWizard {...props} />
</ThemeProvider>
```
### Custom Hooks
```typescript
import { useResearchWizard, useResearchExecution } from '../components/Research';
function CustomResearchComponent() {
const wizard = useResearchWizard();
const execution = useResearchExecution();
// Custom logic here
return <div>Custom UI</div>;
}
```
---
## 🔧 Backend Services
### UnifiedResearchAnalyzer
**Location**: `backend/services/research/intent/unified_research_analyzer.py`
**Purpose**: Single AI call for intent inference, query generation, and parameter optimization
**Usage**:
```python
from backend.services.research.intent.unified_research_analyzer import UnifiedResearchAnalyzer
analyzer = UnifiedResearchAnalyzer()
result = await analyzer.analyze(
user_input="AI marketing tools",
industry="Technology",
target_audience="Marketing professionals",
user_id="user_123"
)
```
### IntentAwareAnalyzer
**Location**: `backend/services/research/intent/intent_aware_analyzer.py`
**Purpose**: Analyzes raw research results based on user intent
**Usage**:
```python
from backend.services.research.intent.intent_aware_analyzer import IntentAwareAnalyzer
analyzer = IntentAwareAnalyzer()
result = await analyzer.analyze(
raw_results={...},
intent=research_intent,
user_id="user_123"
)
```
---
## 📝 Type Definitions
### Research Types
```typescript
// research.types.ts
export interface WizardState {
currentStep: number;
keywords: string[];
industry: string;
target_audience: string;
research_mode: ResearchMode;
config: ResearchConfig;
results: BlogResearchResponse | null;
}
export interface ResearchWizardProps {
onComplete?: (results: BlogResearchResponse) => void;
onCancel?: () => void;
initialKeywords?: string[];
initialIndustry?: string;
initialTargetAudience?: string;
initialResearchMode?: ResearchMode;
initialConfig?: ResearchConfig;
initialResults?: BlogResearchResponse | null;
}
```
### Intent Types
```typescript
// intent.types.ts
export interface ResearchIntent {
primary_question: string;
research_goals: string[];
deliverables: string[];
industry: string;
target_audience: string;
}
export interface ResearchQuery {
query: string;
provider: 'exa' | 'tavily' | 'google';
justification?: string;
}
export interface IntentDrivenResearchResult {
summary: string;
deliverables: {
statistics: StatisticWithCitation[];
expert_quotes: ExpertQuote[];
case_studies: CaseStudySummary[];
trends: TrendAnalysis[];
};
sources: Source[];
analysis: string;
}
```
---
## 🧪 Testing
### Standalone Testing
Navigate to `/research-test` for isolated testing:
- Test research flow
- Debug intent analysis
- Review results
- Export data
### Integration Testing
1. Import `ResearchWizard` in your component
2. Test with various initial data
3. Verify `onComplete` callback
4. Check error handling
---
## 🚀 Best Practices
### 1. Always Provide Initial Data When Available
```typescript
// Good: Pre-fill from user data
<ResearchWizard
initialIndustry={userProfile.industry}
initialTargetAudience={userProfile.targetAudience}
/>
// Avoid: Empty wizard when data is available
<ResearchWizard />
```
### 2. Handle Results Properly
```typescript
<ResearchWizard
onComplete={(results) => {
// Save results
saveResearchResults(results);
// Use in your component
setResearchData(results);
// Navigate if needed
navigate('/blog-writer', { state: { research: results } });
}}
/>
```
### 3. Use Research Persona
```typescript
// Research persona automatically pre-fills:
// - Industry
// - Target audience
// - Research preferences
// - Provider settings
// No additional code needed - it's automatic!
```
---
## 🔄 Migration from Old Architecture
### Old Architecture (Deprecated)
- 4-step wizard (StepKeyword → StepOptions → StepProgress → StepResults)
- Strategy pattern (Basic/Comprehensive/Targeted modes)
- Rule-based parameter optimization
### New Architecture
- 3-step wizard (ResearchInput → StepProgress → StepResults)
- Intent-driven (AI infers intent)
- Unified AI analyzer (single call)
- AI-optimized parameters
### Migration Steps
1. Replace old wizard components with `ResearchWizard`
2. Remove mode selection UI (handled by AI)
3. Update API calls to use intent-driven endpoints
4. Update result handling for new result structure
---
## 📚 Additional Resources
- **Architecture Rules**: `.cursor/rules/researcher-architecture.mdc`
- **Implementation Guide**: `RESEARCH_WIZARD_IMPLEMENTATION.md`
- **Intent-Driven Guide**: `INTENT_DRIVEN_RESEARCH_GUIDE.md`
- **Current Architecture**: `CURRENT_ARCHITECTURE_OVERVIEW.md`
---
## ✅ Implementation Status
- ✅ Intent-driven research implemented
- ✅ UnifiedResearchAnalyzer working
- ✅ IntentAwareAnalyzer working
- ✅ Google Trends integrated
- ✅ Research persona integrated
- ✅ My Projects feature (auto-save)
- ✅ Component refactoring complete
---
**Status**: Current and Accurate