kunthawat/ALwrity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Sprint 1 - Deep discovery, lead persistence, and dashboard nav

Browse Source 
- Add BacklinkOutreachScraper (Exa + DuckDuckGo deep scraping)
- Extend DB and Pydantic models for lead enrichment columns
- Add StorageService methods for lead CRUD with auto-migration
- Add backend endpoints: deep discover, campaign detail, lead management
- Extend frontend API client and store with discovery + lead actions
- Create BacklinkOutreachDashboard component with campaigns/discover/leads tabs
- Register route at /backlink-outreach under SEO feature flag
- Add nav entry under Enterprise & Advanced in tool categories
This commit is contained in:
ajaysi
2026-05-23 17:07:33 +05:30
parent 816d59a30a
commit 090d69761f
22 changed files with 3494 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

839
docs/SEO/PHASE2A_IMPLEMENTATION.md Normal file
View File

@@ -0,0 +1,839 @@
"""
Phase 2A Implementation: Enterprise SEO Suite & Advanced GSC Integration
COMPREHENSIVE DOCUMENTATION & DEPLOYMENT GUIDE
========================================
OVERVIEW: What's Implemented
========================================
This Phase 2A implementation provides:
1. **Enterprise SEO Service v2.0** (backend/services/seo_tools/enterprise_seo_service.py)
- Complete multi-tool orchestration
- Parallel component execution (Technical, On-Page, PageSpeed, Sitemap, Content)
- Competitive intelligence analysis
- AI-powered insights generation
- Executive reporting with ROI calculation
- Two audit modes: Complete (15-20 min) + Quick (5 min)
2. **Advanced GSC Analyzer Service** (backend/services/seo_tools/gsc_analyzer_service.py)
- Search performance analysis with trends
- Keyword-level performance breakdown
- Page-level opportunity identification
- Content opportunity engine (15+ scored opportunities)
- Technical SEO signal detection
- Competitive positioning analysis
- AI recommendations generation
- Detailed content opportunities report with phased implementation
3. **New API Endpoints** (Added to backend/routers/seo_tools.py)
========================================
NEW API ENDPOINTS (Complete Reference)
========================================
## ENTERPRISE AUDIT ENDPOINTS
### 1. Complete Enterprise SEO Audit
**Endpoint**: POST /api/seo/enterprise/complete-audit
**Method**: POST
**Authentication**: Required (Clerk)
**Response Time**: 15-20 minutes
**Rate Limit**: 1 per hour per user
**Request Body**:
```json
{
"website_url": "https://example.com",
"competitors": [
"https://competitor1.com",
"https://competitor2.com"
],
"target_keywords": [
"AI content creation",
"SEO tools",
"meta description generator"
],
"include_content_analysis": true,
"include_competitive_analysis": true,
"generate_executive_report": true
}
```
**Response**:
```json
{
"success": true,
"message": "Complete enterprise audit executed successfully",
"execution_time": 1245.67,
"data": {
"audit_id": "audit_20260523_143022",
"website_url": "https://example.com",
"audit_type": "complete_enterprise_audit",
"overall_score": 78.5,
"overall_status": "good",
"components_analyzed": 5,
"components_successful": 5,
"component_results": {
"technical_seo": {
"status": "completed",
"score": 80,
"critical_issues": [...],
"recommendations": [...],
"execution_time": 245.3
},
"on_page_seo": {...},
"pagespeed": {...},
"sitemap": {...},
"content_strategy": {...}
},
"component_scores": {
"technical_seo": 80,
"on_page_seo": 75,
"pagespeed": 70,
"sitemap": 90,
"content_strategy": 85
},
"priority_actions": [
{
"priority": "critical",
"recommendation": "Fix technical SEO issues...",
"source": "technical_seo",
"estimated_effort": "quick-win",
"potential_impact": "High",
"implementation_steps": [...]
}
],
"estimated_impact": "15-25% potential improvement",
"estimated_traffic_improvement": "15-35%",
"implementation_timeline": "2-4 weeks (with dedicated resources)",
"ai_insights": {
"status": "completed",
"ai_analysis": "Strategic analysis...",
"generated_at": "2026-05-23T14:30:22.123456"
},
"next_steps": [...]
}
}
```
**Error Handling**:
- 400: Invalid URL or request parameters
- 401: Not authenticated
- 429: Rate limit exceeded
- 500: Service error with error_id for support reference
---
### 2. Quick Enterprise Audit (5 Minutes)
**Endpoint**: POST /api/seo/enterprise/quick-audit
**Method**: POST
**Authentication**: Required
**Response Time**: 5 minutes
**Parameters**:
- `website_url` (required): URL to audit
**Response Structure**:
```json
{
"success": true,
"message": "Quick audit completed",
"data": {
"audit_type": "quick_audit",
"website_url": "https://example.com",
"quick_score": 75.2,
"critical_issues": [
"3 critical technical issues detected",
"Page speed below recommended threshold",
"5 indexing errors in GSC"
],
"top_recommendation": "Fix critical technical SEO issues and improve page speed"
}
}
```
---
### 3. Enterprise Services Health Check
**Endpoint**: GET /api/seo/enterprise/health
**Method**: GET
**Response Time**: < 1 second
**Response**:
```json
{
"success": true,
"message": "Enterprise services health check completed",
"data": {
"enterprise_seo_service": {
"status": "operational",
"service": "enterprise_seo_suite",
"version": "2.0",
"sub_services": {
"technical_seo": "operational",
"on_page_seo": "operational",
"pagespeed": "operational",
"sitemap": "operational",
"content_strategy": "operational"
}
},
"gsc_analyzer_service": {
"status": "operational",
"service": "gsc_analyzer",
"gsc_service_available": true,
"llm_integration": "available"
}
}
}
```
---
## ADVANCED GSC ANALYSIS ENDPOINTS
### 1. Comprehensive Search Performance Analysis
**Endpoint**: POST /api/seo/gsc/analyze-search-performance
**Method**: POST
**Authentication**: Required
**Response Time**: 2-3 minutes
**Rate Limit**: 5 per hour per user
**Request Body**:
```json
{
"site_url": "https://example.com",
"date_range_days": 90,
"include_opportunities": true,
"include_competitive": true
}
```
**Response**:
```json
{
"success": true,
"message": "GSC search performance analysis completed",
"data": {
"status": "completed",
"site_url": "https://example.com",
"analysis_period": "Last 90 days",
"execution_time_seconds": 125.4,
"performance_overview": {
"total_clicks": 5700,
"total_impressions": 37000,
"overall_ctr": 15.4,
"average_position": 4.9,
"total_keywords_tracked": 120,
"total_pages_indexed": 450,
"top_performing_keyword": "AI content creation",
"top_performing_page": "https://example.com/meta-description",
"device_breakdown": {
"mobile": 17.8,
"desktop": 16.7,
"tablet": 15.0
}
},
"keyword_analysis": {
"top_keywords": [...],
"total_keywords": 120,
"high_volume_low_ctr_keywords": [...],
"ranking_in_top_3": 45,
"avg_position": 4.9,
"keyword_trends": {
"improving": [...],
"declining": [...]
}
},
"page_analysis": {
"top_pages": [...],
"total_pages": 450,
"pages_with_impressions": 380,
"pages_with_no_clicks": 25,
"average_page_ctr": 14.8
},
"content_opportunities": [
{
"keyword": "AI content creation",
"current_position": 5,
"impressions": 2500,
"clicks": 250,
"ctr": 10,
"priority_score": 8.5,
"opportunity_type": "ranking_improvement",
"recommendation": "Optimize content and build backlinks to improve ranking position"
}
],
"technical_insights": {
"index_coverage": "Good - 98% of pages indexed",
"mobile_usability": "Good - No major issues detected",
"crawl_stats": {...}
},
"competitive_analysis": {
"market_position": "Strong in niche keywords",
"domain_visibility": "Growing trend",
"visibility_score": 72.5,
"competitive_keywords": [...],
"vulnerabilities": [...],
"recommendations": [...]
},
"ai_insights": {
"status": "completed",
"recommendations": "Strategic recommendations..."
},
"summary": {
"total_keywords": 120,
"total_pages": 450,
"opportunities_identified": 15,
"critical_issues": 3
}
}
}
```
---
### 2. Content Opportunities Report
**Endpoint**: POST /api/seo/gsc/content-opportunities
**Method**: POST
**Authentication**: Required
**Response Time**: 3-5 minutes
**Request Body**:
```json
{
"site_url": "https://example.com",
"min_impressions": 100,
"date_range_days": 90
}
```
**Response**:
```json
{
"success": true,
"message": "Content opportunities report generated",
"data": {
"status": "completed",
"site_url": "https://example.com",
"report_generated": "2026-05-23T14:30:22.123456",
"opportunities_identified": 15,
"estimated_additional_clicks": 450,
"estimated_traffic_increase": "25-40%",
"opportunities": [
{
"keyword": "High volume keyword",
"current_position": 8,
"impressions": 2000,
"clicks": 150,
"ctr": 7.5,
"priority_score": 9.2,
"opportunity_type": "high_volume_low_ctr",
"recommendation": "Improve meta title and description to increase CTR"
}
],
"implementation_priority": [
{
"phase": "Phase 1 (Weeks 1-2)",
"tasks": [
{
"keyword": "..." ,
"strategy": "Meta/title optimization"
}
]
},
{
"phase": "Phase 2 (Weeks 3-4)",
"tasks": [...]
},
{
"phase": "Phase 3 (Month 2)",
"tasks": [...]
}
]
}
}
```
---
## ERROR HANDLING
All endpoints include comprehensive error handling with structured error responses:
**400 Bad Request**:
```json
{
"success": false,
"message": "Invalid request parameters",
"error_type": "ValidationError",
"error_details": "min_impressions must be >= 10",
"timestamp": "2026-05-23T14:30:22.123456"
}
```
**401 Unauthorized**:
```json
{
"success": false,
"message": "Authentication required",
"error_type": "AuthenticationError",
"timestamp": "2026-05-23T14:30:22.123456"
}
```
**429 Rate Limited**:
```json
{
"success": false,
"message": "Rate limit exceeded",
"error_type": "RateLimitError",
"error_details": "Maximum 1 audit per hour allowed",
"timestamp": "2026-05-23T14:30:22.123456"
}
```
**500 Server Error**:
```json
{
"success": false,
"message": "Server error occurred",
"error_type": "InternalServerError",
"error_details": "error_id: seo_execute_enterprise_audit_20260523_143022",
"timestamp": "2026-05-23T14:30:22.123456"
}
```
---
========================================
FEATURE BREAKDOWN: What Each Service Does
========================================
## Enterprise SEO Service Features
### Complete Audit (execute_complete_audit)
**What it does**:
- Orchestrates 5 SEO analysis tools in parallel
- Collects results into unified report
- Scores each component (0-100)
- Calculates weighted overall score (0-100)
- Identifies competitive advantages/gaps
- Prioritizes 15+ actionable recommendations
- Generates AI-powered strategic insights
- Estimates ROI and implementation timeline
**Key Components**:
1. **Technical SEO Audit** (25% weight)
- Site crawl analysis (1-5 levels deep)
- Issue identification and severity
- Critical, High, Medium, Low classifications
- Robots.txt analysis
- Redirect and error detection
2. **On-Page SEO Audit** (25% weight)
- Meta tags analysis (title, description, viewport)
- Content quality assessment
- Keyword presence and density
- H1-H6 tag structure
- Image alt text evaluation
- Accessibility compliance
3. **PageSpeed Analysis** (20% weight)
- Core Web Vitals metrics
- Mobile & Desktop performance
- Optimization recommendations
- Performance score (0-100)
- Mobile/Desktop comparison
4. **Sitemap Analysis** (10% weight)
- URL structure evaluation
- Publishing frequency trends
- Content distribution analysis
- Competitive benchmarking
- Size and completeness
5. **Content Strategy** (20% weight)
- Content gap identification
- Keyword opportunity scoring
- Competitive content analysis
- Topic clustering
- Content recommendations
### Quick Audit (execute_quick_audit)
**What it does**:
- 5-minute rapid assessment
- Identifies 3-5 critical issues
- Top 1-2 immediate actions
- Quick overall scoring
- Suitable for time-constrained reviews
**Speed optimizations**:
- Only runs technical + pagespeed
- Limited crawl depth
- Cached competitor data
- Streamlined reporting
---
## GSC Analyzer Service Features
### Search Performance Analysis
**What it does**:
- Analyzes GSC data over specified period
- Calculates 30+ metrics across 8 dimensions
- Identifies trends and patterns
- Detects opportunities and issues
**Analysis Dimensions**:
1. **Performance Overview**
- Total clicks, impressions, CTR
- Average position
- Device breakdown (mobile/desktop/tablet)
- Search type distribution (web/news/image)
- Geographic performance
2. **Keyword Performance**
- Top 10 keywords by clicks
- Keywords ranking top 3
- High-volume, low-CTR keywords
- Trending keywords (up/down)
- Long-tail opportunities
3. **Page Performance**
- Top 10 pages by clicks
- Pages with zero clicks (opportunity)
- Average page CTR
- Page distribution analysis
4. **Content Opportunities** (15 scored opportunities)
- High-volume, low-CTR (meta/title optimization)
- Position 4-10 keywords (ranking improvement)
- Low-volume, top-3 keywords (expansion)
- Priority score (0-10)
- Opportunity type and recommendation
5. **Technical SEO Signals**
- Index coverage percentage
- Mobile usability issues
- Core Web Vitals status
- Crawl statistics
- Error tracking
6. **Competitive Position**
- Market position assessment
- Competitive keywords analysis
- Visibility trends
- Vulnerabilities vs competitors
- Recommendations for competitive edge
7. **Trend Analysis**
- Clicks trending (up/down/stable)
- Impressions trending
- CTR trending
- Position improvement/decline
- Seasonal patterns
8. **AI Insights**
- Strategic recommendations
- Quick wins (implementable in days)
- Long-term strategy (implementable in months)
- Competitive positioning advice
### Content Opportunities Report
**What it does**:
- Detailed deep-dive into content gaps
- Filters by minimum impressions threshold
- Ranks 15+ opportunities by priority
- Provides phased 3-month implementation plan
**Opportunity Types**:
1. **High-Volume, Low-CTR** (Priority: CRITICAL)
- Strategy: Meta/title/snippet optimization
- Effort: Quick-win (2-3 hours)
- Impact: +10-30% CTR potential
- Timeline: 1-2 weeks
2. **Ranking Improvement** (Priority: HIGH)
- Strategy: Content optimization + link building
- Effort: Short-term (1-2 days)
- Impact: +2-3 positions potential
- Timeline: 2-4 weeks
3. **Long-Tail Expansion** (Priority: MEDIUM)
- Strategy: Content expansion + topic clustering
- Effort: Medium-term (3-5 days)
- Impact: +50-100 new keywords
- Timeline: 1-2 months
---
========================================
SERVICE INITIALIZATION & INTEGRATION
========================================
Both services are automatically initialized when imported:
```python
# In routers/seo_tools.py
from services.seo_tools.enterprise_seo_service import EnterpriseSEOService
from services.seo_tools.gsc_analyzer_service import GSCAnalyzerService
# Initialization in endpoints
enterprise_service = EnterpriseSEOService() # Auto-initializes all sub-services
gsc_service = GSCAnalyzerService() # Auto-initializes GSC connection
```
**Sub-services automatically initialized by EnterpriseSEOService**:
- technical_seo_service: TechnicalSEOService()
- on_page_seo_service: OnPageSEOService()
- pagespeed_service: PageSpeedService()
- sitemap_service: SitemapService()
- content_strategy_service: ContentStrategyService()
---
========================================
DATABASE INTEGRATION
========================================
Both services support optional database integration:
```python
# User-specific audit results can be saved
user_id = current_user.get("id")
db_session = get_session_for_user(user_id)
# Store audit results for later retrieval
# Save to audit_results table with audit_id for tracking
```
**Data persistence**:
- Audit results cached for 24 hours
- GSC data updated on analysis execution
- Historical trends maintained in database
- User dashboard integration ready
---
========================================
CONCURRENT EXECUTION & PERFORMANCE
========================================
**Enterprise Audit Concurrency**:
- All 5 components run in parallel
- Expected runtime: 15-20 minutes (vs ~60 min if sequential)
- Uses asyncio.gather() for coordination
- Graceful error handling per component
**GSC Analysis Concurrency**:
- All 8 analysis tasks run in parallel
- Expected runtime: 2-3 minutes
- Database queries optimized with indexing
- Mock data generator for development/testing
**Performance Optimizations**:
1. Parallel component execution
2. Result caching (24 hour TTL)
3. Lazy loading for sub-components
4. Streaming large datasets
5. Connection pooling for database
---
========================================
DEPLOYMENT CHECKLIST
========================================
## Pre-Deployment Steps
- [x] Import services in routers/seo_tools.py
- [x] Add request/response models
- [x] Create API endpoints
- [x] Add error handling
- [x] Create comprehensive tests
- [x] Update service documentation
- [ ] Configure environment variables:
- GOOGLE_CLIENT_ID (for GSC auth)
- GOOGLE_CLIENT_SECRET
- GSC_REDIRECT_URI
- LLM_API_KEY (for AI insights)
## Deployment Commands
```bash
# 1. Install any new dependencies
pip install -r requirements.txt
# 2. Run syntax checks
python -m py_compile backend/services/seo_tools/enterprise_seo_service.py
python -m py_compile backend/services/seo_tools/gsc_analyzer_service.py
# 3. Run test suite
pytest backend/tests/test_enterprise_gsc_services.py -v
# 4. Update database schema if needed
python backend/alembic/env.py upgrade head
# 5. Restart backend server
pkill -f "start_alwrity_backend.py"
python backend/start_alwrity_backend.py --dev
# 6. Verify endpoints
curl http://localhost:8000/api/seo/enterprise/health
```
---
========================================
USAGE EXAMPLES
========================================
### Python Client Example
```python
import asyncio
from services.seo_tools.enterprise_seo_service import EnterpriseSEOService
async def run_audit():
service = EnterpriseSEOService()
result = await service.execute_complete_audit(
website_url="https://mysite.com",
competitors=["https://competitor.com"],
target_keywords=["my keyword", "another keyword"],
include_content_analysis=True
)
print(f"Overall Score: {result['overall_score']}")
print(f"Status: {result['overall_status']}")
print(f"Priority Actions: {len(result['priority_actions'])}")
asyncio.run(run_audit())
```
### cURL Examples
```bash
# Complete Enterprise Audit
curl -X POST http://localhost:8000/api/seo/enterprise/complete-audit \\
-H "Authorization: Bearer {token}" \\
-H "Content-Type: application/json" \\
-d '{
"website_url": "https://example.com",
"target_keywords": ["AI", "SEO"]
}'
# GSC Search Performance
curl -X POST http://localhost:8000/api/seo/gsc/analyze-search-performance \\
-H "Authorization: Bearer {token}" \\
-H "Content-Type: application/json" \\
-d '{
"site_url": "https://example.com",
"date_range_days": 90
}'
# Content Opportunities
curl -X POST http://localhost:8000/api/seo/gsc/content-opportunities \\
-H "Authorization: Bearer {token}" \\
-H "Content-Type: application/json" \\
-d '{
"site_url": "https://example.com",
"min_impressions": 100
}'
```
---
========================================
MONITORING & LOGGING
========================================
All endpoints generate detailed logs:
**Log Location**: backend/logs/seo_tools/
**Log Levels**:
- INFO: Request start, component execution
- ERROR: Failed components, validation errors
- DEBUG: Detailed component metrics, intermediate results
**Example Log Format**:
```
2026-05-23 14:30:22 | INFO | [audit_20260523_143022] Starting complete audit for https://example.com
2026-05-23 14:30:45 | INFO | [audit_20260523_143022] Starting technical SEO audit...
2026-05-23 14:31:00 | INFO | [audit_20260523_143022] Technical audit completed in 245.3s
2026-05-23 14:32:55 | INFO | [audit_20260523_143022] Audit completed successfully in 1245.67s with score 78.5
```
---
========================================
TROUBLESHOOTING
========================================
**Issue**: Audit times out (> 30 seconds)
**Solution**:
- Check network connectivity
- Verify target website is accessible
- Reduce crawl depth for technical audit
- Use quick audit instead
**Issue**: "GSC credentials not found"
**Solution**:
- Set GOOGLE_CLIENT_ID environment variable
- Set GOOGLE_CLIENT_SECRET environment variable
- Ensure gsc_credentials.json exists in backend/
**Issue**: "LLM insights unavailable"
**Solution**:
- Check LLM_API_KEY environment variable
- Verify LLM service is running
- Fallback text will be returned
**Issue**: "Rate limit exceeded"
**Solution**:
- Enterprise audit: 1 per hour
- GSC analysis: 5 per hour
- Implement request queuing if needed
---
========================================
FUTURE ENHANCEMENTS (Phase 2B/2C)
========================================
### Phase 2B (Next 1-2 weeks)
- [ ] Schema markup generation service
- [ ] Text readability analyzer integration
- [ ] Advanced competitor analysis API
- [ ] Custom reporting templates
- [ ] Automated scheduled audits
### Phase 2C (Optional)
- [ ] Image optimization service
- [ ] Advanced backlink analysis
- [ ] Real-time monitoring dashboard
- [ ] Slack/Email notifications
- [ ] API rate limiting configuration
---
========================================
SUPPORT & DOCUMENTATION
========================================
**File Locations**:
- Services: backend/services/seo_tools/
- Routes: backend/routers/seo_tools.py
- Tests: backend/tests/test_enterprise_gsc_services.py
- Docs: docs/SEO/PHASE2A_IMPLEMENTATION.md (this file)
**Questions?**:
- Check test file for usage examples
- Review inline code comments
- Check error logs in backend/logs/seo_tools/
---
Last Updated: May 23, 2026
Implementation Status: Phase 2A Complete (73% → 85% migration)
"""