Add AI SEO tools with FastAPI endpoints and comprehensive services

Co-authored-by: ajay.calsoft <ajay.calsoft@gmail.com>
2025-08-24 11:47:42 +00:00
parent 5d8d1cfb73
commit 512f82b7b0
17 changed files with 3637 additions and 0 deletions
--- a/backend/docs/SEO_TOOLS_MIGRATION.md
+++ b/backend/docs/SEO_TOOLS_MIGRATION.md
@@ -0,0 +1,401 @@
+# AI SEO Tools Migration Documentation
+
+## Overview
+
+This document describes the successful migration of AI SEO tools from the `ToBeMigrated/ai_seo_tools` directory to FastAPI endpoints in the backend services. The migration maintains all existing functionality while adding intelligent logging, exception handling, and structured API responses.
+
+## Migration Summary
+
+### What Was Migrated
+
+The following SEO tools have been converted to FastAPI endpoints:
+
+1. **Meta Description Generator** - AI-powered meta description generation
+2. **Google PageSpeed Insights Analyzer** - Performance analysis with AI insights
+3. **Sitemap Analyzer** - Website structure and content trend analysis
+4. **Image Alt Text Generator** - AI-powered alt text generation
+5. **OpenGraph Tags Generator** - Social media optimization tags
+6. **On-Page SEO Analyzer** - Comprehensive on-page SEO analysis
+7. **Technical SEO Analyzer** - Website crawling and technical analysis
+8. **Enterprise SEO Suite** - Complete SEO audit workflows
+9. **Content Strategy Analyzer** - AI-powered content gap analysis
+
+### New Architecture
+
+```
+backend/
+├── services/seo_tools/           # SEO tool services
+│   ├── meta_description_service.py
+│   ├── pagespeed_service.py
+│   ├── sitemap_service.py
+│   ├── image_alt_service.py
+│   ├── opengraph_service.py
+│   ├── on_page_seo_service.py
+│   ├── technical_seo_service.py
+│   ├── enterprise_seo_service.py
+│   └── content_strategy_service.py
+├── routers/seo_tools.py         # FastAPI router
+├── middleware/logging_middleware.py  # Intelligent logging
+└── logs/seo_tools/              # Structured log files
+```
+
+## API Endpoints
+
+### Base URL
+All SEO tools are available under: `/api/seo`
+
+### Individual Tool Endpoints
+
+#### 1. Meta Description Generation
+- **Endpoint**: `POST /api/seo/meta-description`
+- **Purpose**: Generate AI-powered SEO meta descriptions
+- **Request**:
+```json
+{
+  "keywords": ["SEO", "content marketing"],
+  "tone": "Professional",
+  "search_intent": "Informational Intent",
+  "language": "English",
+  "custom_prompt": "Optional custom prompt"
+}
+```
+- **Response**: Structured response with 5 meta descriptions, analysis, and recommendations
+
+#### 2. PageSpeed Analysis
+- **Endpoint**: `POST /api/seo/pagespeed-analysis`
+- **Purpose**: Analyze website performance using Google PageSpeed Insights
+- **Request**:
+```json
+{
+  "url": "https://example.com",
+  "strategy": "DESKTOP",
+  "locale": "en",
+  "categories": ["performance", "accessibility", "best-practices", "seo"]
+}
+```
+- **Response**: Performance metrics, Core Web Vitals, AI insights, and optimization plan
+
+#### 3. Sitemap Analysis
+- **Endpoint**: `POST /api/seo/sitemap-analysis`
+- **Purpose**: Analyze website sitemap structure and content patterns
+- **Request**:
+```json
+{
+  "sitemap_url": "https://example.com/sitemap.xml",
+  "analyze_content_trends": true,
+  "analyze_publishing_patterns": true
+}
+```
+- **Response**: Structure analysis, content trends, publishing patterns, and AI insights
+
+#### 4. Image Alt Text Generation
+- **Endpoint**: `POST /api/seo/image-alt-text`
+- **Purpose**: Generate SEO-optimized alt text for images
+- **Request**: Form data with image file or JSON with image URL
+- **Response**: Generated alt text with confidence score and suggestions
+
+#### 5. OpenGraph Tags Generation
+- **Endpoint**: `POST /api/seo/opengraph-tags`
+- **Purpose**: Generate OpenGraph tags for social media optimization
+- **Request**:
+```json
+{
+  "url": "https://example.com",
+  "title_hint": "Optional title hint",
+  "description_hint": "Optional description hint",
+  "platform": "General"
+}
+```
+- **Response**: Complete OpenGraph tags with platform-specific optimizations
+
+#### 6. On-Page SEO Analysis
+- **Endpoint**: `POST /api/seo/on-page-analysis`
+- **Purpose**: Comprehensive on-page SEO analysis
+- **Request**:
+```json
+{
+  "url": "https://example.com",
+  "target_keywords": ["keyword1", "keyword2"],
+  "analyze_images": true,
+  "analyze_content_quality": true
+}
+```
+- **Response**: SEO score, content analysis, keyword optimization, and recommendations
+
+#### 7. Technical SEO Analysis
+- **Endpoint**: `POST /api/seo/technical-seo`
+- **Purpose**: Technical SEO crawling and analysis
+- **Request**:
+```json
+{
+  "url": "https://example.com",
+  "crawl_depth": 3,
+  "include_external_links": true,
+  "analyze_performance": true
+}
+```
+- **Response**: Technical issues, site structure, performance metrics, and recommendations
+
+### Workflow Endpoints
+
+#### 1. Complete Website Audit
+- **Endpoint**: `POST /api/seo/workflow/website-audit`
+- **Purpose**: Execute comprehensive SEO audit workflow
+- **Request**:
+```json
+{
+  "website_url": "https://example.com",
+  "workflow_type": "complete_audit",
+  "competitors": ["https://competitor1.com"],
+  "target_keywords": ["keyword1", "keyword2"]
+}
+```
+
+#### 2. Content Analysis Workflow
+- **Endpoint**: `POST /api/seo/workflow/content-analysis`
+- **Purpose**: AI-powered content strategy analysis
+- **Request**:
+```json
+{
+  "website_url": "https://example.com",
+  "workflow_type": "content_analysis",
+  "competitors": ["https://competitor1.com"],
+  "target_keywords": ["content", "strategy"]
+}
+```
+
+### Health and Status Endpoints
+
+- **GET** `/api/seo/health` - Health check for SEO tools
+- **GET** `/api/seo/tools/status` - Status of all SEO tools and dependencies
+
+## Key Features
+
+### 1. Intelligent Logging
+- **Structured Logging**: All operations logged to JSONL files
+- **Performance Tracking**: Execution time monitoring
+- **Error Logging**: Comprehensive error tracking with stack traces
+- **AI Analysis Logging**: Prompt/response tracking for AI operations
+
+**Log Files**:
+- `/backend/logs/seo_tools/operations.jsonl` - Successful operations
+- `/backend/logs/seo_tools/errors.jsonl` - Error logs
+- `/backend/logs/seo_tools/ai_analysis.jsonl` - AI prompt/response logs
+- `/backend/logs/seo_tools/external_apis.jsonl` - External API calls
+- `/backend/logs/seo_tools/crawling.jsonl` - Web crawling operations
+
+### 2. Exception Handling
+- **Never Mock Data**: Real API failures return proper error responses
+- **Graceful Degradation**: AI analysis failures don't break core functionality
+- **Detailed Error Messages**: Clear error descriptions for debugging
+- **Error IDs**: Unique error identifiers for tracking
+
+### 3. AI Enhancement
+- **Gemini Integration**: Uses `gemini_provide` functionality for AI analysis
+- **Structured Responses**: AI responses parsed into structured data
+- **Context-Aware Analysis**: AI considers user type (content creators, marketers)
+- **Business Impact Focus**: AI recommendations focus on practical business outcomes
+
+### 4. Background Processing
+- **Async Operations**: All heavy operations run asynchronously
+- **Background Tasks**: Logging and cleanup run in background
+- **Non-blocking**: API responses don't wait for logging operations
+
+## Response Format
+
+All endpoints follow a consistent response format:
+
+```json
+{
+  "success": true,
+  "message": "Operation completed successfully",
+  "timestamp": "2024-01-15T10:30:00Z",
+  "execution_time": 2.45,
+  "data": {
+    // Tool-specific data
+  }
+}
+```
+
+**Error Response**:
+```json
+{
+  "success": false,
+  "message": "Error description",
+  "timestamp": "2024-01-15T10:30:00Z",
+  "execution_time": 1.23,
+  "error_type": "ValueError",
+  "error_details": "Detailed error message",
+  "traceback": "Full traceback (only in debug mode)"
+}
+```
+
+## Dependencies
+
+### New Dependencies Added
+```
+aiofiles>=23.2.0      # Async file operations
+crawl4ai>=0.2.0       # Web crawling (placeholder)
+```
+
+### Existing Dependencies Used
+- `fastapi` - Web framework
+- `pydantic` - Data validation
+- `aiohttp` - Async HTTP client
+- `beautifulsoup4` - HTML parsing
+- `advertools` - SEO analysis
+- `loguru` - Logging
+- `google-genai` - AI analysis
+
+## Testing
+
+### Test Script
+Run the comprehensive test suite:
+```bash
+cd /workspace/backend
+python test_seo_tools.py
+```
+
+### Manual Testing
+1. Start the FastAPI server:
+```bash
+uvicorn app:app --reload --host 0.0.0.0 --port 8000
+```
+
+2. Access API documentation:
+- Swagger UI: `http://localhost:8000/docs`
+- ReDoc: `http://localhost:8000/redoc`
+
+3. Test individual endpoints using the documentation interface
+
+## Configuration
+
+### Environment Variables
+Set these environment variables for full functionality:
+
+```bash
+# Google PageSpeed Insights API Key (optional)
+GOOGLE_PAGESPEED_API_KEY=your_api_key_here
+
+# AI Provider API Keys (at least one required)
+GEMINI_API_KEY=your_gemini_key
+OPENAI_API_KEY=your_openai_key
+ANTHROPIC_API_KEY=your_anthropic_key
+
+# Debug mode (optional)
+DEBUG=false
+```
+
+### Logging Configuration
+Logs are automatically rotated daily and retained for 30 days. Configure in:
+`/workspace/backend/middleware/logging_middleware.py`
+
+## Migration Benefits
+
+### For Content Creators
+- **User-Friendly**: API responses tailored for non-technical users
+- **Actionable Insights**: Clear recommendations with business impact
+- **Comprehensive Analysis**: All-in-one SEO analysis platform
+- **AI-Enhanced**: Advanced AI provides strategic insights
+
+### For Digital Marketers
+- **Performance Tracking**: Detailed metrics and optimization plans
+- **Competitive Analysis**: Built-in competitor intelligence
+- **Workflow Automation**: Complete audit workflows
+- **ROI Focus**: Recommendations tied to business outcomes
+
+### For Solopreneurs
+- **Cost-Effective**: Single API for multiple SEO tools
+- **Time-Saving**: Automated analysis and recommendations
+- **Easy Integration**: RESTful API with clear documentation
+- **Scalable**: Handles small to enterprise-level analysis
+
+### For Developers
+- **Modern Architecture**: FastAPI with async support
+- **Comprehensive Logging**: Full observability
+- **Error Handling**: Robust error management
+- **Documentation**: Auto-generated API docs
+
+## Monitoring and Maintenance
+
+### Log Analysis
+Use the built-in log analyzer for insights:
+```python
+from middleware.logging_middleware import log_analyzer
+
+# Get performance summary
+performance = await log_analyzer.get_performance_summary(hours=24)
+
+# Get error summary  
+errors = await log_analyzer.get_error_summary(hours=24)
+```
+
+### Health Monitoring
+Monitor service health via:
+- `/api/seo/health` - Overall health
+- `/api/seo/tools/status` - Individual tool status
+
+### Performance Optimization
+- Monitor execution times in logs
+- Optimize slow-performing tools
+- Scale based on usage patterns
+
+## Future Enhancements
+
+### Planned Features
+1. **Real-time Monitoring Dashboard** - Visual monitoring interface
+2. **Batch Processing** - Process multiple URLs simultaneously
+3. **Webhook Support** - Async notifications for long-running operations
+4. **Rate Limiting** - Prevent API abuse
+5. **Caching** - Cache frequently requested analyses
+6. **Authentication** - API key-based authentication
+7. **Usage Analytics** - Track API usage and popular tools
+
+### Extension Points
+1. **New SEO Tools** - Easy to add new tools following existing patterns
+2. **Custom AI Models** - Support for additional AI providers
+3. **Export Formats** - PDF, Excel, CSV export options
+4. **Integration APIs** - Connect with popular marketing tools
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Import Errors**
+   - Ensure all dependencies are installed: `pip install -r requirements.txt`
+   - Check Python path configuration
+
+2. **AI Analysis Failures**
+   - Verify API keys are set correctly
+   - Check internet connectivity
+   - Review error logs for specific issues
+
+3. **PageSpeed API Errors**
+   - Get Google PageSpeed API key for higher rate limits
+   - Verify URL format and accessibility
+
+4. **Logging Issues**
+   - Ensure write permissions to `/workspace/backend/logs/`
+   - Check disk space availability
+
+### Debug Mode
+Enable debug mode for detailed error information:
+```bash
+export DEBUG=true
+```
+
+This will include full tracebacks in API responses.
+
+## Conclusion
+
+The AI SEO Tools migration successfully transforms individual Python scripts into a cohesive, scalable FastAPI service. The new architecture provides:
+
+- ✅ **Complete Functionality Preservation**
+- ✅ **Enhanced Error Handling** 
+- ✅ **Intelligent Logging**
+- ✅ **AI-Powered Insights**
+- ✅ **Workflow Automation**
+- ✅ **Developer-Friendly API**
+- ✅ **Business-Focused Outputs**
+
+The system is now ready for production use and can easily scale to serve content creators, digital marketers, and solopreneurs with professional-grade SEO analysis capabilities.