kunthawat/ALwrity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fix/add-matplotlib-backend-dependency
ALwrity/docs/ALwrity Researcher/GOOGLE_TRENDS_PHASE1_IMPLEMENTATION.md

9.7 KiB
Raw Permalink Blame History

Google Trends Phase 1 Implementation Summary

Date: 2025-01-29
Status: Phase 1 Core Service Complete

✅ What Was Implemented

1. Google Trends Service ⭐

File: backend/services/research/trends/google_trends_service.py

Features:

  • ✅ analyze_trends() - Comprehensive trends analysis
  • ✅ get_trending_searches() - Current trending searches
  • ✅ Interest over time
  • ✅ Interest by region
  • ✅ Related topics (top & rising)
  • ✅ Related queries (top & rising)
  • ✅ Rate limiting (1 req/sec)
  • ✅ Caching (24-hour TTL)
  • ✅ Async support
  • ✅ Error handling with fallback
  • ✅ Data serialization (DataFrames → dicts)

Key Methods:

async def analyze_trends(
    keywords: List[str],
    timeframe: str = "today 12-m",
    geo: str = "US",
    user_id: Optional[str] = None
) -> Dict[str, Any]

2. Rate Limiter ⭐

File: backend/services/research/trends/rate_limiter.py

Features:

  • ✅ Async rate limiting
  • ✅ Thread-safe with locks
  • ✅ Configurable (max_calls, period)
  • ✅ Automatic cleanup of old calls

3. Data Models ⭐

File: backend/models/research_trends_models.py

Models Created:

  • ✅ GoogleTrendsData - Structured trends data
  • ✅ TrendsConfig - AI-driven trends configuration
  • ✅ TrendsAnalysisResponse - API response model

4. Extended UnifiedResearchAnalyzer ⭐

File: backend/services/research/intent/unified_research_analyzer.py

Enhancements:

  • ✅ Added "PART 4: GOOGLE TRENDS KEYWORDS" to unified prompt
  • ✅ AI suggests optimized keywords for trends analysis
  • ✅ AI suggests timeframe and geo with justifications
  • ✅ AI lists expected insights trends will uncover
  • ✅ Added trends_config to unified schema
  • ✅ Added trends_config to response parser

Prompt Addition:

### PART 4: GOOGLE TRENDS KEYWORDS (if trends in deliverables)
If "trends" is in expected_deliverables OR purpose is "explore_trends":
- Suggest 1-3 optimized keywords for Google Trends analysis
- These may differ from research queries (trends need broader, searchable terms)
- Consider: What keywords will show meaningful trends over time?
- Consider: What timeframe will show relevant trends?
- Consider: What geographic region is most relevant?
- Explain what insights trends will uncover for content generation

5. Enhanced API Router ⭐

File: backend/api/research/router.py

Enhancements:

  • ✅ Added trends_config to AnalyzeIntentResponse
  • ✅ Added trends_config to IntentDrivenResearchRequest
  • ✅ Added google_trends_data to IntentDrivenResearchResponse
  • ✅ Parallel execution of research + trends
  • ✅ Trends data merging into results
  • ✅ Helper function _merge_trends_data()

Parallel Execution:

# Execute research and trends in parallel
research_task = asyncio.create_task(engine.research(context))
trends_task = asyncio.create_task(trends_service.analyze_trends(...))

# Wait for both
raw_result = await research_task
trends_data = await trends_task

🎯 Design Decisions Made

Decision 1: Extend Unified Prompt ✅

Answer: Extended UnifiedResearchAnalyzer to include trends keyword suggestions

Rationale:

  • Maintains single LLM call pattern
  • Coherent reasoning across research + trends
  • Consistent with Exa/Tavily optimization approach
  • Trends keywords align with research intent

Decision 2: Parallel Execution ✅

Answer: Execute trends in parallel with research

Implementation:

  • Use asyncio.create_task() for both
  • Use asyncio.gather() or await sequentially
  • Merge trends data into results after both complete

Decision 3: Trends Config Display ✅

Answer: Show in IntentConfirmationPanel with expected insights

What User Sees:

  • Trends keywords (AI-suggested, editable)
  • Timeframe & geo (with justifications)
  • Expected insights preview (what trends will uncover)

📊 Data Flow

User Input → UnifiedResearchAnalyzer
     │
     ├── Infers Intent
     ├── Generates Research Queries
     ├── Optimizes Exa/Tavily Params
     └── Suggests Trends Keywords ← NEW
     │
     ▼
IntentConfirmationPanel
     ├── Shows Intent
     ├── Shows Research Queries
     ├── Shows Exa/Tavily Settings
     └── Shows Trends Config ← NEW
          ├── Keywords (editable)
          ├── Timeframe & Geo (with justifications)
          └── Expected Insights Preview
     │
     ▼
User Clicks "Research"
     │
     ▼
Parallel Execution
     ├── Research Task (Exa/Tavily/Google)
     └── Trends Task (Google Trends) ← NEW
     │
     ▼
Merge Results
     ├── Analyze Research Results
     └── Merge Trends Data ← NEW
     │
     ▼
IntentResultsDisplay
     └── Enhanced Trends Tab ← TODO (Frontend)

🔧 Technical Implementation

Service Structure

backend/services/research/trends/
├── __init__.py
├── google_trends_service.py  ✅ Created
└── rate_limiter.py           ✅ Created

Key Features

  1. Async Support: All methods are async, use asyncio.to_thread() for pytrends
  2. Rate Limiting: 1 request per second (prevents Google blocking)
  3. Caching: 24-hour TTL (trends don't change frequently)
  4. Error Handling: Graceful fallback, partial data return
  5. Data Serialization: Converts DataFrames to dicts for API responses

Integration Points

  1. UnifiedResearchAnalyzer: Extended prompt and schema
  2. API Router: Parallel execution and data merging
  3. Response Models: Added trends_config and google_trends_data

📝 Next Steps (Frontend Integration)

Phase 2: Frontend Updates

  1. Update Types:

    • Add trends_config to AnalyzeIntentResponse type
    • Add google_trends_data to IntentDrivenResearchResponse type

  2. Enhance IntentConfirmationPanel:

    • Add trends section (accordion)
    • Show trends keywords (editable)
    • Show expected insights preview
    • Show timeframe & geo with justifications

  3. Enhance IntentResultsDisplay:

    • Add interest over time chart
    • Add interest by region table/map
    • Add related topics/queries display
    • Merge with AI-extracted trends

✅ Testing Checklist

Backend Testing

  • Test GoogleTrendsService.analyze_trends() with sample keywords
  • Test rate limiting (multiple rapid requests)
  • Test caching (same keywords return cached data)
  • Test error handling (invalid keywords, API failures)
  • Test parallel execution (research + trends)
  • Test data merging (trends data in results)

Integration Testing

  • Test intent analysis with trends in deliverables
  • Test trends_config in API response
  • Test parallel execution in research endpoint
  • Test trends data in final response

🚀 Usage Example

Backend Usage

from services.research.trends.google_trends_service import GoogleTrendsService

service = GoogleTrendsService()
trends_data = await service.analyze_trends(
    keywords=["AI marketing", "marketing automation"],
    timeframe="today 12-m",
    geo="US",
    user_id=user_id
)

# Returns:
# {
#   "interest_over_time": [...],
#   "interest_by_region": [...],
#   "related_topics": {"top": [...], "rising": [...]},
#   "related_queries": {"top": [...], "rising": [...]},
#   "timeframe": "today 12-m",
#   "geo": "US",
#   "keywords": ["AI marketing", "marketing automation"],
#   "timestamp": "2025-01-29T...",
#   "cached": false
# }

API Usage

POST /api/research/intent/analyze
{
  "user_input": "AI marketing tools for small businesses",
  "keywords": ["AI", "marketing", "tools"]
}

Response:
{
  "success": true,
  "intent": {...},
  "trends_config": {
    "enabled": true,
    "keywords": ["AI marketing", "marketing automation"],
    "keywords_justification": "These keywords will show search interest trends...",
    "timeframe": "today 12-m",
    "timeframe_justification": "12 months provides enough data...",
    "geo": "US",
    "geo_justification": "US market is most relevant...",
    "expected_insights": [
      "Search interest trends over the past year",
      "Regional interest distribution",
      "Related topics for content expansion",
      "Related queries for FAQ sections",
      "Optimal publication timing based on interest peaks"
    ]
  }
}

📋 Dependencies

Required Package

# requirements.txt
pytrends>=4.9.2  # Google Trends API

Installation

pip install pytrends>=4.9.2

⚠️ Known Limitations

  1. Pytrends Rate Limits: Google Trends API is rate-limited (1 req/sec)

    • Mitigation: Rate limiter implemented, caching reduces API calls

  2. Data Availability: Some keywords may have insufficient data

    • Mitigation: Graceful fallback, return partial data if available

  3. Geographic Limitations: Some regions may have limited data

    • Mitigation: Default to "US" if region unavailable

🎯 Success Metrics

  • Google Trends service created and working
  • Rate limiting preventing blocks
  • Caching working (24-hour TTL)
  • Error handling graceful
  • Parallel execution implemented
  • Data merging working
  • Frontend integration (Phase 2)
  • User testing and feedback

📝 Files Created/Modified

Created:

  • ✅ backend/services/research/trends/__init__.py
  • ✅ backend/services/research/trends/google_trends_service.py
  • ✅ backend/services/research/trends/rate_limiter.py
  • ✅ backend/models/research_trends_models.py

Modified:

  • ✅ backend/services/research/intent/unified_research_analyzer.py
  • ✅ backend/api/research/router.py

Status: Phase 1 Complete - Core Service Ready

Next: Phase 2 - Frontend Integration (IntentConfirmationPanel + IntentResultsDisplay)

Reference in New Issue View Git Blame Copy Permalink