ALwrity/docs/ALwrity Researcher/GOOGLE_TRENDS_PHASE1_IMPLEMENTATION.md

Google Trends Phase 1 Implementation Summary

Date: 2025-01-29
Status: Phase 1 Core Service Complete

---

✅ What Was Implemented

1. Google Trends Service ⭐

File: backend/services/research/trends/google_trends_service.py

Features:

• ✅ analyze_trends() - Comprehensive trends analysis
• ✅ get_trending_searches() - Current trending searches
• ✅ Interest over time
• ✅ Interest by region
• ✅ Related topics (top & rising)
• ✅ Related queries (top & rising)
• ✅ Rate limiting (1 req/sec)
• ✅ Caching (24-hour TTL)
• ✅ Async support
• ✅ Error handling with fallback
• ✅ Data serialization (DataFrames → dicts)

Key Methods:

async def analyze_trends(
    keywords: List[str],
    timeframe: str = "today 12-m",
    geo: str = "US",
    user_id: Optional[str] = None
) -> Dict[str, Any]

2. Rate Limiter ⭐

File: backend/services/research/trends/rate_limiter.py

Features:

• ✅ Async rate limiting
• ✅ Thread-safe with locks
• ✅ Configurable (max_calls, period)
• ✅ Automatic cleanup of old calls

3. Data Models ⭐

File: backend/models/research_trends_models.py

Models Created:

• ✅ GoogleTrendsData - Structured trends data
• ✅ TrendsConfig - AI-driven trends configuration
• ✅ TrendsAnalysisResponse - API response model

4. Extended UnifiedResearchAnalyzer ⭐

File: backend/services/research/intent/unified_research_analyzer.py

Enhancements:

• ✅ Added "PART 4: GOOGLE TRENDS KEYWORDS" to unified prompt
• ✅ AI suggests optimized keywords for trends analysis
• ✅ AI suggests timeframe and geo with justifications
• ✅ AI lists expected insights trends will uncover
• ✅ Added trends_config to unified schema
• ✅ Added trends_config to response parser

Prompt Addition:

### PART 4: GOOGLE TRENDS KEYWORDS (if trends in deliverables)
If "trends" is in expected_deliverables OR purpose is "explore_trends":
- Suggest 1-3 optimized keywords for Google Trends analysis
- These may differ from research queries (trends need broader, searchable terms)
- Consider: What keywords will show meaningful trends over time?
- Consider: What timeframe will show relevant trends?
- Consider: What geographic region is most relevant?
- Explain what insights trends will uncover for content generation

5. Enhanced API Router ⭐

File: backend/api/research/router.py

Enhancements:

• ✅ Added trends_config to AnalyzeIntentResponse
• ✅ Added trends_config to IntentDrivenResearchRequest
• ✅ Added google_trends_data to IntentDrivenResearchResponse
• ✅ Parallel execution of research + trends
• ✅ Trends data merging into results
• ✅ Helper function _merge_trends_data()

Parallel Execution:

# Execute research and trends in parallel
research_task = asyncio.create_task(engine.research(context))
trends_task = asyncio.create_task(trends_service.analyze_trends(...))

# Wait for both
raw_result = await research_task
trends_data = await trends_task

---

🎯 Design Decisions Made

Decision 1: Extend Unified Prompt ✅

Answer: Extended UnifiedResearchAnalyzer to include trends keyword suggestions

Rationale:

• Maintains single LLM call pattern
• Coherent reasoning across research + trends
• Consistent with Exa/Tavily optimization approach
• Trends keywords align with research intent

Decision 2: Parallel Execution ✅

Answer: Execute trends in parallel with research

Implementation:

• Use asyncio.create_task() for both
• Use asyncio.gather() or await sequentially
• Merge trends data into results after both complete

Decision 3: Trends Config Display ✅

Answer: Show in IntentConfirmationPanel with expected insights

What User Sees:

• Trends keywords (AI-suggested, editable)
• Timeframe & geo (with justifications)
• Expected insights preview (what trends will uncover)

---

📊 Data Flow

User Input → UnifiedResearchAnalyzer
     │
     ├── Infers Intent
     ├── Generates Research Queries
     ├── Optimizes Exa/Tavily Params
     └── Suggests Trends Keywords ← NEW
     │
     ▼
IntentConfirmationPanel
     ├── Shows Intent
     ├── Shows Research Queries
     ├── Shows Exa/Tavily Settings
     └── Shows Trends Config ← NEW
          ├── Keywords (editable)
          ├── Timeframe & Geo (with justifications)
          └── Expected Insights Preview
     │
     ▼
User Clicks "Research"
     │
     ▼
Parallel Execution
     ├── Research Task (Exa/Tavily/Google)
     └── Trends Task (Google Trends) ← NEW
     │
     ▼
Merge Results
     ├── Analyze Research Results
     └── Merge Trends Data ← NEW
     │
     ▼
IntentResultsDisplay
     └── Enhanced Trends Tab ← TODO (Frontend)

---

🔧 Technical Implementation

Service Structure

backend/services/research/trends/
├── __init__.py
├── google_trends_service.py  ✅ Created
└── rate_limiter.py           ✅ Created

Key Features

1. Async Support: All methods are async, use asyncio.to_thread() for pytrends
2. Rate Limiting: 1 request per second (prevents Google blocking)
3. Caching: 24-hour TTL (trends don't change frequently)
4. Error Handling: Graceful fallback, partial data return
5. Data Serialization: Converts DataFrames to dicts for API responses

Integration Points

1. UnifiedResearchAnalyzer: Extended prompt and schema
2. API Router: Parallel execution and data merging
3. Response Models: Added trends_config and google_trends_data

---

📝 Next Steps (Frontend Integration)

Phase 2: Frontend Updates

1. Update Types:

   • Add trends_config to AnalyzeIntentResponse type
   • Add google_trends_data to IntentDrivenResearchResponse type

2. Enhance IntentConfirmationPanel:

   • Add trends section (accordion)
   • Show trends keywords (editable)
   • Show expected insights preview
   • Show timeframe & geo with justifications

3. Enhance IntentResultsDisplay:

   • Add interest over time chart
   • Add interest by region table/map
   • Add related topics/queries display
   • Merge with AI-extracted trends

---

✅ Testing Checklist

Backend Testing

• Test GoogleTrendsService.analyze_trends() with sample keywords
• Test rate limiting (multiple rapid requests)
• Test caching (same keywords return cached data)
• Test error handling (invalid keywords, API failures)
• Test parallel execution (research + trends)
• Test data merging (trends data in results)

Integration Testing

• Test intent analysis with trends in deliverables
• Test trends_config in API response
• Test parallel execution in research endpoint
• Test trends data in final response

---

🚀 Usage Example

Backend Usage

from services.research.trends.google_trends_service import GoogleTrendsService

service = GoogleTrendsService()
trends_data = await service.analyze_trends(
    keywords=["AI marketing", "marketing automation"],
    timeframe="today 12-m",
    geo="US",
    user_id=user_id
)

# Returns:
# {
#   "interest_over_time": [...],
#   "interest_by_region": [...],
#   "related_topics": {"top": [...], "rising": [...]},
#   "related_queries": {"top": [...], "rising": [...]},
#   "timeframe": "today 12-m",
#   "geo": "US",
#   "keywords": ["AI marketing", "marketing automation"],
#   "timestamp": "2025-01-29T...",
#   "cached": false
# }

API Usage

POST /api/research/intent/analyze
{
  "user_input": "AI marketing tools for small businesses",
  "keywords": ["AI", "marketing", "tools"]
}

Response:
{
  "success": true,
  "intent": {...},
  "trends_config": {
    "enabled": true,
    "keywords": ["AI marketing", "marketing automation"],
    "keywords_justification": "These keywords will show search interest trends...",
    "timeframe": "today 12-m",
    "timeframe_justification": "12 months provides enough data...",
    "geo": "US",
    "geo_justification": "US market is most relevant...",
    "expected_insights": [
      "Search interest trends over the past year",
      "Regional interest distribution",
      "Related topics for content expansion",
      "Related queries for FAQ sections",
      "Optimal publication timing based on interest peaks"
    ]
  }
}

---

📋 Dependencies

Required Package

# requirements.txt
pytrends>=4.9.2  # Google Trends API

Installation

pip install pytrends>=4.9.2

---

⚠️ Known Limitations

1. Pytrends Rate Limits: Google Trends API is rate-limited (1 req/sec)

   • Mitigation: Rate limiter implemented, caching reduces API calls

2. Data Availability: Some keywords may have insufficient data

   • Mitigation: Graceful fallback, return partial data if available

3. Geographic Limitations: Some regions may have limited data

   • Mitigation: Default to "US" if region unavailable

---

🎯 Success Metrics

• Google Trends service created and working
• Rate limiting preventing blocks
• Caching working (24-hour TTL)
• Error handling graceful
• Parallel execution implemented
• Data merging working
• Frontend integration (Phase 2)
• User testing and feedback

---

📝 Files Created/Modified

Created:

• ✅ backend/services/research/trends/__init__.py
• ✅ backend/services/research/trends/google_trends_service.py
• ✅ backend/services/research/trends/rate_limiter.py
• ✅ backend/models/research_trends_models.py

Modified:

• ✅ backend/services/research/intent/unified_research_analyzer.py
• ✅ backend/api/research/router.py

---

Status: Phase 1 Complete - Core Service Ready

Next: Phase 2 - Frontend Integration (IntentConfirmationPanel + IntentResultsDisplay)