86 KiB
86 KiB
Google Integration Implementation Plan
This document outlines the step-by-step implementation plan for integrating Google login and Google Search Console (GSC) with AI-Writer to enhance content creation with real user insights.
Overview
The integration will allow users to:
- Sign in with their Google account
- Connect to Google Search Console
- Access search analytics data for content optimization
- Use real keyword data for content creation
- Track content performance over time
Implementation Steps
Phase 1: Google OAuth Integration
Step 1: Set Up Google Cloud Project
-
Create a Google Cloud Project
- Go to Google Cloud Console
- Create a new project named "AI-Writer"
- Note the Project ID for configuration
-
Configure OAuth Consent Screen
- Navigate to "APIs & Services" > "OAuth consent screen"
- Select "External" user type
- Fill in application information:
- App name: "AI-Writer"
- User support email: support@alwrity.com
- Developer contact information
- Add scopes:
https://www.googleapis.com/auth/userinfo.emailhttps://www.googleapis.com/auth/userinfo.profilehttps://www.googleapis.com/auth/webmasters.readonly
- Add test users for development
-
Create OAuth Credentials
- Navigate to "APIs & Services" > "Credentials"
- Click "Create Credentials" > "OAuth client ID"
- Select "Web application" as application type
- Set name to "AI-Writer Web Client"
- Add authorized JavaScript origins:
http://localhost:8501(for development)https://your-production-domain.com(for production)
- Add authorized redirect URIs:
http://localhost:8501/oauth/callback(for development)https://your-production-domain.com/oauth/callback(for production)
- Save and note the Client ID and Client Secret
-
Enable Required APIs
- Navigate to "APIs & Services" > "Library"
- Search for and enable:
- Google Search Console API
- Google OAuth2 API
- Google People API
Step 2: Implement Backend Authentication
- Install Required Packages
pip install google-auth google-auth-oauthlib google-auth-httplib2 google-api-python-client
- Create Authentication Module
Create a new file at /workspace/AI-Writer/lib/integrations/google/auth.py:
"""Google authentication module for AI-Writer."""
import os
import json
from pathlib import Path
from typing import Dict, Optional, Tuple, Any
import streamlit as st
from google_auth_oauthlib.flow import Flow
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from loguru import logger
# Define scopes needed for the application
SCOPES = [
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/webmasters.readonly'
]
# Configuration directory
CONFIG_DIR = Path(__file__).parent.parent.parent.parent / 'config'
CREDENTIALS_FILE = CONFIG_DIR / 'google_credentials.json'
TOKENS_DIR = CONFIG_DIR / 'tokens'
def get_google_auth_url() -> str:
"""Generate Google OAuth authorization URL.
Returns:
str: Authorization URL for Google OAuth
"""
try:
# Ensure config directory exists
CONFIG_DIR.mkdir(exist_ok=True)
TOKENS_DIR.mkdir(exist_ok=True)
# Load client configuration
client_config = st.secrets.get("google_oauth", None)
if not client_config:
logger.error("Google OAuth client configuration not found in secrets")
return ""
# Create OAuth flow instance
flow = Flow.from_client_config(
client_config=client_config,
scopes=SCOPES,
redirect_uri=client_config["web"]["redirect_uris"][0]
)
# Generate authorization URL
auth_url, _ = flow.authorization_url(
access_type='offline',
include_granted_scopes='true',
prompt='consent'
)
# Store flow in session state for later use
st.session_state.google_auth_flow = flow
return auth_url
except Exception as e:
logger.error(f"Error generating Google auth URL: {str(e)}")
return ""
def handle_auth_callback(code: str) -> Tuple[bool, Optional[Dict[str, Any]]]:
"""Handle OAuth callback and exchange code for tokens.
Args:
code: Authorization code from Google
Returns:
Tuple[bool, Optional[Dict]]: Success status and user info if successful
"""
try:
# Get flow from session state
flow = st.session_state.get("google_auth_flow")
if not flow:
logger.error("Auth flow not found in session state")
return False, None
# Exchange code for tokens
flow.fetch_token(code=code)
# Get credentials
credentials = flow.credentials
# Save credentials
save_credentials(credentials)
# Get user info
user_info = get_user_info(credentials)
# Store in session state
st.session_state.google_credentials = credentials_to_dict(credentials)
st.session_state.google_user_info = user_info
return True, user_info
except Exception as e:
logger.error(f"Error handling auth callback: {str(e)}")
return False, None
def save_credentials(credentials: Credentials) -> bool:
"""Save Google credentials to file.
Args:
credentials: Google OAuth credentials
Returns:
bool: Success status
"""
try:
# Convert credentials to dict
creds_dict = credentials_to_dict(credentials)
# Get user email from credentials
user_info = get_user_info(credentials)
user_email = user_info.get("email", "unknown")
# Create user-specific token file
token_file = TOKENS_DIR / f"{user_email}.json"
# Save credentials to file
with open(token_file, 'w') as f:
json.dump(creds_dict, f)
logger.info(f"Saved credentials for {user_email}")
return True
except Exception as e:
logger.error(f"Error saving credentials: {str(e)}")
return False
def load_credentials(user_email: str) -> Optional[Credentials]:
"""Load Google credentials from file.
Args:
user_email: Email of the user
Returns:
Optional[Credentials]: Google credentials if found
"""
try:
# Get user-specific token file
token_file = TOKENS_DIR / f"{user_email}.json"
# Check if file exists
if not token_file.exists():
logger.warning(f"No credentials found for {user_email}")
return None
# Load credentials from file
with open(token_file, 'r') as f:
creds_dict = json.load(f)
# Create credentials object
credentials = Credentials.from_authorized_user_info(creds_dict, SCOPES)
# Check if credentials are valid
if credentials.expired and credentials.refresh_token:
credentials.refresh(Request())
save_credentials(credentials)
return credentials
except Exception as e:
logger.error(f"Error loading credentials: {str(e)}")
return None
def get_user_info(credentials: Credentials) -> Dict[str, Any]:
"""Get user information from Google.
Args:
credentials: Google OAuth credentials
Returns:
Dict[str, Any]: User information
"""
try:
# Build people API service
service = build('people', 'v1', credentials=credentials)
# Get user profile
profile = service.people().get(
resourceName='people/me',
personFields='names,emailAddresses,photos'
).execute()
# Extract relevant information
user_info = {
"email": profile.get("emailAddresses", [{}])[0].get("value", ""),
"name": profile.get("names", [{}])[0].get("displayName", ""),
"picture": profile.get("photos", [{}])[0].get("url", "")
}
return user_info
except Exception as e:
logger.error(f"Error getting user info: {str(e)}")
return {}
def credentials_to_dict(credentials: Credentials) -> Dict[str, Any]:
"""Convert Google credentials to dictionary.
Args:
credentials: Google OAuth credentials
Returns:
Dict[str, Any]: Credentials as dictionary
"""
return {
'token': credentials.token,
'refresh_token': credentials.refresh_token,
'token_uri': credentials.token_uri,
'client_id': credentials.client_id,
'client_secret': credentials.client_secret,
'scopes': credentials.scopes
}
def is_authenticated() -> bool:
"""Check if user is authenticated with Google.
Returns:
bool: True if authenticated, False otherwise
"""
return 'google_credentials' in st.session_state and 'google_user_info' in st.session_state
def logout() -> None:
"""Log out user from Google."""
if 'google_credentials' in st.session_state:
del st.session_state.google_credentials
if 'google_user_info' in st.session_state:
del st.session_state.google_user_info
if 'google_auth_flow' in st.session_state:
del st.session_state.google_auth_flow
- Create OAuth Callback Handler
Create a new file at /workspace/AI-Writer/lib/integrations/google/callback_handler.py:
"""Google OAuth callback handler for Streamlit."""
import streamlit as st
from urllib.parse import parse_qs, urlparse
from .auth import handle_auth_callback
def handle_oauth_callback():
"""Handle OAuth callback in Streamlit."""
# Get current URL
query_params = st.experimental_get_query_params()
# Check if this is a callback
if 'code' in query_params:
code = query_params['code'][0]
# Exchange code for tokens
success, user_info = handle_auth_callback(code)
if success:
# Clear query parameters to avoid reprocessing
st.experimental_set_query_params()
# Show success message
st.success(f"Successfully logged in as {user_info.get('name', 'User')}")
# Redirect to main page
st.experimental_rerun()
else:
st.error("Failed to authenticate with Google. Please try again.")
# Check for error
if 'error' in query_params:
error = query_params['error'][0]
st.error(f"Authentication error: {error}")
st.experimental_set_query_params()
Step 3: Implement Google Search Console API
- Create GSC API Module
Create a new file at /workspace/AI-Writer/lib/integrations/google/search_console.py:
"""Google Search Console API integration for AI-Writer."""
from typing import Dict, List, Optional, Any
from datetime import datetime, timedelta
import streamlit as st
from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
from loguru import logger
def get_search_console_service(credentials: Dict[str, Any]) -> Any:
"""Build Google Search Console API service.
Args:
credentials: Google OAuth credentials dictionary
Returns:
Any: Search Console API service
"""
try:
# Convert dict to credentials object
creds = Credentials.from_authorized_user_info(credentials)
# Build service
service = build('searchconsole', 'v1', credentials=creds)
return service
except Exception as e:
logger.error(f"Error building Search Console service: {str(e)}")
return None
def get_site_list() -> List[Dict[str, Any]]:
"""Get list of sites from Search Console.
Returns:
List[Dict[str, Any]]: List of sites
"""
try:
# Check if user is authenticated
if 'google_credentials' not in st.session_state:
logger.warning("User not authenticated with Google")
return []
# Get credentials
credentials = st.session_state.google_credentials
# Build service
service = get_search_console_service(credentials)
if not service:
return []
# Get site list
sites = service.sites().list().execute()
# Extract site entries
site_entries = sites.get('siteEntry', [])
# Filter for verified sites
verified_sites = [
{
'site_url': site.get('siteUrl', ''),
'permission_level': site.get('permissionLevel', ''),
'site_type': 'Web' if site.get('siteUrl', '').startswith('http') else 'Domain Property'
}
for site in site_entries
if site.get('permissionLevel') in ['siteOwner', 'siteFullUser']
]
return verified_sites
except Exception as e:
logger.error(f"Error getting site list: {str(e)}")
return []
def get_search_analytics(
site_url: str,
start_date: datetime,
end_date: datetime,
dimensions: List[str] = ['query'],
row_limit: int = 1000
) -> Dict[str, Any]:
"""Get search analytics data from Search Console.
Args:
site_url: URL of the site
start_date: Start date for data
end_date: End date for data
dimensions: Dimensions to include (query, page, device, country, date)
row_limit: Maximum number of rows to return
Returns:
Dict[str, Any]: Search analytics data
"""
try:
# Check if user is authenticated
if 'google_credentials' not in st.session_state:
logger.warning("User not authenticated with Google")
return {'rows': []}
# Get credentials
credentials = st.session_state.google_credentials
# Build service
service = get_search_console_service(credentials)
if not service:
return {'rows': []}
# Format dates
start_date_str = start_date.strftime('%Y-%m-%d')
end_date_str = end_date.strftime('%Y-%m-%d')
# Prepare request body
request_body = {
'startDate': start_date_str,
'endDate': end_date_str,
'dimensions': dimensions,
'rowLimit': row_limit,
'startRow': 0,
'searchType': 'web'
}
# Execute request
response = service.searchanalytics().query(
siteUrl=site_url,
body=request_body
).execute()
return response
except Exception as e:
logger.error(f"Error getting search analytics: {str(e)}")
return {'rows': []}
def get_top_keywords(site_url: str, days: int = 30, limit: int = 100) -> List[Dict[str, Any]]:
"""Get top keywords for a site.
Args:
site_url: URL of the site
days: Number of days to include
limit: Maximum number of keywords to return
Returns:
List[Dict[str, Any]]: List of top keywords with metrics
"""
try:
# Calculate date range
end_date = datetime.now()
start_date = end_date - timedelta(days=days)
# Get search analytics data
analytics = get_search_analytics(
site_url=site_url,
start_date=start_date,
end_date=end_date,
dimensions=['query'],
row_limit=limit
)
# Process rows
rows = analytics.get('rows', [])
# Format results
keywords = []
for row in rows:
keywords.append({
'keyword': row.get('keys', [''])[0],
'clicks': row.get('clicks', 0),
'impressions': row.get('impressions', 0),
'ctr': row.get('ctr', 0) * 100, # Convert to percentage
'position': row.get('position', 0)
})
# Sort by clicks (descending)
keywords.sort(key=lambda x: x['clicks'], reverse=True)
return keywords
except Exception as e:
logger.error(f"Error getting top keywords: {str(e)}")
return []
def get_top_pages(site_url: str, days: int = 30, limit: int = 100) -> List[Dict[str, Any]]:
"""Get top pages for a site.
Args:
site_url: URL of the site
days: Number of days to include
limit: Maximum number of pages to return
Returns:
List[Dict[str, Any]]: List of top pages with metrics
"""
try:
# Calculate date range
end_date = datetime.now()
start_date = end_date - timedelta(days=days)
# Get search analytics data
analytics = get_search_analytics(
site_url=site_url,
start_date=start_date,
end_date=end_date,
dimensions=['page'],
row_limit=limit
)
# Process rows
rows = analytics.get('rows', [])
# Format results
pages = []
for row in rows:
pages.append({
'page': row.get('keys', [''])[0],
'clicks': row.get('clicks', 0),
'impressions': row.get('impressions', 0),
'ctr': row.get('ctr', 0) * 100, # Convert to percentage
'position': row.get('position', 0)
})
# Sort by clicks (descending)
pages.sort(key=lambda x: x['clicks'], reverse=True)
return pages
except Exception as e:
logger.error(f"Error getting top pages: {str(e)}")
return []
def get_keyword_insights(keyword: str, site_url: str, days: int = 90) -> Dict[str, Any]:
"""Get detailed insights for a specific keyword.
Args:
keyword: Keyword to analyze
site_url: URL of the site
days: Number of days to include
Returns:
Dict[str, Any]: Keyword insights
"""
try:
# Calculate date range
end_date = datetime.now()
start_date = end_date - timedelta(days=days)
# Get search analytics data with date dimension
analytics = get_search_analytics(
site_url=site_url,
start_date=start_date,
end_date=end_date,
dimensions=['query', 'date'],
row_limit=1000
)
# Process rows
rows = analytics.get('rows', [])
# Filter for the specific keyword
keyword_rows = [
row for row in rows
if row.get('keys', ['', ''])[0].lower() == keyword.lower()
]
# Prepare time series data
time_series = []
for row in keyword_rows:
date_str = row.get('keys', ['', ''])[1]
time_series.append({
'date': date_str,
'clicks': row.get('clicks', 0),
'impressions': row.get('impressions', 0),
'ctr': row.get('ctr', 0) * 100,
'position': row.get('position', 0)
})
# Sort by date
time_series.sort(key=lambda x: x['date'])
# Get pages ranking for this keyword
page_analytics = get_search_analytics(
site_url=site_url,
start_date=start_date,
end_date=end_date,
dimensions=['query', 'page'],
row_limit=100
)
# Filter for the specific keyword
keyword_pages = [
{
'page': row.get('keys', ['', ''])[1],
'clicks': row.get('clicks', 0),
'impressions': row.get('impressions', 0),
'ctr': row.get('ctr', 0) * 100,
'position': row.get('position', 0)
}
for row in page_analytics.get('rows', [])
if row.get('keys', ['', ''])[0].lower() == keyword.lower()
]
# Sort by position (ascending)
keyword_pages.sort(key=lambda x: x['position'])
# Calculate totals
total_clicks = sum(row.get('clicks', 0) for row in keyword_rows)
total_impressions = sum(row.get('impressions', 0) for row in keyword_rows)
avg_ctr = (sum(row.get('ctr', 0) for row in keyword_rows) / len(keyword_rows)) * 100 if keyword_rows else 0
avg_position = sum(row.get('position', 0) for row in keyword_rows) / len(keyword_rows) if keyword_rows else 0
# Compile insights
insights = {
'keyword': keyword,
'total_clicks': total_clicks,
'total_impressions': total_impressions,
'avg_ctr': avg_ctr,
'avg_position': avg_position,
'time_series': time_series,
'pages': keyword_pages
}
return insights
except Exception as e:
logger.error(f"Error getting keyword insights: {str(e)}")
return {
'keyword': keyword,
'total_clicks': 0,
'total_impressions': 0,
'avg_ctr': 0,
'avg_position': 0,
'time_series': [],
'pages': []
}
Step 4: Create UI Components
- Create Login Component
Create a new file at /workspace/AI-Writer/lib/integrations/google/components.py:
"""UI components for Google integration."""
import streamlit as st
from .auth import get_google_auth_url, is_authenticated, logout
from .search_console import get_site_list, get_top_keywords, get_top_pages, get_keyword_insights
def render_google_login_button():
"""Render Google login button."""
if not is_authenticated():
auth_url = get_google_auth_url()
st.markdown("""
<style>
.google-button {
display: inline-flex;
align-items: center;
background-color: #4285F4;
color: white;
border: none;
border-radius: 4px;
padding: 0;
cursor: pointer;
font-family: 'Roboto', sans-serif;
font-size: 14px;
font-weight: 500;
height: 40px;
transition: background-color 0.3s;
}
.google-button:hover {
background-color: #357AE8;
}
.google-button:active {
background-color: #2A67D4;
}
.google-icon-wrapper {
display: flex;
align-items: center;
justify-content: center;
width: 40px;
height: 40px;
background-color: white;
border-radius: 2px;
margin-right: 1px;
}
.google-icon {
width: 18px;
height: 18px;
}
.google-text {
padding: 0 16px;
}
</style>
<a href="{auth_url}" class="google-button">
<div class="google-icon-wrapper">
<img class="google-icon" src="https://upload.wikimedia.org/wikipedia/commons/5/53/Google_%22G%22_Logo.svg"/>
</div>
<span class="google-text">Sign in with Google</span>
</a>
""".format(auth_url=auth_url), unsafe_allow_html=True)
else:
user_info = st.session_state.google_user_info
# Display user info
col1, col2 = st.columns([1, 3])
with col1:
st.image(user_info.get('picture', ''), width=50)
with col2:
st.markdown(f"**{user_info.get('name', 'User')}**")
st.markdown(f"{user_info.get('email', '')}")
# Logout button
if st.button("Sign Out"):
logout()
st.experimental_rerun()
def render_site_selector():
"""Render Google Search Console site selector."""
if not is_authenticated():
st.warning("Please sign in with Google to access Search Console data.")
return None
# Get site list
sites = get_site_list()
if not sites:
st.warning("No verified sites found in your Search Console account.")
return None
# Create site options
site_options = [site['site_url'] for site in sites]
# Select site
selected_site = st.selectbox(
"Select a website",
options=site_options,
index=0 if site_options else None,
format_func=lambda x: x.replace("sc-domain:", "")
)
return selected_site
def render_search_console_dashboard(site_url):
"""Render Google Search Console dashboard.
Args:
site_url: URL of the selected site
"""
if not site_url:
return
st.markdown("## Search Console Insights")
# Create tabs
tab1, tab2, tab3 = st.tabs(["Top Keywords", "Top Pages", "Keyword Research"])
with tab1:
st.markdown("### Top Keywords")
# Date range selector
days = st.slider("Time period (days)", min_value=7, max_value=90, value=30, step=1)
# Get top keywords
with st.spinner("Loading keyword data..."):
keywords = get_top_keywords(site_url, days=days, limit=100)
if not keywords:
st.info("No keyword data available for this site.")
return
# Display keywords table
st.dataframe(
keywords,
column_config={
"keyword": "Keyword",
"clicks": st.column_config.NumberColumn("Clicks", format="%d"),
"impressions": st.column_config.NumberColumn("Impressions", format="%d"),
"ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
"position": st.column_config.NumberColumn("Position", format="%.1f")
},
hide_index=True
)
with tab2:
st.markdown("### Top Pages")
# Date range selector
days = st.slider("Time period (days)", min_value=7, max_value=90, value=30, step=1, key="pages_days")
# Get top pages
with st.spinner("Loading page data..."):
pages = get_top_pages(site_url, days=days, limit=100)
if not pages:
st.info("No page data available for this site.")
return
# Display pages table
st.dataframe(
pages,
column_config={
"page": "Page URL",
"clicks": st.column_config.NumberColumn("Clicks", format="%d"),
"impressions": st.column_config.NumberColumn("Impressions", format="%d"),
"ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
"position": st.column_config.NumberColumn("Position", format="%.1f")
},
hide_index=True
)
with tab3:
st.markdown("### Keyword Research")
# Keyword input
keyword = st.text_input("Enter a keyword to analyze")
if keyword:
# Get keyword insights
with st.spinner(f"Analyzing '{keyword}'..."):
insights = get_keyword_insights(keyword, site_url, days=90)
# Display insights
col1, col2, col3, col4 = st.columns(4)
with col1:
st.metric("Clicks", f"{insights['total_clicks']:,}")
with col2:
st.metric("Impressions", f"{insights['total_impressions']:,}")
with col3:
st.metric("Avg. CTR", f"{insights['avg_ctr']:.2f}%")
with col4:
st.metric("Avg. Position", f"{insights['avg_position']:.1f}")
# Time series chart
if insights['time_series']:
st.markdown("#### Performance Over Time")
# Prepare data for chart
chart_data = {
'date': [item['date'] for item in insights['time_series']],
'clicks': [item['clicks'] for item in insights['time_series']],
'impressions': [item['impressions'] for item in insights['time_series']],
'position': [item['position'] for item in insights['time_series']]
}
# Create tabs for different metrics
chart_tab1, chart_tab2, chart_tab3 = st.tabs(["Clicks", "Impressions", "Position"])
with chart_tab1:
st.line_chart(chart_data, x='date', y='clicks')
with chart_tab2:
st.line_chart(chart_data, x='date', y='impressions')
with chart_tab3:
# Invert position axis (lower is better)
position_chart = {
'date': chart_data['date'],
'position': [-pos for pos in chart_data['position']] # Invert values
}
st.line_chart(position_chart, x='date', y='position')
st.caption("Note: Position axis is inverted (higher is better)")
# Ranking pages
if insights['pages']:
st.markdown("#### Pages Ranking for this Keyword")
st.dataframe(
insights['pages'],
column_config={
"page": "Page URL",
"clicks": st.column_config.NumberColumn("Clicks", format="%d"),
"impressions": st.column_config.NumberColumn("Impressions", format="%d"),
"ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
"position": st.column_config.NumberColumn("Position", format="%.1f")
},
hide_index=True
)
# Content suggestions
st.markdown("#### Content Suggestions")
if st.button("Generate Content Ideas"):
with st.spinner("Generating content ideas..."):
# This would connect to your AI content generation system
# For now, we'll just show a placeholder
st.info("This would connect to your AI content generation system to create content ideas based on this keyword's performance data.")
- Create Main Integration Page
Create a new file at /workspace/AI-Writer/lib/integrations/google/page.py:
"""Google integration page for AI-Writer."""
import streamlit as st
from .components import render_google_login_button, render_site_selector, render_search_console_dashboard
from .callback_handler import handle_oauth_callback
def render_google_integration_page():
"""Render the Google integration page."""
st.title("Google Integration")
# Handle OAuth callback if present
handle_oauth_callback()
# Display login/user info
st.markdown("### Google Account")
render_google_login_button()
# Separator
st.markdown("---")
# Search Console section
st.markdown("### Google Search Console")
st.markdown("""
Connect to Google Search Console to access search analytics data for your websites.
This data will help you optimize your content for better search visibility.
""")
# Site selector
selected_site = render_site_selector()
# Display dashboard if site is selected
if selected_site:
render_search_console_dashboard(selected_site)
Step 5: Update Main Application
- Add Google Integration to Sidebar
Update the sidebar in the main application file (alwrity.py):
# Add to imports
from lib.integrations.google.page import render_google_integration_page
# Add to sidebar menu
with st.sidebar:
st.title("AI-Writer")
# Existing menu items...
# Add Google Integration option
if st.sidebar.selectbox("Integrations", ["None", "Google"]) == "Google":
page = "google_integration"
# Existing sidebar code...
# Add to page router
if page == "google_integration":
render_google_integration_page()
- Add Google Credentials to Streamlit Secrets
Create a file at /.streamlit/secrets.toml:
[google_oauth]
web = {
"client_id": "YOUR_CLIENT_ID",
"project_id": "ai-writer",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_secret": "YOUR_CLIENT_SECRET",
"redirect_uris": ["http://localhost:8501/oauth/callback"],
"javascript_origins": ["http://localhost:8501"]
}
Phase 2: Content Enhancement with GSC Data
Step 1: Create Keyword Research Module
Create a new file at /workspace/AI-Writer/lib/integrations/google/keyword_research.py:
"""Keyword research module using Google Search Console data."""
from typing import Dict, List, Any, Optional
import streamlit as st
from loguru import logger
from .search_console import get_top_keywords, get_keyword_insights
def get_content_opportunities(site_url: str, min_impressions: int = 100, max_position: float = 20.0) -> List[Dict[str, Any]]:
"""Find content opportunities based on GSC data.
Args:
site_url: URL of the site
min_impressions: Minimum impressions threshold
max_position: Maximum position threshold
Returns:
List[Dict[str, Any]]: List of content opportunities
"""
try:
# Get top keywords
keywords = get_top_keywords(site_url, days=90, limit=1000)
# Filter for opportunities
opportunities = [
keyword for keyword in keywords
if keyword['impressions'] >= min_impressions
and keyword['position'] <= max_position
and keyword['position'] > 3 # Not already in top 3
]
# Sort by potential (impressions * (11 - position)/10)
# This prioritizes keywords with high impressions and positions 4-10
for opp in opportunities:
opp['potential'] = opp['impressions'] * (11 - min(opp['position'], 10)) / 10
opportunities.sort(key=lambda x: x['potential'], reverse=True)
return opportunities[:100] # Return top 100 opportunities
except Exception as e:
logger.error(f"Error finding content opportunities: {str(e)}")
return []
def get_keyword_clusters(keywords: List[Dict[str, Any]], threshold: int = 3) -> List[Dict[str, Any]]:
"""Group keywords into clusters based on common words.
Args:
keywords: List of keywords with metrics
threshold: Minimum number of keywords to form a cluster
Returns:
List[Dict[str, Any]]: List of keyword clusters
"""
try:
# Extract keyword strings
keyword_strings = [k['keyword'].lower() for k in keywords]
# Create word frequency map
word_map = {}
for kw in keyword_strings:
words = kw.split()
for word in words:
if len(word) > 3: # Ignore short words
if word not in word_map:
word_map[word] = []
word_map[word].append(kw)
# Find clusters
clusters = []
for word, kws in word_map.items():
if len(kws) >= threshold:
# Get full keyword data for each keyword in cluster
cluster_data = [
k for k in keywords
if k['keyword'].lower() in kws
]
# Calculate cluster metrics
total_impressions = sum(k['impressions'] for k in cluster_data)
total_clicks = sum(k['clicks'] for k in cluster_data)
avg_position = sum(k['position'] for k in cluster_data) / len(cluster_data)
clusters.append({
'topic': word,
'keywords': cluster_data,
'keyword_count': len(cluster_data),
'total_impressions': total_impressions,
'total_clicks': total_clicks,
'avg_position': avg_position
})
# Sort clusters by total impressions
clusters.sort(key=lambda x: x['total_impressions'], reverse=True)
return clusters
except Exception as e:
logger.error(f"Error creating keyword clusters: {str(e)}")
return []
def generate_content_brief(keyword: str, site_url: str) -> Dict[str, Any]:
"""Generate a content brief for a keyword based on GSC data.
Args:
keyword: Target keyword
site_url: URL of the site
Returns:
Dict[str, Any]: Content brief
"""
try:
# Get keyword insights
insights = get_keyword_insights(keyword, site_url, days=90)
# Get top keywords (for related keywords)
all_keywords = get_top_keywords(site_url, days=90, limit=1000)
# Find related keywords
related_keywords = [
k for k in all_keywords
if keyword.lower() in k['keyword'].lower() and k['keyword'].lower() != keyword.lower()
]
# Sort related keywords by impressions
related_keywords.sort(key=lambda x: x['impressions'], reverse=True)
# Create content brief
brief = {
'primary_keyword': keyword,
'search_metrics': {
'monthly_impressions': insights['total_impressions'] // 3, # Approximate monthly
'monthly_clicks': insights['total_clicks'] // 3,
'avg_position': insights['avg_position'],
'avg_ctr': insights['avg_ctr']
},
'related_keywords': related_keywords[:20], # Top 20 related keywords
'competing_pages': insights['pages'],
'recommended_headings': [], # Will be filled by AI
'content_suggestions': [] # Will be filled by AI
}
return brief
except Exception as e:
logger.error(f"Error generating content brief: {str(e)}")
return {
'primary_keyword': keyword,
'search_metrics': {
'monthly_impressions': 0,
'monthly_clicks': 0,
'avg_position': 0,
'avg_ctr': 0
},
'related_keywords': [],
'competing_pages': [],
'recommended_headings': [],
'content_suggestions': []
}
Step 2: Integrate with Content Generation
- Create GSC-Enhanced Content Generator
Create a new file at /workspace/AI-Writer/lib/integrations/google/content_generator.py:
"""Content generator enhanced with Google Search Console data."""
from typing import Dict, List, Any, Optional
import streamlit as st
from loguru import logger
from .keyword_research import generate_content_brief
from ...ai_writers.blog_writer import generate_blog_post
def generate_seo_optimized_content(
keyword: str,
site_url: str,
content_type: str = "blog",
length: str = "medium"
) -> Dict[str, Any]:
"""Generate SEO-optimized content using GSC data.
Args:
keyword: Target keyword
site_url: URL of the site
content_type: Type of content to generate
length: Content length
Returns:
Dict[str, Any]: Generated content
"""
try:
# Generate content brief
brief = generate_content_brief(keyword, site_url)
# Extract related keywords
related_keywords = [k['keyword'] for k in brief['related_keywords']]
# Prepare prompt enhancements
prompt_enhancements = {
'primary_keyword': brief['primary_keyword'],
'related_keywords': related_keywords,
'search_position': brief['search_metrics']['avg_position'],
'monthly_impressions': brief['search_metrics']['monthly_impressions'],
'competing_urls': [p['page'] for p in brief['competing_pages'][:5]]
}
# Generate content with enhanced prompt
content = generate_blog_post(
keyword=keyword,
content_type=content_type,
length=length,
prompt_enhancements=prompt_enhancements
)
# Add brief data to the result
content['brief'] = brief
return content
except Exception as e:
logger.error(f"Error generating SEO-optimized content: {str(e)}")
return {
'title': f"Error generating content for '{keyword}'",
'content': f"An error occurred: {str(e)}",
'brief': {}
}
- Create Content Optimization Component
Create a new file at /workspace/AI-Writer/lib/integrations/google/content_optimizer.py:
"""Content optimization using Google Search Console data."""
from typing import Dict, List, Any, Optional
import streamlit as st
from loguru import logger
from .search_console import get_keyword_insights
from ...seo_tools.analyzer import analyze_content_for_seo
def optimize_existing_content(
content: str,
keyword: str,
site_url: str
) -> Dict[str, Any]:
"""Optimize existing content using GSC data.
Args:
content: Existing content
keyword: Target keyword
site_url: URL of the site
Returns:
Dict[str, Any]: Optimization suggestions
"""
try:
# Get keyword insights
insights = get_keyword_insights(keyword, site_url, days=90)
# Analyze content for SEO
seo_analysis = analyze_content_for_seo(content, keyword)
# Generate optimization suggestions
suggestions = []
# Check keyword density
if seo_analysis['keyword_density'] < 0.5:
suggestions.append({
'type': 'keyword_density',
'severity': 'high',
'message': f"Keyword density is too low ({seo_analysis['keyword_density']:.2f}%). Aim for 1-2%.",
'action': "Add more instances of the keyword in a natural way."
})
elif seo_analysis['keyword_density'] > 3:
suggestions.append({
'type': 'keyword_density',
'severity': 'medium',
'message': f"Keyword density is too high ({seo_analysis['keyword_density']:.2f}%). Aim for 1-2%.",
'action': "Reduce keyword usage to avoid keyword stuffing."
})
# Check title
if keyword.lower() not in seo_analysis['title'].lower():
suggestions.append({
'type': 'title',
'severity': 'high',
'message': "Primary keyword is missing from the title.",
'action': f"Add '{keyword}' to the title in a natural way."
})
# Check headings
if not any(keyword.lower() in h.lower() for h in seo_analysis['headings']):
suggestions.append({
'type': 'headings',
'severity': 'medium',
'message': "Primary keyword is missing from all headings.",
'action': f"Add '{keyword}' to at least one heading (preferably H2)."
})
# Check content length
if seo_analysis['word_count'] < 300:
suggestions.append({
'type': 'content_length',
'severity': 'high',
'message': f"Content is too short ({seo_analysis['word_count']} words). Aim for at least 800 words.",
'action': "Expand the content with more valuable information."
})
elif seo_analysis['word_count'] < 800:
suggestions.append({
'type': 'content_length',
'severity': 'medium',
'message': f"Content is relatively short ({seo_analysis['word_count']} words). Aim for 1000+ words for competitive keywords.",
'action': "Consider adding more depth to the content."
})
# Check readability
if seo_analysis['readability_score'] < 50:
suggestions.append({
'type': 'readability',
'severity': 'medium',
'message': f"Content readability is low ({seo_analysis['readability_score']}/100).",
'action': "Simplify sentences, use shorter paragraphs, and avoid jargon."
})
# Check competing pages
competing_pages = insights['pages']
if competing_pages:
top_competing_page = competing_pages[0]
suggestions.append({
'type': 'competition',
'severity': 'info',
'message': f"Your top competing page ranks at position {top_competing_page['position']:.1f}.",
'action': f"Review the content at {top_competing_page['page']} for insights."
})
# Compile results
result = {
'keyword': keyword,
'search_metrics': {
'monthly_impressions': insights['total_impressions'] // 3,
'monthly_clicks': insights['total_clicks'] // 3,
'avg_position': insights['avg_position'],
'avg_ctr': insights['avg_ctr']
},
'seo_analysis': seo_analysis,
'suggestions': suggestions
}
return result
except Exception as e:
logger.error(f"Error optimizing content: {str(e)}")
return {
'keyword': keyword,
'search_metrics': {},
'seo_analysis': {},
'suggestions': [{
'type': 'error',
'severity': 'high',
'message': f"Error analyzing content: {str(e)}",
'action': "Please try again or contact support."
}]
}
Step 3: Create Content Planning Component
Create a new file at /workspace/AI-Writer/lib/integrations/google/content_planner.py:
"""Content planning using Google Search Console data."""
import streamlit as st
from typing import Dict, List, Any, Optional
from datetime import datetime, timedelta
from loguru import logger
from .keyword_research import get_content_opportunities, get_keyword_clusters
from .search_console import get_top_keywords
def render_content_planner(site_url: str):
"""Render content planning interface using GSC data.
Args:
site_url: URL of the site
"""
st.markdown("## Content Planning")
st.markdown("""
Use your Google Search Console data to plan your content strategy.
Identify opportunities, organize topics, and create a content calendar.
""")
# Create tabs
tab1, tab2, tab3 = st.tabs(["Content Opportunities", "Topic Clusters", "Content Calendar"])
with tab1:
st.markdown("### Content Opportunities")
st.markdown("""
Find keywords where you're ranking on page 1-2 but not in the top 3.
These are opportunities to improve existing content or create new content.
""")
# Filters
col1, col2 = st.columns(2)
with col1:
min_impressions = st.slider(
"Minimum monthly impressions",
min_value=10,
max_value=1000,
value=100,
step=10
)
with col2:
max_position = st.slider(
"Maximum position",
min_value=5.0,
max_value=30.0,
value=20.0,
step=1.0
)
# Get opportunities
if st.button("Find Opportunities"):
with st.spinner("Analyzing search data..."):
opportunities = get_content_opportunities(
site_url=site_url,
min_impressions=min_impressions,
max_position=max_position
)
if not opportunities:
st.info("No content opportunities found with the current filters.")
else:
# Display opportunities
st.markdown(f"Found {len(opportunities)} content opportunities:")
# Create dataframe
st.dataframe(
opportunities,
column_config={
"keyword": "Keyword",
"clicks": st.column_config.NumberColumn("Clicks", format="%d"),
"impressions": st.column_config.NumberColumn("Impressions", format="%d"),
"ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
"position": st.column_config.NumberColumn("Position", format="%.1f"),
"potential": st.column_config.NumberColumn("Potential", format="%.1f")
},
hide_index=True
)
# Action buttons for selected opportunity
st.markdown("### Take Action")
selected_keyword = st.selectbox(
"Select a keyword to take action on",
options=[o['keyword'] for o in opportunities]
)
col1, col2 = st.columns(2)
with col1:
if st.button("Create New Content", key="create_new"):
st.session_state.selected_keyword = selected_keyword
st.session_state.action = "create_content"
st.experimental_rerun()
with col2:
if st.button("Optimize Existing Content", key="optimize"):
st.session_state.selected_keyword = selected_keyword
st.session_state.action = "optimize_content"
st.experimental_rerun()
with tab2:
st.markdown("### Topic Clusters")
st.markdown("""
Group your keywords into topic clusters to organize your content strategy.
This helps identify themes and create comprehensive content around topics.
""")
# Cluster settings
min_cluster_size = st.slider(
"Minimum cluster size",
min_value=2,
max_value=10,
value=3,
step=1
)
# Get clusters
if st.button("Generate Topic Clusters"):
with st.spinner("Analyzing keywords..."):
# Get top keywords
keywords = get_top_keywords(site_url, days=90, limit=1000)
# Generate clusters
clusters = get_keyword_clusters(
keywords=keywords,
threshold=min_cluster_size
)
if not clusters:
st.info("No topic clusters found with the current settings.")
else:
# Display clusters
st.markdown(f"Found {len(clusters)} topic clusters:")
for i, cluster in enumerate(clusters):
with st.expander(f"Cluster: {cluster['topic']} ({cluster['keyword_count']} keywords)"):
st.markdown(f"**Total Impressions:** {cluster['total_impressions']:,}")
st.markdown(f"**Total Clicks:** {cluster['total_clicks']:,}")
st.markdown(f"**Average Position:** {cluster['avg_position']:.1f}")
# Display keywords in cluster
st.dataframe(
cluster['keywords'],
column_config={
"keyword": "Keyword",
"clicks": st.column_config.NumberColumn("Clicks", format="%d"),
"impressions": st.column_config.NumberColumn("Impressions", format="%d"),
"position": st.column_config.NumberColumn("Position", format="%.1f")
},
hide_index=True
)
# Action button
if st.button("Create Cluster Content", key=f"cluster_{i}"):
st.session_state.selected_cluster = cluster
st.session_state.action = "create_cluster_content"
st.experimental_rerun()
with tab3:
st.markdown("### Content Calendar")
st.markdown("""
Create a content calendar based on your search data and opportunities.
Plan your content strategy for the coming weeks or months.
""")
# Calendar settings
num_weeks = st.slider(
"Number of weeks to plan",
min_value=1,
max_value=12,
value=4,
step=1
)
posts_per_week = st.slider(
"Posts per week",
min_value=1,
max_value=7,
value=2,
step=1
)
# Generate calendar
if st.button("Generate Content Calendar"):
with st.spinner("Creating content calendar..."):
# Get opportunities
opportunities = get_content_opportunities(
site_url=site_url,
min_impressions=50,
max_position=30.0
)
# Sort by potential
opportunities.sort(key=lambda x: x['potential'], reverse=True)
# Calculate total posts needed
total_posts = num_weeks * posts_per_week
# Limit opportunities to needed posts
selected_opportunities = opportunities[:total_posts]
# Create calendar
calendar = []
start_date = datetime.now() + timedelta(days=7) # Start next week
for week in range(num_weeks):
week_start = start_date + timedelta(weeks=week)
for day in range(posts_per_week):
opp_index = week * posts_per_week + day
if opp_index < len(selected_opportunities):
post_date = week_start + timedelta(days=day)
calendar.append({
'date': post_date.strftime('%Y-%m-%d'),
'keyword': selected_opportunities[opp_index]['keyword'],
'impressions': selected_opportunities[opp_index]['impressions'],
'position': selected_opportunities[opp_index]['position'],
'potential': selected_opportunities[opp_index]['potential']
})
if not calendar:
st.info("Could not generate content calendar. Try adjusting your filters.")
else:
# Display calendar
st.markdown(f"Generated content calendar with {len(calendar)} posts:")
# Group by week
weeks = {}
for post in calendar:
post_date = datetime.strptime(post['date'], '%Y-%m-%d')
week_num = post_date.isocalendar()[1]
week_start = post_date - timedelta(days=post_date.weekday())
week_key = f"Week {week_num} ({week_start.strftime('%b %d')})"
if week_key not in weeks:
weeks[week_key] = []
weeks[week_key].append(post)
# Display by week
for week_key, posts in weeks.items():
with st.expander(week_key, expanded=True):
for post in posts:
st.markdown(f"**{post['date']}:** {post['keyword']}")
st.markdown(f"Impressions: {post['impressions']:,} | Position: {post['position']:.1f}")
st.markdown("---")
# Export button
if st.download_button(
label="Export Calendar (CSV)",
data="\n".join([
"date,keyword,impressions,position,potential",
*[f"{post['date']},{post['keyword']},{post['impressions']},{post['position']},{post['potential']}" for post in calendar]
]),
file_name="content_calendar.csv",
mime="text/csv"
):
st.success("Calendar exported successfully!")
Step 4: Integrate with Content Writers
- Update Blog Writer
Modify the existing blog writer to accept GSC data:
# Add to blog_writer.py
def generate_blog_post(
keyword: str,
content_type: str = "blog",
length: str = "medium",
prompt_enhancements: Dict[str, Any] = None
) -> Dict[str, Any]:
"""Generate a blog post with optional GSC data enhancement.
Args:
keyword: Target keyword
content_type: Type of content to generate
length: Content length
prompt_enhancements: Optional GSC data for prompt enhancement
Returns:
Dict[str, Any]: Generated content
"""
# Existing code...
# Enhance prompt with GSC data if available
if prompt_enhancements:
# Add related keywords to prompt
if 'related_keywords' in prompt_enhancements:
related_kw_str = ", ".join(prompt_enhancements['related_keywords'][:5])
prompt += f"\nInclude these related keywords naturally: {related_kw_str}"
# Add search position context
if 'search_position' in prompt_enhancements:
position = prompt_enhancements['search_position']
if position > 10:
prompt += f"\nThis keyword currently ranks at position {position:.1f}, which is on page 2. Create comprehensive content to improve ranking."
else:
prompt += f"\nThis keyword currently ranks at position {position:.1f}. Enhance the content to improve ranking further."
# Add impression data
if 'monthly_impressions' in prompt_enhancements:
impressions = prompt_enhancements['monthly_impressions']
prompt += f"\nThis keyword gets approximately {impressions} monthly impressions."
# Add competing URLs
if 'competing_urls' in prompt_enhancements:
prompt += "\nMake sure your content is more comprehensive than competing pages."
# Continue with existing code...
- Add GSC Data to SEO Tools
Update the SEO analyzer to incorporate GSC data:
# Add to seo_tools/analyzer.py
def analyze_content_with_gsc(
content: str,
keyword: str,
gsc_data: Dict[str, Any]
) -> Dict[str, Any]:
"""Analyze content with GSC data enhancement.
Args:
content: Content to analyze
keyword: Target keyword
gsc_data: Google Search Console data
Returns:
Dict[str, Any]: Enhanced analysis
"""
# Get basic SEO analysis
analysis = analyze_content_for_seo(content, keyword)
# Enhance with GSC data
if gsc_data:
# Add search metrics
analysis['search_metrics'] = {
'impressions': gsc_data.get('impressions', 0),
'clicks': gsc_data.get('clicks', 0),
'position': gsc_data.get('position', 0),
'ctr': gsc_data.get('ctr', 0)
}
# Add competition analysis
if 'competing_pages' in gsc_data:
analysis['competition'] = {
'competing_pages': gsc_data['competing_pages'],
'ranking_gap': analysis['word_count'] - gsc_data.get('avg_competitor_length', 0)
}
# Add keyword opportunity score
if 'position' in gsc_data and 'impressions' in gsc_data:
position = gsc_data['position']
impressions = gsc_data['impressions']
# Calculate opportunity score (higher for keywords with good impressions but not yet in top 3)
if position <= 3:
opportunity_score = 30 # Already ranking well
elif position <= 10:
opportunity_score = 70 # On first page but not top 3
elif position <= 20:
opportunity_score = 50 # On second page
else:
opportunity_score = 30 # Beyond second page
# Adjust for impressions
if impressions > 1000:
opportunity_score += 30
elif impressions > 500:
opportunity_score += 20
elif impressions > 100:
opportunity_score += 10
# Cap at 100
opportunity_score = min(opportunity_score, 100)
analysis['opportunity_score'] = opportunity_score
return analysis
Phase 3: Testing Plan
Step 1: Unit Testing
- Create Test Files
Create a new file at /workspace/AI-Writer/tests/integrations/google/test_auth.py:
"""Unit tests for Google authentication module."""
import unittest
from unittest.mock import patch, MagicMock
import os
import sys
import json
from pathlib import Path
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
from lib.integrations.google.auth import (
get_google_auth_url,
handle_auth_callback,
save_credentials,
load_credentials,
get_user_info,
credentials_to_dict,
is_authenticated
)
class TestGoogleAuth(unittest.TestCase):
"""Test cases for Google authentication module."""
def setUp(self):
"""Set up test environment."""
# Mock session state
self.session_state_patch = patch('streamlit.session_state', {})
self.mock_session_state = self.session_state_patch.start()
# Mock secrets
self.secrets_patch = patch('streamlit.secrets', {
'google_oauth': {
'web': {
'client_id': 'test_client_id',
'client_secret': 'test_client_secret',
'redirect_uris': ['http://localhost:8501/oauth/callback']
}
}
})
self.mock_secrets = self.secrets_patch.start()
def tearDown(self):
"""Clean up after tests."""
self.session_state_patch.stop()
self.secrets_patch.stop()
@patch('lib.integrations.google.auth.Flow')
def test_get_google_auth_url(self, mock_flow):
"""Test generating Google auth URL."""
# Mock flow instance
mock_flow_instance = MagicMock()
mock_flow_instance.authorization_url.return_value = ('https://test-auth-url.com', None)
mock_flow.from_client_config.return_value = mock_flow_instance
# Call function
url = get_google_auth_url()
# Assertions
self.assertEqual(url, 'https://test-auth-url.com')
mock_flow.from_client_config.assert_called_once()
mock_flow_instance.authorization_url.assert_called_once()
@patch('lib.integrations.google.auth.get_user_info')
def test_handle_auth_callback(self, mock_get_user_info):
"""Test handling auth callback."""
# Mock flow in session state
mock_flow = MagicMock()
mock_flow.credentials = MagicMock()
self.mock_session_state['google_auth_flow'] = mock_flow
# Mock user info
mock_get_user_info.return_value = {'email': 'test@example.com', 'name': 'Test User'}
# Call function
success, user_info = handle_auth_callback('test_code')
# Assertions
self.assertTrue(success)
self.assertEqual(user_info['email'], 'test@example.com')
mock_flow.fetch_token.assert_called_once_with(code='test_code')
mock_get_user_info.assert_called_once()
@patch('builtins.open', new_callable=unittest.mock.mock_open)
@patch('json.dump')
@patch('lib.integrations.google.auth.get_user_info')
def test_save_credentials(self, mock_get_user_info, mock_json_dump, mock_open):
"""Test saving credentials."""
# Mock credentials
mock_credentials = MagicMock()
mock_credentials.token = 'test_token'
mock_credentials.refresh_token = 'test_refresh_token'
# Mock user info
mock_get_user_info.return_value = {'email': 'test@example.com'}
# Call function
result = save_credentials(mock_credentials)
# Assertions
self.assertTrue(result)
mock_get_user_info.assert_called_once()
mock_open.assert_called_once()
mock_json_dump.assert_called_once()
def test_credentials_to_dict(self):
"""Test converting credentials to dictionary."""
# Mock credentials
mock_credentials = MagicMock()
mock_credentials.token = 'test_token'
mock_credentials.refresh_token = 'test_refresh_token'
mock_credentials.token_uri = 'test_token_uri'
mock_credentials.client_id = 'test_client_id'
mock_credentials.client_secret = 'test_client_secret'
mock_credentials.scopes = ['test_scope']
# Call function
result = credentials_to_dict(mock_credentials)
# Assertions
self.assertEqual(result['token'], 'test_token')
self.assertEqual(result['refresh_token'], 'test_refresh_token')
self.assertEqual(result['client_id'], 'test_client_id')
self.assertEqual(result['scopes'], ['test_scope'])
def test_is_authenticated(self):
"""Test authentication check."""
# Test not authenticated
self.assertFalse(is_authenticated())
# Test authenticated
self.mock_session_state['google_credentials'] = {'token': 'test'}
self.mock_session_state['google_user_info'] = {'email': 'test@example.com'}
self.assertTrue(is_authenticated())
if __name__ == '__main__':
unittest.main()
- Create Test for Search Console API
Create a new file at /workspace/AI-Writer/tests/integrations/google/test_search_console.py:
"""Unit tests for Google Search Console API module."""
import unittest
from unittest.mock import patch, MagicMock
import os
import sys
from pathlib import Path
from datetime import datetime, timedelta
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
from lib.integrations.google.search_console import (
get_search_console_service,
get_site_list,
get_search_analytics,
get_top_keywords,
get_top_pages,
get_keyword_insights
)
class TestSearchConsole(unittest.TestCase):
"""Test cases for Google Search Console API module."""
def setUp(self):
"""Set up test environment."""
# Mock session state
self.session_state_patch = patch('streamlit.session_state', {
'google_credentials': {
'token': 'test_token',
'refresh_token': 'test_refresh_token',
'token_uri': 'test_token_uri',
'client_id': 'test_client_id',
'client_secret': 'test_client_secret',
'scopes': ['test_scope']
}
})
self.mock_session_state = self.session_state_patch.start()
def tearDown(self):
"""Clean up after tests."""
self.session_state_patch.stop()
@patch('lib.integrations.google.search_console.build')
@patch('lib.integrations.google.search_console.Credentials')
def test_get_search_console_service(self, mock_credentials, mock_build):
"""Test building Search Console service."""
# Mock credentials
mock_credentials_instance = MagicMock()
mock_credentials.from_authorized_user_info.return_value = mock_credentials_instance
# Mock service
mock_service = MagicMock()
mock_build.return_value = mock_service
# Call function
service = get_search_console_service(self.mock_session_state['google_credentials'])
# Assertions
self.assertEqual(service, mock_service)
mock_credentials.from_authorized_user_info.assert_called_once()
mock_build.assert_called_once_with('searchconsole', 'v1', credentials=mock_credentials_instance)
@patch('lib.integrations.google.search_console.get_search_console_service')
def test_get_site_list(self, mock_get_service):
"""Test getting site list."""
# Mock service
mock_service = MagicMock()
mock_sites = MagicMock()
mock_sites.list().execute.return_value = {
'siteEntry': [
{
'siteUrl': 'https://example.com/',
'permissionLevel': 'siteOwner'
},
{
'siteUrl': 'sc-domain:example.org',
'permissionLevel': 'siteFullUser'
},
{
'siteUrl': 'https://example.net/',
'permissionLevel': 'siteRestrictedUser' # Should be filtered out
}
]
}
mock_service.sites.return_value = mock_sites
mock_get_service.return_value = mock_service
# Call function
sites = get_site_list()
# Assertions
self.assertEqual(len(sites), 2) # Only 2 sites with sufficient permissions
self.assertEqual(sites[0]['site_url'], 'https://example.com/')
self.assertEqual(sites[0]['permission_level'], 'siteOwner')
self.assertEqual(sites[0]['site_type'], 'Web')
self.assertEqual(sites[1]['site_url'], 'sc-domain:example.org')
self.assertEqual(sites[1]['site_type'], 'Domain Property')
@patch('lib.integrations.google.search_console.get_search_console_service')
def test_get_search_analytics(self, mock_get_service):
"""Test getting search analytics data."""
# Mock service
mock_service = MagicMock()
mock_analytics = MagicMock()
mock_analytics.query().execute.return_value = {
'rows': [
{
'keys': ['test keyword'],
'clicks': 100,
'impressions': 1000,
'ctr': 0.1,
'position': 5.5
}
]
}
mock_service.searchanalytics.return_value = mock_analytics
mock_get_service.return_value = mock_service
# Call function
start_date = datetime.now() - timedelta(days=30)
end_date = datetime.now()
result = get_search_analytics(
site_url='https://example.com/',
start_date=start_date,
end_date=end_date,
dimensions=['query'],
row_limit=100
)
# Assertions
self.assertEqual(len(result['rows']), 1)
mock_analytics.query.assert_called_once()
call_args = mock_analytics.query.call_args[1]
self.assertEqual(call_args['siteUrl'], 'https://example.com/')
self.assertEqual(call_args['body']['dimensions'], ['query'])
self.assertEqual(call_args['body']['rowLimit'], 100)
@patch('lib.integrations.google.search_console.get_search_analytics')
def test_get_top_keywords(self, mock_get_analytics):
"""Test getting top keywords."""
# Mock analytics response
mock_get_analytics.return_value = {
'rows': [
{
'keys': ['keyword1'],
'clicks': 100,
'impressions': 1000,
'ctr': 0.1,
'position': 5.5
},
{
'keys': ['keyword2'],
'clicks': 200,
'impressions': 2000,
'ctr': 0.1,
'position': 3.2
}
]
}
# Call function
keywords = get_top_keywords(
site_url='https://example.com/',
days=30,
limit=100
)
# Assertions
self.assertEqual(len(keywords), 2)
self.assertEqual(keywords[0]['keyword'], 'keyword1')
self.assertEqual(keywords[0]['clicks'], 100)
self.assertEqual(keywords[0]['impressions'], 1000)
self.assertEqual(keywords[0]['ctr'], 10.0) # Converted to percentage
self.assertEqual(keywords[0]['position'], 5.5)
mock_get_analytics.assert_called_once()
if __name__ == '__main__':
unittest.main()
Step 2: Integration Testing
- Create Integration Test Script
Create a new file at /workspace/AI-Writer/tests/integrations/google/test_integration.py:
"""Integration tests for Google integration."""
import unittest
import os
import sys
from pathlib import Path
import streamlit as st
from unittest.mock import patch, MagicMock
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
from lib.integrations.google.auth import get_google_auth_url, handle_auth_callback
from lib.integrations.google.search_console import get_site_list, get_top_keywords
from lib.integrations.google.keyword_research import get_content_opportunities
from lib.integrations.google.content_generator import generate_seo_optimized_content
class TestGoogleIntegration(unittest.TestCase):
"""Integration tests for Google integration."""
@classmethod
def setUpClass(cls):
"""Set up test environment."""
# Check if we have real credentials for testing
cls.has_real_credentials = os.environ.get('GOOGLE_TEST_CREDENTIALS') is not None
if cls.has_real_credentials:
# Load real credentials from environment
import json
credentials_json = os.environ.get('GOOGLE_TEST_CREDENTIALS')
cls.test_credentials = json.loads(credentials_json)
# Set up session state with real credentials
st.session_state['google_credentials'] = cls.test_credentials
st.session_state['google_user_info'] = {
'email': os.environ.get('GOOGLE_TEST_EMAIL', 'test@example.com'),
'name': 'Test User'
}
def test_auth_flow(self):
"""Test authentication flow."""
# Skip if no real credentials
if not self.has_real_credentials:
self.skipTest("No real credentials available for testing")
# Get auth URL
auth_url = get_google_auth_url()
# Verify URL format
self.assertTrue(auth_url.startswith('https://accounts.google.com/o/oauth2/auth'))
self.assertIn('client_id=', auth_url)
self.assertIn('redirect_uri=', auth_url)
self.assertIn('scope=', auth_url)
def test_search_console_access(self):
"""Test Search Console API access."""
# Skip if no real credentials
if not self.has_real_credentials:
self.skipTest("No real credentials available for testing")
# Get site list
sites = get_site_list()
# Verify we got some sites
self.assertIsInstance(sites, list)
# If we have sites, test keyword data
if sites:
site_url = sites[0]['site_url']
# Get top keywords
keywords = get_top_keywords(site_url, days=30, limit=10)
# Verify we got keyword data
self.assertIsInstance(keywords, list)
# If we have keywords, test content opportunities
if keywords:
# Get content opportunities
opportunities = get_content_opportunities(site_url, min_impressions=10, max_position=50.0)
# Verify we got opportunities
self.assertIsInstance(opportunities, list)
@patch('lib.integrations.google.content_generator.generate_blog_post')
def test_content_generation(self, mock_generate_blog_post):
"""Test content generation with GSC data."""
# Mock blog post generation
mock_generate_blog_post.return_value = {
'title': 'Test Blog Post',
'content': 'This is a test blog post.'
}
# Generate content
content = generate_seo_optimized_content(
keyword='test keyword',
site_url='https://example.com/',
content_type='blog',
length='medium'
)
# Verify content was generated
self.assertEqual(content['title'], 'Test Blog Post')
self.assertEqual(content['content'], 'This is a test blog post.')
# Verify prompt enhancements were passed
mock_generate_blog_post.assert_called_once()
call_args = mock_generate_blog_post.call_args[1]
self.assertEqual(call_args['keyword'], 'test keyword')
self.assertEqual(call_args['content_type'], 'blog')
self.assertEqual(call_args['length'], 'medium')
self.assertIn('prompt_enhancements', call_args)
if __name__ == '__main__':
unittest.main()
Step 3: End-to-End Testing
- Create Manual Test Plan
Create a new file at /workspace/AI-Writer/tests/integrations/google/manual_test_plan.md:
# Google Integration Manual Test Plan
This document outlines the manual testing procedures for the Google integration in AI-Writer.
## Prerequisites
- Google account with access to Search Console
- Website verified in Google Search Console
- AI-Writer development environment set up
## Test Cases
### 1. Authentication Flow
#### 1.1 Google Login
1. Navigate to the Google Integration page
2. Click "Sign in with Google"
3. Complete Google authentication flow
4. Verify user is redirected back to AI-Writer
5. Verify user info is displayed correctly
**Expected Result:** User is successfully authenticated and user info is displayed.
#### 1.2 Logout
1. Click "Sign Out" button
2. Verify user is logged out
3. Verify login button is displayed again
**Expected Result:** User is successfully logged out.
### 2. Search Console Access
#### 2.1 Site Selection
1. Authenticate with Google
2. Verify site selector displays user's verified sites
3. Select a site from the dropdown
4. Verify Search Console dashboard loads
**Expected Result:** User's verified sites are displayed and can be selected.
#### 2.2 Top Keywords
1. Select a site
2. Navigate to "Top Keywords" tab
3. Adjust time period slider
4. Verify keyword data loads and updates
**Expected Result:** Keyword data is displayed and updates when time period changes.
#### 2.3 Top Pages
1. Select a site
2. Navigate to "Top Pages" tab
3. Adjust time period slider
4. Verify page data loads and updates
**Expected Result:** Page data is displayed and updates when time period changes.
#### 2.4 Keyword Research
1. Select a site
2. Navigate to "Keyword Research" tab
3. Enter a keyword
4. Verify keyword insights are displayed
5. Verify performance charts are displayed
6. Verify ranking pages are displayed
**Expected Result:** Keyword insights and related data are displayed correctly.
### 3. Content Planning
#### 3.1 Content Opportunities
1. Navigate to Content Planning
2. Select "Content Opportunities" tab
3. Adjust filters
4. Click "Find Opportunities"
5. Verify opportunities are displayed
6. Select a keyword and click "Create New Content"
7. Verify redirection to content creation
**Expected Result:** Content opportunities are found and can be used for content creation.
#### 3.2 Topic Clusters
1. Navigate to Content Planning
2. Select "Topic Clusters" tab
3. Adjust cluster settings
4. Click "Generate Topic Clusters"
5. Verify clusters are displayed
6. Expand a cluster to view keywords
7. Click "Create Cluster Content"
8. Verify redirection to content creation
**Expected Result:** Topic clusters are generated and can be used for content creation.
#### 3.3 Content Calendar
1. Navigate to Content Planning
2. Select "Content Calendar" tab
3. Adjust calendar settings
4. Click "Generate Content Calendar"
5. Verify calendar is displayed
6. Verify export functionality works
**Expected Result:** Content calendar is generated and can be exported.
### 4. Content Creation
#### 4.1 SEO-Optimized Content
1. Select a keyword from opportunities
2. Click "Create New Content"
3. Verify GSC data is incorporated in the prompt
4. Generate content
5. Verify content includes related keywords
6. Verify content addresses search intent
**Expected Result:** Generated content is optimized based on GSC data.
#### 4.2 Content Optimization
1. Select a keyword
2. Click "Optimize Existing Content"
3. Enter or paste existing content
4. Click "Analyze Content"
5. Verify optimization suggestions are displayed
6. Apply suggestions
7. Verify content improvements
**Expected Result:** Content optimization suggestions are provided and can be applied.
## Test Execution Checklist
| Test Case | Status | Notes |
|-----------|--------|-------|
| 1.1 Google Login | | |
| 1.2 Logout | | |
| 2.1 Site Selection | | |
| 2.2 Top Keywords | | |
| 2.3 Top Pages | | |
| 2.4 Keyword Research | | |
| 3.1 Content Opportunities | | |
| 3.2 Topic Clusters | | |
| 3.3 Content Calendar | | |
| 4.1 SEO-Optimized Content | | |
| 4.2 Content Optimization | | |
Step 4: Automated Testing Setup
- Create GitHub Actions Workflow
Create a new file at /.github/workflows/google_integration_tests.yml:
name: Google Integration Tests
on:
push:
branches: [ main ]
paths:
- 'lib/integrations/google/**'
- 'tests/integrations/google/**'
pull_request:
branches: [ main ]
paths:
- 'lib/integrations/google/**'
- 'tests/integrations/google/**'
workflow_dispatch:
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install pytest pytest-cov
- name: Run unit tests
run: |
pytest tests/integrations/google/test_auth.py tests/integrations/google/test_search_console.py -v
- name: Generate coverage report
run: |
pytest --cov=lib.integrations.google tests/integrations/google/ --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: google-integration
fail_ci_if_error: false
integration-tests:
runs-on: ubuntu-latest
if: github.event_name == 'workflow_dispatch'
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip install pytest
- name: Run integration tests
env:
GOOGLE_TEST_CREDENTIALS: ${{ secrets.GOOGLE_TEST_CREDENTIALS }}
GOOGLE_TEST_EMAIL: ${{ secrets.GOOGLE_TEST_EMAIL }}
run: |
pytest tests/integrations/google/test_integration.py -v
Deployment Plan
Phase 1: Development Environment
-
Initial Setup
- Set up Google Cloud Project
- Configure OAuth consent screen
- Create OAuth credentials
- Enable required APIs
- Implement authentication module
-
Local Testing
- Test authentication flow
- Test Search Console API access
- Test content generation with GSC data
- Run unit tests
Phase 2: Staging Environment
-
Configuration
- Create staging Google Cloud Project
- Configure OAuth for staging domain
- Update redirect URIs
- Set up staging environment variables
-
Integration Testing
- Deploy to staging environment
- Test end-to-end flow
- Verify data accuracy
- Test with multiple users and sites
-
Performance Testing
- Test with large datasets
- Optimize API calls
- Implement caching if needed
- Monitor API usage limits
Phase 3: Production Deployment
-
Final Configuration
- Create production Google Cloud Project
- Configure OAuth for production domain
- Set up production environment variables
- Verify API quotas and limits
-
Deployment
- Deploy to production
- Monitor for errors
- Verify authentication flow
- Test with real user accounts
-
Post-Deployment
- Monitor API usage
- Collect user feedback
- Address any issues
- Document the integration
User Documentation
User Guide
Create a user guide that explains:
- How to connect Google account
- How to access Search Console data
- How to use content opportunities
- How to create SEO-optimized content
- How to use the content calendar
Video Tutorial
Create a video tutorial demonstrating:
- Initial setup process
- Finding content opportunities
- Creating content with GSC data
- Optimizing existing content
- Planning content with the calendar
Conclusion
This implementation plan provides a comprehensive approach to integrating Google login and Google Search Console with AI-Writer. By following these steps, you'll enhance the platform with real user insights for content creation, improving the relevance and effectiveness of generated content.
The integration will provide significant value to users by:
- Leveraging real search data for content creation
- Identifying content opportunities based on actual performance
- Optimizing content for keywords that matter to the user's audience
- Creating a data-driven content strategy
- Measuring content performance over time
This feature will differentiate AI-Writer from competitors by providing a closed-loop system for content creation, optimization, and performance tracking.