diff --git a/Getting Started/README.md b/Getting Started/README.md
new file mode 100644
index 00000000..6cb73585
--- /dev/null
+++ b/Getting Started/README.md
@@ -0,0 +1,167 @@
+## Easy Installation Guide for Content Creators
+
+### Step 1: Install Python 3.11
+1. Download Python 3.11 installer:
+ - Visit [Python 3.11.6 Download Page](https://www.python.org/downloads/release/python-3116/)
+ - Scroll down and click on "Windows installer (64-bit)"
+ - Save the file to your computer
+
+2. Run the installer:
+ - Double click the downloaded file
+ - β IMPORTANT: Check "Add Python 3.11 to PATH"
+ - Click "Install Now"
+ - Wait for installation to complete
+ - Click "Close"
+
+### Step 2: Install ALwrity
+1. Download this project:
+ - Click the green "Code" button above
+ - Select "Download ZIP"
+ - Extract the ZIP file to your desired location
+
+2. Open Command Prompt:
+ - Press Windows + R
+ - Type "cmd" and press Enter
+ - Navigate to the extracted folder:
+ ```
+ cd path\to\ALwrity
+ ```
+
+3. Run the automatic installer:
+ ```
+ python setup.py install
+ ```
+
+### Troubleshooting
+If you encounter any issues:
+1. Make sure Python 3.11 is installed correctly:
+ - Open Command Prompt
+ - Type: `python --version`
+ - Should show: `Python 3.11.x`
+
+2. Common Issues:
+ - If you see "Python is not recognized": Restart your computer
+ - If you get package errors: Run `pip install --upgrade pip` first
+
+Need help? [Open an issue](../../issues) and we'll assist you!
+
+## For Developers
+If you're a developer or want to contribute:
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/ALwrity.git
+
+# Create virtual environment
+python -m venv venv
+
+# Activate virtual environment
+# On Windows:
+.\venv\Scripts\activate
+# On Mac/Linux:
+source venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+# ALwrity - AI Content Writing Assistant
+
+## Quick Start Guide for Non-Technical Users
+
+### Option 1: One-Click Installation (Recommended)
+1. Download this project:
+ - Click the green "Code" button above
+ - Select "Download ZIP"
+ - Extract the ZIP file to your desired location (e.g., Desktop)
+
+2. Run the installer:
+ - Double-click `install.bat` in the extracted folder
+ - If Windows asks for permission, click "Yes"
+ - Follow the on-screen instructions
+ - Wait for the installation to complete
+
+3. Start ALwrity:
+ - Open Command Prompt (Windows + R, type "cmd", press Enter)
+ - Navigate to the ALwrity folder:
+ ```
+ cd path\to\ALwrity
+ ```
+ - Type `alwrity` and press Enter
+
+### Option 2: Manual Installation
+If the one-click installer doesn't work, follow these steps:
+
+1. Install Python 3.11:
+ - Visit [Python 3.11.6 Download Page](https://www.python.org/downloads/release/python-3116/)
+ - Click "Windows installer (64-bit)"
+ - Run the installer
+ - β IMPORTANT: Check "Add Python 3.11 to PATH"
+ - Click "Install Now"
+ - Wait for installation to complete
+
+2. Install ALwrity:
+ - Open Command Prompt (Windows + R, type "cmd", press Enter)
+ - Navigate to the ALwrity folder:
+ ```
+ cd path\to\ALwrity
+ ```
+ - Run the installation:
+ ```
+ python setup.py install
+ ```
+ - Follow any on-screen instructions
+
+3. Start ALwrity:
+ - In the same Command Prompt window, type:
+ ```
+ alwrity
+ ```
+ - Press Enter
+
+### Troubleshooting Guide
+
+#### Common Issues and Solutions:
+
+1. "Python is not recognized"
+ - Solution: Restart your computer after installing Python
+ - Make sure you checked "Add Python 3.11 to PATH" during installation
+
+2. "Visual C++ Build Tools not found"
+ - Solution: Run this command in an administrative PowerShell:
+ ```
+ winget install Microsoft.VisualStudio.2022.BuildTools --silent --override "--wait --quiet --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"
+ ```
+
+3. "Rust compiler not found"
+ - Solution: Run these commands in PowerShell:
+ ```
+ Invoke-WebRequest -Uri https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe -OutFile rustup-init.exe
+ ./rustup-init.exe -y
+ ```
+
+4. Installation Errors
+ - Check the `install_errors.log` file in the ALwrity folder
+ - Share the error message with our support team
+
+#### Need Help?
+- Open an issue on GitHub
+- Join our support community
+- Contact our support team
+
+### System Requirements
+- Windows 10 or later
+- Python 3.11.x
+- At least 4GB RAM
+- 2GB free disk space
+
+### First-Time Setup
+After installation:
+1. The first time you run ALwrity, it will ask for your API keys
+2. Follow the on-screen instructions to enter your keys
+3. Your keys will be saved securely for future use
+
+### Updating ALwrity
+To update to the latest version:
+1. Download the latest release
+2. Run `install.bat` again
+3. Follow the on-screen instructions
\ No newline at end of file
diff --git a/install.bat b/Getting Started/install.bat
similarity index 100%
rename from install.bat
rename to Getting Started/install.bat
diff --git a/setup.py b/Getting Started/setup.py
similarity index 100%
rename from setup.py
rename to Getting Started/setup.py
diff --git a/README.md b/README.md
index 2fdb144a..7f059b6b 100644
--- a/README.md
+++ b/README.md
@@ -1,31 +1,105 @@
# Alwrity: Redefining Content Lifecycle with AI
### π **ALwrity: Your All-in-One Content Platform** π
+
+
+[](https://opensource.org/licenses/MIT)
+[](https://www.python.org/downloads/)
+[](https://streamlit.io/)
+
> **NOTE**
-> *Alwrity is a comprehensive content lifecycle platform tailored for content creators, digital marketers, and writers β no prior AI knowledge required.*
+> *Alwrity is a comprehensive content lifecycle platform tailored for content creators, digital marketers, and writers β no prior AI knowledge required.*
-Alwrity streamlines every phase of the content lifecycle, from **planning and research** to **personalized content generation**, **SEO audits**, **publishing**, and **analytics**. Our mission is to empower creators with AI-driven tools that simplify content creation while maintaining quality.
+## π Table of Contents
-Explore the building blocks of Alwrity and contribute to shaping the future of AI-driven content creation.
-> **New Release**
-> [*ALwrity AI story writer App*](https://storyme.app.io/)
-Learn more about our journey: [What's Alwrity Up To?](https://www.alwrity.com/post/whats-alwrity-upto)
+- [Overview](#overview)
+- [Key Features](#key-features)
+- [System Architecture](#system-architecture)
+- [Getting Started](#getting-started)
+ - [Prerequisites](#prerequisites)
+ - [Installation](#installation)
+ - [Configuration](#configuration)
+- [Usage Guide](#usage-guide)
+ - [AI Writers](#ai-writers)
+ - [SEO Tools](#seo-tools)
+ - [Social Media Tools](#social-media-tools)
+ - [Content Planning](#content-planning)
+- [API Documentation](#api-documentation)
+- [Contributing](#contributing)
+- [Roadmap](#roadmap)
+- [License](#license)
+- [Acknowledgements](#acknowledgements)
+---
+
+## π Overview
+
+Alwrity streamlines every phase of the content lifecycle, from **planning and research** to **personalized content generation**, **SEO audits**, **publishing**, and **analytics**. Our mission is to empower creators with AI-driven tools that simplify content creation while maintaining quality.
+
+The platform integrates state-of-the-art AI technologies to provide a seamless content creation experience:
+
+| **Content Category** | **Technologies/Models** |
+|--------------------------|-------------------------------------------|
+| Text Generation Models | OpenAI, Gemini, Anthropic |
+| Image Creation Tools | Stability.ai |
+| Speech-to-Text Systems | Whisper, AssemblyAI |
+| AI-Powered Web Research | Tavily AI, exa AI, Serper.dev |
---
+
+## π Key Features
+
+### AI Writer Tools
+
+- **AI Blog Writer**: Generate blog content based on web research
+- **AI YouTube to Content Writer**: Transform YouTube videos into written content
+- **AI Long Form Content**: Create detailed articles with proper structure
+- **AI Story Writer**: Craft engaging narratives and stories
+- **AI Email Writer**: Generate professional and business emails
+- **AI LinkedIn Post Generator**: Create optimized LinkedIn content
+- **AI Product Description Generator**: Write compelling product descriptions
+
+### SEO Tools
+
+- **Rich Snippet Generator**: Create structured data for better SERP visibility
+- **On-Page SEO Analyzer**: Evaluate and optimize web pages
+- **URL SEO Checker**: Assess URL structure and performance
+- **Backlinking Tool**: Discover high-quality backlink opportunities
+- **Image Alt Text Generator**: Create accessible image descriptions
+- **Meta Description Generator**: Generate SEO-friendly meta descriptions
+
+### Social Media Tools
+
+- **X Tweet Generator**: Create engaging tweets
+- **Instagram Caption Generator**: Write compelling Instagram captions
+- **Facebook Post Generator**: Develop Facebook-optimized content
+- **YouTube Content Tools**: Generate titles, descriptions, and scripts
+
+### Content Planning
+
+- **Content Calendar**: Plan content for weeks or months ahead
+- **Blog Image Creation**: Generate images to complement your content
+- **Agentic Content Creation**: Use AI agents for specialized content tasks
+- **Web Research Integration**: Gather factual information for your content
+
+## ποΈ System Architecture
+
+Alwrity is built with a modular architecture that enables flexibility and extensibility:
+
+
+
+The platform consists of several key components:
+
+1. **User Interface Layer**: Streamlit-based web interface
+2. **Core Services Layer**: AI Writers, Web Research, SEO Tools, Analytics
+3. **Data Storage Layer**: Vector Database (ChromaDB), Relational Database (SQLite)
+4. **External Integrations Layer**: LLM Providers, Search Providers, Image Generation, Publishing Platforms
+
+For more detailed architecture information, see the [Architecture Documentation](docs/architecture/index.rst).
+
## Getting Started with ALwrity: "AI at Every Stage of Content Lifecycle."
Alwrity empowers content creators, solopreneurs and digital marketers with cutting-edge tools for keyword research, AI-driven writing, and social media content generation. From creating high-quality copywriting frameworks to crafting engaging YouTube scripts, our platform simplifies every step of your content creation journey.
-We seamlessly integrate state-of-the-art AI technologies, including:
-| **Content Category** | **Technologies/Models** |
-|--------------------------|-------------------------------------------|
-| Text Generation Models | OpenAI, Gemini, Anthropic |
-| Image Creation Tools | Stability.ai |
-| Speech-to-Text Systems | Whisper, AssemblyAI |
-| AI-Powered Web Research | Tavily AI, exa AI, Serper.dev |
-
-By bringing these powerful solutions together, Alwrity ensures a streamlined workflow and exceptional output with minimal effort. Get ready to supercharge your content creation process!
-
---
> 
---
@@ -46,48 +120,6 @@ By bringing these powerful solutions together, Alwrity ensures a streamlined wor
>
> See Details
>
-## Prerequisites
-
-### Windows
-- Python 3.10+ (3.12 recommended)
-- Microsoft Visual C++ Build Tools 14.0 or greater
- - Install with: `winget install Microsoft.VisualStudio.2022.BuildTools --silent --override "--wait --quiet --add Microsoft.VisualStudio.Workload.VCTools --includeRecommended"`
-- Rust Compiler
- - Install with: `Invoke-WebRequest -Uri https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe -OutFile rustup-init.exe; ./rustup-init.exe -y`
-
-### Linux
-- Python 3.10+ (3.12 recommended)
-- C/C++ compiler and development tools
- - Install with: `sudo apt update && sudo apt install build-essential python3-dev`
-- Rust Compiler
- - Install with: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y; source $HOME/.cargo/env`
-
-> ```
-> 1). git clone https://github.com/AJaySi/AI-Writer.git
-> 2). pip install -r requirements.txt
-> 3). streamlit run alwrity.py
->
-> 4). Visit Alwrity UI in a Browser & Start generation AI personalized content.
-> ```
->
-> ### System Dependencies
-> ```
-> You can run the install_dependencies.py script to check for system dependencies and assist with installation:
->
-> python install_dependencies.py
-> ```
-> This script will help verify that all necessary system dependencies are installed and guide you through the installation process.
-> ### Updating to latest Code: (Existing users)
-> ```
-> 1). Git pull
-> 2). pip install -U -r requirements.txt
-> 3). streamlit run alwrity.py
-> ```
->
->
----
-Still stuck, [Open issue here](https://github.com/AJaySi/AI-Writer/issues) & Someone will bail you out.
-> 
### 
---
@@ -96,8 +128,74 @@ Still stuck, [Open issue here](https://github.com/AJaySi/AI-Writer/issues) & Som
- 
- 
- 
-- RoadMap - Coming Soon..
-- Present Focus: Nextjs react Alwrity App - Coming Soon..
+
+---
+
+## π Usage Guide
+
+### AI Writers
+
+1. **Blog Writer**:
+ - Enter your target keywords
+ - Select blog type and length
+ - Choose whether to include web research
+ - Generate and edit your blog content
+
+ 
+
+2. **Long Form Content**:
+ - Provide a detailed topic
+ - Select content structure
+ - Generate comprehensive content with proper sections
+ - Edit and refine as needed
+
+3. **Email Writer**:
+ - Select email type (professional, business, etc.)
+ - Enter recipient and purpose
+ - Generate appropriate email content
+ - Customize tone and style
+
+### SEO Tools
+
+1. **Rich Snippet Generator**:
+ - Enter your URL or content
+ - Select snippet type (FAQ, Product, etc.)
+ - Generate structured data
+ - Copy and implement on your website
+
+2. **On-Page SEO Analyzer**:
+ - Enter your URL
+ - Receive comprehensive SEO analysis
+ - Get actionable recommendations
+ - Implement suggested changes
+
+### Social Media Tools
+
+1. **X Tweet Generator**:
+ - Enter topic or keywords
+ - Select tweet style
+ - Generate engaging tweets
+ - Schedule or post directly
+
+2. **YouTube Script Generator**:
+ - Enter video topic
+ - Select video length and style
+ - Generate complete script with sections
+ - Export for recording
+
+### Content Planning
+
+1. **Content Calendar**:
+ - Enter your niche or industry
+ - Select timeframe (weeks/months)
+ - Generate content ideas with titles
+ - Export to your preferred format
+
+2. **Web Research**:
+ - Enter research topic
+ - Select research depth
+ - Receive comprehensive research results
+ - Use in your content generation
---
@@ -238,216 +336,68 @@ Still stuck, [Open issue here](https://github.com/AJaySi/AI-Writer/issues) & Som
---
-
Notes from underground:
- 1). Focus is on writing/generating highly unique, SEO optimized blog content.
- 2). Models: Openai, gemini, ollama are interesting. Minstral API is also worth exploring. Cohere API is purpose made.
- Focus is getting the prompts right. Shit in, shit out, irrespective of dollars and cutting edge models.
- Pydantically speakng, Due to experimental nature of prompting, its getting expensive soon enough. Gemini is free for now.
- 3). Missing frontend: A smart backend will enable a good frontend. WIP, backend. So, frontend; coming soon.
- 4).Getting AI agents to 'brainstrom' blog ideas seems more pressing. CrewAI seems more straightforward than autogen.
- 5). Too Many APIs floating around: The implementation is using tools that dont depend on API keys and rather scrape them.
- Duh, scraping wont scale, that is GPT vision based scraping will come in handy.
- 6). firecrawl is interesting, gpt-researcher is providing local docsqa.
- 7). Had to provide streamlit UI as Alwrity's audience arent comfortable with commandline.
- 8). Local folder RAG and Chat with your content, website is on the cards.
- 9). AI models are better, not sure until when 'Free' APIs will be "Free".
- 10). The code is always a mess, lot of changes happening..
- 11). Focus is to stop making any more AI content tools, but rather revisit & improve user experience & content quality.
- 12). To Err is Human & AI....
- 13). ....
-
+## π API Documentation
+
+Alwrity provides a comprehensive API for programmatic access to its features. The API documentation is available at:
+
+- [API Reference](docs/api/index.rst)
+- [API Examples](docs/api/examples.rst)
+
+## π€ Contributing
+
+We welcome contributions to Alwrity! Please see our [Contributing Guide](CONTRIBUTING.md) for details on how to get started.
+
+## πΊοΈ Roadmap
+
+- [Read Detailed Roadmap Here](Roadmap TBDs/ROADMAP.md)
+- [ALwrity Roadmap](docs/roadmap.rst)
+
+Our development roadmap includes:
+
+- **Short-term (0-3 months)**:
+ - Enhanced multi-language support
+ - Improved image generation capabilities
+ - Additional AI model integrations
+
+- **Medium-term (3-6 months)**:
+ - Advanced analytics dashboard
+ - Content performance tracking
+ - Collaborative editing features
+
+- **Long-term (6+ months)**:
+ - NextJS React Alwrity App
+ - API-first architecture
+ - Enterprise features for teams
+
+## π License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## π Acknowledgements
+
+Alwrity stands on the shoulders of giants:
+
+- **APIs**:
+ - [Exa API](https://exa.ai/): Semantic search capabilities
+ - [Tavily API](https://tavily.com/): AI-powered web search
+ - [SerperDev API](https://serper.dev/): Search engine results
+ - [YOU.com](https://you.com/): Enhanced web search
+ - [Stability AI](https://stability.ai/): Image generation
+ - [OpenAI API](https://openai.com/): LLM capabilities
+ - [Gemini API](https://gemini.google.com/app): Google's LLM
+ - [Ollama](https://ollama.com/): Local LLM provider
+ - [CrewAI](https://www.crewai.com/): Collaborative AI agents
+
+## π Support
+
+If you encounter any issues or have questions, please [open an issue](https://github.com/AJaySi/AI-Writer/issues) on GitHub.
---
-#### LICENSE
-> [!NOTE]
-> MIT License
->
-> Copyright (c) 2024 Alwrity
->
-> Permission is hereby granted, free of charge, to any person obtaining a copy
-> of this software and associated documentation files (the "Software"), to deal
-> in the Software without restriction, including without limitation the rights
-> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-> copies of the Software, and to permit persons to whom the Software is
-> furnished to do so, subject to the following conditions:
->
-> The above copyright notice and this permission notice shall be included in all
-> copies or substantial portions of the Software.
->
-> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-> SOFTWARE.
->
+ Sign in with Google
+
+ """.format(auth_url=auth_url), unsafe_allow_html=True)
+ else:
+ user_info = st.session_state.google_user_info
+
+ # Display user info
+ col1, col2 = st.columns([1, 3])
+
+ with col1:
+ st.image(user_info.get('picture', ''), width=50)
+
+ with col2:
+ st.markdown(f"**{user_info.get('name', 'User')}**")
+ st.markdown(f"{user_info.get('email', '')}")
+
+ # Logout button
+ if st.button("Sign Out"):
+ logout()
+ st.experimental_rerun()
+
+def render_site_selector():
+ """Render Google Search Console site selector."""
+ if not is_authenticated():
+ st.warning("Please sign in with Google to access Search Console data.")
+ return None
+
+ # Get site list
+ sites = get_site_list()
+
+ if not sites:
+ st.warning("No verified sites found in your Search Console account.")
+ return None
+
+ # Create site options
+ site_options = [site['site_url'] for site in sites]
+
+ # Select site
+ selected_site = st.selectbox(
+ "Select a website",
+ options=site_options,
+ index=0 if site_options else None,
+ format_func=lambda x: x.replace("sc-domain:", "")
+ )
+
+ return selected_site
+
+def render_search_console_dashboard(site_url):
+ """Render Google Search Console dashboard.
+
+ Args:
+ site_url: URL of the selected site
+ """
+ if not site_url:
+ return
+
+ st.markdown("## Search Console Insights")
+
+ # Create tabs
+ tab1, tab2, tab3 = st.tabs(["Top Keywords", "Top Pages", "Keyword Research"])
+
+ with tab1:
+ st.markdown("### Top Keywords")
+
+ # Date range selector
+ days = st.slider("Time period (days)", min_value=7, max_value=90, value=30, step=1)
+
+ # Get top keywords
+ with st.spinner("Loading keyword data..."):
+ keywords = get_top_keywords(site_url, days=days, limit=100)
+
+ if not keywords:
+ st.info("No keyword data available for this site.")
+ return
+
+ # Display keywords table
+ st.dataframe(
+ keywords,
+ column_config={
+ "keyword": "Keyword",
+ "clicks": st.column_config.NumberColumn("Clicks", format="%d"),
+ "impressions": st.column_config.NumberColumn("Impressions", format="%d"),
+ "ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
+ "position": st.column_config.NumberColumn("Position", format="%.1f")
+ },
+ hide_index=True
+ )
+
+ with tab2:
+ st.markdown("### Top Pages")
+
+ # Date range selector
+ days = st.slider("Time period (days)", min_value=7, max_value=90, value=30, step=1, key="pages_days")
+
+ # Get top pages
+ with st.spinner("Loading page data..."):
+ pages = get_top_pages(site_url, days=days, limit=100)
+
+ if not pages:
+ st.info("No page data available for this site.")
+ return
+
+ # Display pages table
+ st.dataframe(
+ pages,
+ column_config={
+ "page": "Page URL",
+ "clicks": st.column_config.NumberColumn("Clicks", format="%d"),
+ "impressions": st.column_config.NumberColumn("Impressions", format="%d"),
+ "ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
+ "position": st.column_config.NumberColumn("Position", format="%.1f")
+ },
+ hide_index=True
+ )
+
+ with tab3:
+ st.markdown("### Keyword Research")
+
+ # Keyword input
+ keyword = st.text_input("Enter a keyword to analyze")
+
+ if keyword:
+ # Get keyword insights
+ with st.spinner(f"Analyzing '{keyword}'..."):
+ insights = get_keyword_insights(keyword, site_url, days=90)
+
+ # Display insights
+ col1, col2, col3, col4 = st.columns(4)
+
+ with col1:
+ st.metric("Clicks", f"{insights['total_clicks']:,}")
+
+ with col2:
+ st.metric("Impressions", f"{insights['total_impressions']:,}")
+
+ with col3:
+ st.metric("Avg. CTR", f"{insights['avg_ctr']:.2f}%")
+
+ with col4:
+ st.metric("Avg. Position", f"{insights['avg_position']:.1f}")
+
+ # Time series chart
+ if insights['time_series']:
+ st.markdown("#### Performance Over Time")
+
+ # Prepare data for chart
+ chart_data = {
+ 'date': [item['date'] for item in insights['time_series']],
+ 'clicks': [item['clicks'] for item in insights['time_series']],
+ 'impressions': [item['impressions'] for item in insights['time_series']],
+ 'position': [item['position'] for item in insights['time_series']]
+ }
+
+ # Create tabs for different metrics
+ chart_tab1, chart_tab2, chart_tab3 = st.tabs(["Clicks", "Impressions", "Position"])
+
+ with chart_tab1:
+ st.line_chart(chart_data, x='date', y='clicks')
+
+ with chart_tab2:
+ st.line_chart(chart_data, x='date', y='impressions')
+
+ with chart_tab3:
+ # Invert position axis (lower is better)
+ position_chart = {
+ 'date': chart_data['date'],
+ 'position': [-pos for pos in chart_data['position']] # Invert values
+ }
+ st.line_chart(position_chart, x='date', y='position')
+ st.caption("Note: Position axis is inverted (higher is better)")
+
+ # Ranking pages
+ if insights['pages']:
+ st.markdown("#### Pages Ranking for this Keyword")
+
+ st.dataframe(
+ insights['pages'],
+ column_config={
+ "page": "Page URL",
+ "clicks": st.column_config.NumberColumn("Clicks", format="%d"),
+ "impressions": st.column_config.NumberColumn("Impressions", format="%d"),
+ "ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
+ "position": st.column_config.NumberColumn("Position", format="%.1f")
+ },
+ hide_index=True
+ )
+
+ # Content suggestions
+ st.markdown("#### Content Suggestions")
+
+ if st.button("Generate Content Ideas"):
+ with st.spinner("Generating content ideas..."):
+ # This would connect to your AI content generation system
+ # For now, we'll just show a placeholder
+ st.info("This would connect to your AI content generation system to create content ideas based on this keyword's performance data.")
+```
+
+2. **Create Main Integration Page**
+
+Create a new file at `/workspace/AI-Writer/lib/integrations/google/page.py`:
+
+```python
+"""Google integration page for AI-Writer."""
+
+import streamlit as st
+from .components import render_google_login_button, render_site_selector, render_search_console_dashboard
+from .callback_handler import handle_oauth_callback
+
+def render_google_integration_page():
+ """Render the Google integration page."""
+ st.title("Google Integration")
+
+ # Handle OAuth callback if present
+ handle_oauth_callback()
+
+ # Display login/user info
+ st.markdown("### Google Account")
+ render_google_login_button()
+
+ # Separator
+ st.markdown("---")
+
+ # Search Console section
+ st.markdown("### Google Search Console")
+ st.markdown("""
+ Connect to Google Search Console to access search analytics data for your websites.
+ This data will help you optimize your content for better search visibility.
+ """)
+
+ # Site selector
+ selected_site = render_site_selector()
+
+ # Display dashboard if site is selected
+ if selected_site:
+ render_search_console_dashboard(selected_site)
+```
+
+#### Step 5: Update Main Application
+
+1. **Add Google Integration to Sidebar**
+
+Update the sidebar in the main application file (`alwrity.py`):
+
+```python
+# Add to imports
+from lib.integrations.google.page import render_google_integration_page
+
+# Add to sidebar menu
+with st.sidebar:
+ st.title("AI-Writer")
+
+ # Existing menu items...
+
+ # Add Google Integration option
+ if st.sidebar.selectbox("Integrations", ["None", "Google"]) == "Google":
+ page = "google_integration"
+
+ # Existing sidebar code...
+
+# Add to page router
+if page == "google_integration":
+ render_google_integration_page()
+```
+
+2. **Add Google Credentials to Streamlit Secrets**
+
+Create a file at `/.streamlit/secrets.toml`:
+
+```toml
+[google_oauth]
+web = {
+ "client_id": "YOUR_CLIENT_ID",
+ "project_id": "ai-writer",
+ "auth_uri": "https://accounts.google.com/o/oauth2/auth",
+ "token_uri": "https://oauth2.googleapis.com/token",
+ "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
+ "client_secret": "YOUR_CLIENT_SECRET",
+ "redirect_uris": ["http://localhost:8501/oauth/callback"],
+ "javascript_origins": ["http://localhost:8501"]
+}
+```
+
+### Phase 2: Content Enhancement with GSC Data
+
+#### Step 1: Create Keyword Research Module
+
+Create a new file at `/workspace/AI-Writer/lib/integrations/google/keyword_research.py`:
+
+```python
+"""Keyword research module using Google Search Console data."""
+
+from typing import Dict, List, Any, Optional
+import streamlit as st
+from loguru import logger
+from .search_console import get_top_keywords, get_keyword_insights
+
+def get_content_opportunities(site_url: str, min_impressions: int = 100, max_position: float = 20.0) -> List[Dict[str, Any]]:
+ """Find content opportunities based on GSC data.
+
+ Args:
+ site_url: URL of the site
+ min_impressions: Minimum impressions threshold
+ max_position: Maximum position threshold
+
+ Returns:
+ List[Dict[str, Any]]: List of content opportunities
+ """
+ try:
+ # Get top keywords
+ keywords = get_top_keywords(site_url, days=90, limit=1000)
+
+ # Filter for opportunities
+ opportunities = [
+ keyword for keyword in keywords
+ if keyword['impressions'] >= min_impressions
+ and keyword['position'] <= max_position
+ and keyword['position'] > 3 # Not already in top 3
+ ]
+
+ # Sort by potential (impressions * (11 - position)/10)
+ # This prioritizes keywords with high impressions and positions 4-10
+ for opp in opportunities:
+ opp['potential'] = opp['impressions'] * (11 - min(opp['position'], 10)) / 10
+
+ opportunities.sort(key=lambda x: x['potential'], reverse=True)
+
+ return opportunities[:100] # Return top 100 opportunities
+ except Exception as e:
+ logger.error(f"Error finding content opportunities: {str(e)}")
+ return []
+
+def get_keyword_clusters(keywords: List[Dict[str, Any]], threshold: int = 3) -> List[Dict[str, Any]]:
+ """Group keywords into clusters based on common words.
+
+ Args:
+ keywords: List of keywords with metrics
+ threshold: Minimum number of keywords to form a cluster
+
+ Returns:
+ List[Dict[str, Any]]: List of keyword clusters
+ """
+ try:
+ # Extract keyword strings
+ keyword_strings = [k['keyword'].lower() for k in keywords]
+
+ # Create word frequency map
+ word_map = {}
+ for kw in keyword_strings:
+ words = kw.split()
+ for word in words:
+ if len(word) > 3: # Ignore short words
+ if word not in word_map:
+ word_map[word] = []
+ word_map[word].append(kw)
+
+ # Find clusters
+ clusters = []
+ for word, kws in word_map.items():
+ if len(kws) >= threshold:
+ # Get full keyword data for each keyword in cluster
+ cluster_data = [
+ k for k in keywords
+ if k['keyword'].lower() in kws
+ ]
+
+ # Calculate cluster metrics
+ total_impressions = sum(k['impressions'] for k in cluster_data)
+ total_clicks = sum(k['clicks'] for k in cluster_data)
+ avg_position = sum(k['position'] for k in cluster_data) / len(cluster_data)
+
+ clusters.append({
+ 'topic': word,
+ 'keywords': cluster_data,
+ 'keyword_count': len(cluster_data),
+ 'total_impressions': total_impressions,
+ 'total_clicks': total_clicks,
+ 'avg_position': avg_position
+ })
+
+ # Sort clusters by total impressions
+ clusters.sort(key=lambda x: x['total_impressions'], reverse=True)
+
+ return clusters
+ except Exception as e:
+ logger.error(f"Error creating keyword clusters: {str(e)}")
+ return []
+
+def generate_content_brief(keyword: str, site_url: str) -> Dict[str, Any]:
+ """Generate a content brief for a keyword based on GSC data.
+
+ Args:
+ keyword: Target keyword
+ site_url: URL of the site
+
+ Returns:
+ Dict[str, Any]: Content brief
+ """
+ try:
+ # Get keyword insights
+ insights = get_keyword_insights(keyword, site_url, days=90)
+
+ # Get top keywords (for related keywords)
+ all_keywords = get_top_keywords(site_url, days=90, limit=1000)
+
+ # Find related keywords
+ related_keywords = [
+ k for k in all_keywords
+ if keyword.lower() in k['keyword'].lower() and k['keyword'].lower() != keyword.lower()
+ ]
+
+ # Sort related keywords by impressions
+ related_keywords.sort(key=lambda x: x['impressions'], reverse=True)
+
+ # Create content brief
+ brief = {
+ 'primary_keyword': keyword,
+ 'search_metrics': {
+ 'monthly_impressions': insights['total_impressions'] // 3, # Approximate monthly
+ 'monthly_clicks': insights['total_clicks'] // 3,
+ 'avg_position': insights['avg_position'],
+ 'avg_ctr': insights['avg_ctr']
+ },
+ 'related_keywords': related_keywords[:20], # Top 20 related keywords
+ 'competing_pages': insights['pages'],
+ 'recommended_headings': [], # Will be filled by AI
+ 'content_suggestions': [] # Will be filled by AI
+ }
+
+ return brief
+ except Exception as e:
+ logger.error(f"Error generating content brief: {str(e)}")
+ return {
+ 'primary_keyword': keyword,
+ 'search_metrics': {
+ 'monthly_impressions': 0,
+ 'monthly_clicks': 0,
+ 'avg_position': 0,
+ 'avg_ctr': 0
+ },
+ 'related_keywords': [],
+ 'competing_pages': [],
+ 'recommended_headings': [],
+ 'content_suggestions': []
+ }
+```
+
+#### Step 2: Integrate with Content Generation
+
+1. **Create GSC-Enhanced Content Generator**
+
+Create a new file at `/workspace/AI-Writer/lib/integrations/google/content_generator.py`:
+
+```python
+"""Content generator enhanced with Google Search Console data."""
+
+from typing import Dict, List, Any, Optional
+import streamlit as st
+from loguru import logger
+from .keyword_research import generate_content_brief
+from ...ai_writers.blog_writer import generate_blog_post
+
+def generate_seo_optimized_content(
+ keyword: str,
+ site_url: str,
+ content_type: str = "blog",
+ length: str = "medium"
+) -> Dict[str, Any]:
+ """Generate SEO-optimized content using GSC data.
+
+ Args:
+ keyword: Target keyword
+ site_url: URL of the site
+ content_type: Type of content to generate
+ length: Content length
+
+ Returns:
+ Dict[str, Any]: Generated content
+ """
+ try:
+ # Generate content brief
+ brief = generate_content_brief(keyword, site_url)
+
+ # Extract related keywords
+ related_keywords = [k['keyword'] for k in brief['related_keywords']]
+
+ # Prepare prompt enhancements
+ prompt_enhancements = {
+ 'primary_keyword': brief['primary_keyword'],
+ 'related_keywords': related_keywords,
+ 'search_position': brief['search_metrics']['avg_position'],
+ 'monthly_impressions': brief['search_metrics']['monthly_impressions'],
+ 'competing_urls': [p['page'] for p in brief['competing_pages'][:5]]
+ }
+
+ # Generate content with enhanced prompt
+ content = generate_blog_post(
+ keyword=keyword,
+ content_type=content_type,
+ length=length,
+ prompt_enhancements=prompt_enhancements
+ )
+
+ # Add brief data to the result
+ content['brief'] = brief
+
+ return content
+ except Exception as e:
+ logger.error(f"Error generating SEO-optimized content: {str(e)}")
+ return {
+ 'title': f"Error generating content for '{keyword}'",
+ 'content': f"An error occurred: {str(e)}",
+ 'brief': {}
+ }
+```
+
+2. **Create Content Optimization Component**
+
+Create a new file at `/workspace/AI-Writer/lib/integrations/google/content_optimizer.py`:
+
+```python
+"""Content optimization using Google Search Console data."""
+
+from typing import Dict, List, Any, Optional
+import streamlit as st
+from loguru import logger
+from .search_console import get_keyword_insights
+from ...seo_tools.analyzer import analyze_content_for_seo
+
+def optimize_existing_content(
+ content: str,
+ keyword: str,
+ site_url: str
+) -> Dict[str, Any]:
+ """Optimize existing content using GSC data.
+
+ Args:
+ content: Existing content
+ keyword: Target keyword
+ site_url: URL of the site
+
+ Returns:
+ Dict[str, Any]: Optimization suggestions
+ """
+ try:
+ # Get keyword insights
+ insights = get_keyword_insights(keyword, site_url, days=90)
+
+ # Analyze content for SEO
+ seo_analysis = analyze_content_for_seo(content, keyword)
+
+ # Generate optimization suggestions
+ suggestions = []
+
+ # Check keyword density
+ if seo_analysis['keyword_density'] < 0.5:
+ suggestions.append({
+ 'type': 'keyword_density',
+ 'severity': 'high',
+ 'message': f"Keyword density is too low ({seo_analysis['keyword_density']:.2f}%). Aim for 1-2%.",
+ 'action': "Add more instances of the keyword in a natural way."
+ })
+ elif seo_analysis['keyword_density'] > 3:
+ suggestions.append({
+ 'type': 'keyword_density',
+ 'severity': 'medium',
+ 'message': f"Keyword density is too high ({seo_analysis['keyword_density']:.2f}%). Aim for 1-2%.",
+ 'action': "Reduce keyword usage to avoid keyword stuffing."
+ })
+
+ # Check title
+ if keyword.lower() not in seo_analysis['title'].lower():
+ suggestions.append({
+ 'type': 'title',
+ 'severity': 'high',
+ 'message': "Primary keyword is missing from the title.",
+ 'action': f"Add '{keyword}' to the title in a natural way."
+ })
+
+ # Check headings
+ if not any(keyword.lower() in h.lower() for h in seo_analysis['headings']):
+ suggestions.append({
+ 'type': 'headings',
+ 'severity': 'medium',
+ 'message': "Primary keyword is missing from all headings.",
+ 'action': f"Add '{keyword}' to at least one heading (preferably H2)."
+ })
+
+ # Check content length
+ if seo_analysis['word_count'] < 300:
+ suggestions.append({
+ 'type': 'content_length',
+ 'severity': 'high',
+ 'message': f"Content is too short ({seo_analysis['word_count']} words). Aim for at least 800 words.",
+ 'action': "Expand the content with more valuable information."
+ })
+ elif seo_analysis['word_count'] < 800:
+ suggestions.append({
+ 'type': 'content_length',
+ 'severity': 'medium',
+ 'message': f"Content is relatively short ({seo_analysis['word_count']} words). Aim for 1000+ words for competitive keywords.",
+ 'action': "Consider adding more depth to the content."
+ })
+
+ # Check readability
+ if seo_analysis['readability_score'] < 50:
+ suggestions.append({
+ 'type': 'readability',
+ 'severity': 'medium',
+ 'message': f"Content readability is low ({seo_analysis['readability_score']}/100).",
+ 'action': "Simplify sentences, use shorter paragraphs, and avoid jargon."
+ })
+
+ # Check competing pages
+ competing_pages = insights['pages']
+ if competing_pages:
+ top_competing_page = competing_pages[0]
+ suggestions.append({
+ 'type': 'competition',
+ 'severity': 'info',
+ 'message': f"Your top competing page ranks at position {top_competing_page['position']:.1f}.",
+ 'action': f"Review the content at {top_competing_page['page']} for insights."
+ })
+
+ # Compile results
+ result = {
+ 'keyword': keyword,
+ 'search_metrics': {
+ 'monthly_impressions': insights['total_impressions'] // 3,
+ 'monthly_clicks': insights['total_clicks'] // 3,
+ 'avg_position': insights['avg_position'],
+ 'avg_ctr': insights['avg_ctr']
+ },
+ 'seo_analysis': seo_analysis,
+ 'suggestions': suggestions
+ }
+
+ return result
+ except Exception as e:
+ logger.error(f"Error optimizing content: {str(e)}")
+ return {
+ 'keyword': keyword,
+ 'search_metrics': {},
+ 'seo_analysis': {},
+ 'suggestions': [{
+ 'type': 'error',
+ 'severity': 'high',
+ 'message': f"Error analyzing content: {str(e)}",
+ 'action': "Please try again or contact support."
+ }]
+ }
+```
+
+#### Step 3: Create Content Planning Component
+
+Create a new file at `/workspace/AI-Writer/lib/integrations/google/content_planner.py`:
+
+```python
+"""Content planning using Google Search Console data."""
+
+import streamlit as st
+from typing import Dict, List, Any, Optional
+from datetime import datetime, timedelta
+from loguru import logger
+from .keyword_research import get_content_opportunities, get_keyword_clusters
+from .search_console import get_top_keywords
+
+def render_content_planner(site_url: str):
+ """Render content planning interface using GSC data.
+
+ Args:
+ site_url: URL of the site
+ """
+ st.markdown("## Content Planning")
+ st.markdown("""
+ Use your Google Search Console data to plan your content strategy.
+ Identify opportunities, organize topics, and create a content calendar.
+ """)
+
+ # Create tabs
+ tab1, tab2, tab3 = st.tabs(["Content Opportunities", "Topic Clusters", "Content Calendar"])
+
+ with tab1:
+ st.markdown("### Content Opportunities")
+ st.markdown("""
+ Find keywords where you're ranking on page 1-2 but not in the top 3.
+ These are opportunities to improve existing content or create new content.
+ """)
+
+ # Filters
+ col1, col2 = st.columns(2)
+
+ with col1:
+ min_impressions = st.slider(
+ "Minimum monthly impressions",
+ min_value=10,
+ max_value=1000,
+ value=100,
+ step=10
+ )
+
+ with col2:
+ max_position = st.slider(
+ "Maximum position",
+ min_value=5.0,
+ max_value=30.0,
+ value=20.0,
+ step=1.0
+ )
+
+ # Get opportunities
+ if st.button("Find Opportunities"):
+ with st.spinner("Analyzing search data..."):
+ opportunities = get_content_opportunities(
+ site_url=site_url,
+ min_impressions=min_impressions,
+ max_position=max_position
+ )
+
+ if not opportunities:
+ st.info("No content opportunities found with the current filters.")
+ else:
+ # Display opportunities
+ st.markdown(f"Found {len(opportunities)} content opportunities:")
+
+ # Create dataframe
+ st.dataframe(
+ opportunities,
+ column_config={
+ "keyword": "Keyword",
+ "clicks": st.column_config.NumberColumn("Clicks", format="%d"),
+ "impressions": st.column_config.NumberColumn("Impressions", format="%d"),
+ "ctr": st.column_config.NumberColumn("CTR", format="%.2f%%"),
+ "position": st.column_config.NumberColumn("Position", format="%.1f"),
+ "potential": st.column_config.NumberColumn("Potential", format="%.1f")
+ },
+ hide_index=True
+ )
+
+ # Action buttons for selected opportunity
+ st.markdown("### Take Action")
+ selected_keyword = st.selectbox(
+ "Select a keyword to take action on",
+ options=[o['keyword'] for o in opportunities]
+ )
+
+ col1, col2 = st.columns(2)
+
+ with col1:
+ if st.button("Create New Content", key="create_new"):
+ st.session_state.selected_keyword = selected_keyword
+ st.session_state.action = "create_content"
+ st.experimental_rerun()
+
+ with col2:
+ if st.button("Optimize Existing Content", key="optimize"):
+ st.session_state.selected_keyword = selected_keyword
+ st.session_state.action = "optimize_content"
+ st.experimental_rerun()
+
+ with tab2:
+ st.markdown("### Topic Clusters")
+ st.markdown("""
+ Group your keywords into topic clusters to organize your content strategy.
+ This helps identify themes and create comprehensive content around topics.
+ """)
+
+ # Cluster settings
+ min_cluster_size = st.slider(
+ "Minimum cluster size",
+ min_value=2,
+ max_value=10,
+ value=3,
+ step=1
+ )
+
+ # Get clusters
+ if st.button("Generate Topic Clusters"):
+ with st.spinner("Analyzing keywords..."):
+ # Get top keywords
+ keywords = get_top_keywords(site_url, days=90, limit=1000)
+
+ # Generate clusters
+ clusters = get_keyword_clusters(
+ keywords=keywords,
+ threshold=min_cluster_size
+ )
+
+ if not clusters:
+ st.info("No topic clusters found with the current settings.")
+ else:
+ # Display clusters
+ st.markdown(f"Found {len(clusters)} topic clusters:")
+
+ for i, cluster in enumerate(clusters):
+ with st.expander(f"Cluster: {cluster['topic']} ({cluster['keyword_count']} keywords)"):
+ st.markdown(f"**Total Impressions:** {cluster['total_impressions']:,}")
+ st.markdown(f"**Total Clicks:** {cluster['total_clicks']:,}")
+ st.markdown(f"**Average Position:** {cluster['avg_position']:.1f}")
+
+ # Display keywords in cluster
+ st.dataframe(
+ cluster['keywords'],
+ column_config={
+ "keyword": "Keyword",
+ "clicks": st.column_config.NumberColumn("Clicks", format="%d"),
+ "impressions": st.column_config.NumberColumn("Impressions", format="%d"),
+ "position": st.column_config.NumberColumn("Position", format="%.1f")
+ },
+ hide_index=True
+ )
+
+ # Action button
+ if st.button("Create Cluster Content", key=f"cluster_{i}"):
+ st.session_state.selected_cluster = cluster
+ st.session_state.action = "create_cluster_content"
+ st.experimental_rerun()
+
+ with tab3:
+ st.markdown("### Content Calendar")
+ st.markdown("""
+ Create a content calendar based on your search data and opportunities.
+ Plan your content strategy for the coming weeks or months.
+ """)
+
+ # Calendar settings
+ num_weeks = st.slider(
+ "Number of weeks to plan",
+ min_value=1,
+ max_value=12,
+ value=4,
+ step=1
+ )
+
+ posts_per_week = st.slider(
+ "Posts per week",
+ min_value=1,
+ max_value=7,
+ value=2,
+ step=1
+ )
+
+ # Generate calendar
+ if st.button("Generate Content Calendar"):
+ with st.spinner("Creating content calendar..."):
+ # Get opportunities
+ opportunities = get_content_opportunities(
+ site_url=site_url,
+ min_impressions=50,
+ max_position=30.0
+ )
+
+ # Sort by potential
+ opportunities.sort(key=lambda x: x['potential'], reverse=True)
+
+ # Calculate total posts needed
+ total_posts = num_weeks * posts_per_week
+
+ # Limit opportunities to needed posts
+ selected_opportunities = opportunities[:total_posts]
+
+ # Create calendar
+ calendar = []
+ start_date = datetime.now() + timedelta(days=7) # Start next week
+
+ for week in range(num_weeks):
+ week_start = start_date + timedelta(weeks=week)
+
+ for day in range(posts_per_week):
+ opp_index = week * posts_per_week + day
+
+ if opp_index < len(selected_opportunities):
+ post_date = week_start + timedelta(days=day)
+
+ calendar.append({
+ 'date': post_date.strftime('%Y-%m-%d'),
+ 'keyword': selected_opportunities[opp_index]['keyword'],
+ 'impressions': selected_opportunities[opp_index]['impressions'],
+ 'position': selected_opportunities[opp_index]['position'],
+ 'potential': selected_opportunities[opp_index]['potential']
+ })
+
+ if not calendar:
+ st.info("Could not generate content calendar. Try adjusting your filters.")
+ else:
+ # Display calendar
+ st.markdown(f"Generated content calendar with {len(calendar)} posts:")
+
+ # Group by week
+ weeks = {}
+ for post in calendar:
+ post_date = datetime.strptime(post['date'], '%Y-%m-%d')
+ week_num = post_date.isocalendar()[1]
+ week_start = post_date - timedelta(days=post_date.weekday())
+ week_key = f"Week {week_num} ({week_start.strftime('%b %d')})"
+
+ if week_key not in weeks:
+ weeks[week_key] = []
+
+ weeks[week_key].append(post)
+
+ # Display by week
+ for week_key, posts in weeks.items():
+ with st.expander(week_key, expanded=True):
+ for post in posts:
+ st.markdown(f"**{post['date']}:** {post['keyword']}")
+ st.markdown(f"Impressions: {post['impressions']:,} | Position: {post['position']:.1f}")
+ st.markdown("---")
+
+ # Export button
+ if st.download_button(
+ label="Export Calendar (CSV)",
+ data="\n".join([
+ "date,keyword,impressions,position,potential",
+ *[f"{post['date']},{post['keyword']},{post['impressions']},{post['position']},{post['potential']}" for post in calendar]
+ ]),
+ file_name="content_calendar.csv",
+ mime="text/csv"
+ ):
+ st.success("Calendar exported successfully!")
+```
+
+#### Step 4: Integrate with Content Writers
+
+1. **Update Blog Writer**
+
+Modify the existing blog writer to accept GSC data:
+
+```python
+# Add to blog_writer.py
+
+def generate_blog_post(
+ keyword: str,
+ content_type: str = "blog",
+ length: str = "medium",
+ prompt_enhancements: Dict[str, Any] = None
+) -> Dict[str, Any]:
+ """Generate a blog post with optional GSC data enhancement.
+
+ Args:
+ keyword: Target keyword
+ content_type: Type of content to generate
+ length: Content length
+ prompt_enhancements: Optional GSC data for prompt enhancement
+
+ Returns:
+ Dict[str, Any]: Generated content
+ """
+ # Existing code...
+
+ # Enhance prompt with GSC data if available
+ if prompt_enhancements:
+ # Add related keywords to prompt
+ if 'related_keywords' in prompt_enhancements:
+ related_kw_str = ", ".join(prompt_enhancements['related_keywords'][:5])
+ prompt += f"\nInclude these related keywords naturally: {related_kw_str}"
+
+ # Add search position context
+ if 'search_position' in prompt_enhancements:
+ position = prompt_enhancements['search_position']
+ if position > 10:
+ prompt += f"\nThis keyword currently ranks at position {position:.1f}, which is on page 2. Create comprehensive content to improve ranking."
+ else:
+ prompt += f"\nThis keyword currently ranks at position {position:.1f}. Enhance the content to improve ranking further."
+
+ # Add impression data
+ if 'monthly_impressions' in prompt_enhancements:
+ impressions = prompt_enhancements['monthly_impressions']
+ prompt += f"\nThis keyword gets approximately {impressions} monthly impressions."
+
+ # Add competing URLs
+ if 'competing_urls' in prompt_enhancements:
+ prompt += "\nMake sure your content is more comprehensive than competing pages."
+
+ # Continue with existing code...
+```
+
+2. **Add GSC Data to SEO Tools**
+
+Update the SEO analyzer to incorporate GSC data:
+
+```python
+# Add to seo_tools/analyzer.py
+
+def analyze_content_with_gsc(
+ content: str,
+ keyword: str,
+ gsc_data: Dict[str, Any]
+) -> Dict[str, Any]:
+ """Analyze content with GSC data enhancement.
+
+ Args:
+ content: Content to analyze
+ keyword: Target keyword
+ gsc_data: Google Search Console data
+
+ Returns:
+ Dict[str, Any]: Enhanced analysis
+ """
+ # Get basic SEO analysis
+ analysis = analyze_content_for_seo(content, keyword)
+
+ # Enhance with GSC data
+ if gsc_data:
+ # Add search metrics
+ analysis['search_metrics'] = {
+ 'impressions': gsc_data.get('impressions', 0),
+ 'clicks': gsc_data.get('clicks', 0),
+ 'position': gsc_data.get('position', 0),
+ 'ctr': gsc_data.get('ctr', 0)
+ }
+
+ # Add competition analysis
+ if 'competing_pages' in gsc_data:
+ analysis['competition'] = {
+ 'competing_pages': gsc_data['competing_pages'],
+ 'ranking_gap': analysis['word_count'] - gsc_data.get('avg_competitor_length', 0)
+ }
+
+ # Add keyword opportunity score
+ if 'position' in gsc_data and 'impressions' in gsc_data:
+ position = gsc_data['position']
+ impressions = gsc_data['impressions']
+
+ # Calculate opportunity score (higher for keywords with good impressions but not yet in top 3)
+ if position <= 3:
+ opportunity_score = 30 # Already ranking well
+ elif position <= 10:
+ opportunity_score = 70 # On first page but not top 3
+ elif position <= 20:
+ opportunity_score = 50 # On second page
+ else:
+ opportunity_score = 30 # Beyond second page
+
+ # Adjust for impressions
+ if impressions > 1000:
+ opportunity_score += 30
+ elif impressions > 500:
+ opportunity_score += 20
+ elif impressions > 100:
+ opportunity_score += 10
+
+ # Cap at 100
+ opportunity_score = min(opportunity_score, 100)
+
+ analysis['opportunity_score'] = opportunity_score
+
+ return analysis
+```
+
+### Phase 3: Testing Plan
+
+#### Step 1: Unit Testing
+
+1. **Create Test Files**
+
+Create a new file at `/workspace/AI-Writer/tests/integrations/google/test_auth.py`:
+
+```python
+"""Unit tests for Google authentication module."""
+
+import unittest
+from unittest.mock import patch, MagicMock
+import os
+import sys
+import json
+from pathlib import Path
+
+# Add project root to path
+sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
+
+from lib.integrations.google.auth import (
+ get_google_auth_url,
+ handle_auth_callback,
+ save_credentials,
+ load_credentials,
+ get_user_info,
+ credentials_to_dict,
+ is_authenticated
+)
+
+class TestGoogleAuth(unittest.TestCase):
+ """Test cases for Google authentication module."""
+
+ def setUp(self):
+ """Set up test environment."""
+ # Mock session state
+ self.session_state_patch = patch('streamlit.session_state', {})
+ self.mock_session_state = self.session_state_patch.start()
+
+ # Mock secrets
+ self.secrets_patch = patch('streamlit.secrets', {
+ 'google_oauth': {
+ 'web': {
+ 'client_id': 'test_client_id',
+ 'client_secret': 'test_client_secret',
+ 'redirect_uris': ['http://localhost:8501/oauth/callback']
+ }
+ }
+ })
+ self.mock_secrets = self.secrets_patch.start()
+
+ def tearDown(self):
+ """Clean up after tests."""
+ self.session_state_patch.stop()
+ self.secrets_patch.stop()
+
+ @patch('lib.integrations.google.auth.Flow')
+ def test_get_google_auth_url(self, mock_flow):
+ """Test generating Google auth URL."""
+ # Mock flow instance
+ mock_flow_instance = MagicMock()
+ mock_flow_instance.authorization_url.return_value = ('https://test-auth-url.com', None)
+ mock_flow.from_client_config.return_value = mock_flow_instance
+
+ # Call function
+ url = get_google_auth_url()
+
+ # Assertions
+ self.assertEqual(url, 'https://test-auth-url.com')
+ mock_flow.from_client_config.assert_called_once()
+ mock_flow_instance.authorization_url.assert_called_once()
+
+ @patch('lib.integrations.google.auth.get_user_info')
+ def test_handle_auth_callback(self, mock_get_user_info):
+ """Test handling auth callback."""
+ # Mock flow in session state
+ mock_flow = MagicMock()
+ mock_flow.credentials = MagicMock()
+ self.mock_session_state['google_auth_flow'] = mock_flow
+
+ # Mock user info
+ mock_get_user_info.return_value = {'email': 'test@example.com', 'name': 'Test User'}
+
+ # Call function
+ success, user_info = handle_auth_callback('test_code')
+
+ # Assertions
+ self.assertTrue(success)
+ self.assertEqual(user_info['email'], 'test@example.com')
+ mock_flow.fetch_token.assert_called_once_with(code='test_code')
+ mock_get_user_info.assert_called_once()
+
+ @patch('builtins.open', new_callable=unittest.mock.mock_open)
+ @patch('json.dump')
+ @patch('lib.integrations.google.auth.get_user_info')
+ def test_save_credentials(self, mock_get_user_info, mock_json_dump, mock_open):
+ """Test saving credentials."""
+ # Mock credentials
+ mock_credentials = MagicMock()
+ mock_credentials.token = 'test_token'
+ mock_credentials.refresh_token = 'test_refresh_token'
+
+ # Mock user info
+ mock_get_user_info.return_value = {'email': 'test@example.com'}
+
+ # Call function
+ result = save_credentials(mock_credentials)
+
+ # Assertions
+ self.assertTrue(result)
+ mock_get_user_info.assert_called_once()
+ mock_open.assert_called_once()
+ mock_json_dump.assert_called_once()
+
+ def test_credentials_to_dict(self):
+ """Test converting credentials to dictionary."""
+ # Mock credentials
+ mock_credentials = MagicMock()
+ mock_credentials.token = 'test_token'
+ mock_credentials.refresh_token = 'test_refresh_token'
+ mock_credentials.token_uri = 'test_token_uri'
+ mock_credentials.client_id = 'test_client_id'
+ mock_credentials.client_secret = 'test_client_secret'
+ mock_credentials.scopes = ['test_scope']
+
+ # Call function
+ result = credentials_to_dict(mock_credentials)
+
+ # Assertions
+ self.assertEqual(result['token'], 'test_token')
+ self.assertEqual(result['refresh_token'], 'test_refresh_token')
+ self.assertEqual(result['client_id'], 'test_client_id')
+ self.assertEqual(result['scopes'], ['test_scope'])
+
+ def test_is_authenticated(self):
+ """Test authentication check."""
+ # Test not authenticated
+ self.assertFalse(is_authenticated())
+
+ # Test authenticated
+ self.mock_session_state['google_credentials'] = {'token': 'test'}
+ self.mock_session_state['google_user_info'] = {'email': 'test@example.com'}
+ self.assertTrue(is_authenticated())
+
+if __name__ == '__main__':
+ unittest.main()
+```
+
+2. **Create Test for Search Console API**
+
+Create a new file at `/workspace/AI-Writer/tests/integrations/google/test_search_console.py`:
+
+```python
+"""Unit tests for Google Search Console API module."""
+
+import unittest
+from unittest.mock import patch, MagicMock
+import os
+import sys
+from pathlib import Path
+from datetime import datetime, timedelta
+
+# Add project root to path
+sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
+
+from lib.integrations.google.search_console import (
+ get_search_console_service,
+ get_site_list,
+ get_search_analytics,
+ get_top_keywords,
+ get_top_pages,
+ get_keyword_insights
+)
+
+class TestSearchConsole(unittest.TestCase):
+ """Test cases for Google Search Console API module."""
+
+ def setUp(self):
+ """Set up test environment."""
+ # Mock session state
+ self.session_state_patch = patch('streamlit.session_state', {
+ 'google_credentials': {
+ 'token': 'test_token',
+ 'refresh_token': 'test_refresh_token',
+ 'token_uri': 'test_token_uri',
+ 'client_id': 'test_client_id',
+ 'client_secret': 'test_client_secret',
+ 'scopes': ['test_scope']
+ }
+ })
+ self.mock_session_state = self.session_state_patch.start()
+
+ def tearDown(self):
+ """Clean up after tests."""
+ self.session_state_patch.stop()
+
+ @patch('lib.integrations.google.search_console.build')
+ @patch('lib.integrations.google.search_console.Credentials')
+ def test_get_search_console_service(self, mock_credentials, mock_build):
+ """Test building Search Console service."""
+ # Mock credentials
+ mock_credentials_instance = MagicMock()
+ mock_credentials.from_authorized_user_info.return_value = mock_credentials_instance
+
+ # Mock service
+ mock_service = MagicMock()
+ mock_build.return_value = mock_service
+
+ # Call function
+ service = get_search_console_service(self.mock_session_state['google_credentials'])
+
+ # Assertions
+ self.assertEqual(service, mock_service)
+ mock_credentials.from_authorized_user_info.assert_called_once()
+ mock_build.assert_called_once_with('searchconsole', 'v1', credentials=mock_credentials_instance)
+
+ @patch('lib.integrations.google.search_console.get_search_console_service')
+ def test_get_site_list(self, mock_get_service):
+ """Test getting site list."""
+ # Mock service
+ mock_service = MagicMock()
+ mock_sites = MagicMock()
+ mock_sites.list().execute.return_value = {
+ 'siteEntry': [
+ {
+ 'siteUrl': 'https://example.com/',
+ 'permissionLevel': 'siteOwner'
+ },
+ {
+ 'siteUrl': 'sc-domain:example.org',
+ 'permissionLevel': 'siteFullUser'
+ },
+ {
+ 'siteUrl': 'https://example.net/',
+ 'permissionLevel': 'siteRestrictedUser' # Should be filtered out
+ }
+ ]
+ }
+ mock_service.sites.return_value = mock_sites
+ mock_get_service.return_value = mock_service
+
+ # Call function
+ sites = get_site_list()
+
+ # Assertions
+ self.assertEqual(len(sites), 2) # Only 2 sites with sufficient permissions
+ self.assertEqual(sites[0]['site_url'], 'https://example.com/')
+ self.assertEqual(sites[0]['permission_level'], 'siteOwner')
+ self.assertEqual(sites[0]['site_type'], 'Web')
+ self.assertEqual(sites[1]['site_url'], 'sc-domain:example.org')
+ self.assertEqual(sites[1]['site_type'], 'Domain Property')
+
+ @patch('lib.integrations.google.search_console.get_search_console_service')
+ def test_get_search_analytics(self, mock_get_service):
+ """Test getting search analytics data."""
+ # Mock service
+ mock_service = MagicMock()
+ mock_analytics = MagicMock()
+ mock_analytics.query().execute.return_value = {
+ 'rows': [
+ {
+ 'keys': ['test keyword'],
+ 'clicks': 100,
+ 'impressions': 1000,
+ 'ctr': 0.1,
+ 'position': 5.5
+ }
+ ]
+ }
+ mock_service.searchanalytics.return_value = mock_analytics
+ mock_get_service.return_value = mock_service
+
+ # Call function
+ start_date = datetime.now() - timedelta(days=30)
+ end_date = datetime.now()
+ result = get_search_analytics(
+ site_url='https://example.com/',
+ start_date=start_date,
+ end_date=end_date,
+ dimensions=['query'],
+ row_limit=100
+ )
+
+ # Assertions
+ self.assertEqual(len(result['rows']), 1)
+ mock_analytics.query.assert_called_once()
+ call_args = mock_analytics.query.call_args[1]
+ self.assertEqual(call_args['siteUrl'], 'https://example.com/')
+ self.assertEqual(call_args['body']['dimensions'], ['query'])
+ self.assertEqual(call_args['body']['rowLimit'], 100)
+
+ @patch('lib.integrations.google.search_console.get_search_analytics')
+ def test_get_top_keywords(self, mock_get_analytics):
+ """Test getting top keywords."""
+ # Mock analytics response
+ mock_get_analytics.return_value = {
+ 'rows': [
+ {
+ 'keys': ['keyword1'],
+ 'clicks': 100,
+ 'impressions': 1000,
+ 'ctr': 0.1,
+ 'position': 5.5
+ },
+ {
+ 'keys': ['keyword2'],
+ 'clicks': 200,
+ 'impressions': 2000,
+ 'ctr': 0.1,
+ 'position': 3.2
+ }
+ ]
+ }
+
+ # Call function
+ keywords = get_top_keywords(
+ site_url='https://example.com/',
+ days=30,
+ limit=100
+ )
+
+ # Assertions
+ self.assertEqual(len(keywords), 2)
+ self.assertEqual(keywords[0]['keyword'], 'keyword1')
+ self.assertEqual(keywords[0]['clicks'], 100)
+ self.assertEqual(keywords[0]['impressions'], 1000)
+ self.assertEqual(keywords[0]['ctr'], 10.0) # Converted to percentage
+ self.assertEqual(keywords[0]['position'], 5.5)
+ mock_get_analytics.assert_called_once()
+
+if __name__ == '__main__':
+ unittest.main()
+```
+
+#### Step 2: Integration Testing
+
+1. **Create Integration Test Script**
+
+Create a new file at `/workspace/AI-Writer/tests/integrations/google/test_integration.py`:
+
+```python
+"""Integration tests for Google integration."""
+
+import unittest
+import os
+import sys
+from pathlib import Path
+import streamlit as st
+from unittest.mock import patch, MagicMock
+
+# Add project root to path
+sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent))
+
+from lib.integrations.google.auth import get_google_auth_url, handle_auth_callback
+from lib.integrations.google.search_console import get_site_list, get_top_keywords
+from lib.integrations.google.keyword_research import get_content_opportunities
+from lib.integrations.google.content_generator import generate_seo_optimized_content
+
+class TestGoogleIntegration(unittest.TestCase):
+ """Integration tests for Google integration."""
+
+ @classmethod
+ def setUpClass(cls):
+ """Set up test environment."""
+ # Check if we have real credentials for testing
+ cls.has_real_credentials = os.environ.get('GOOGLE_TEST_CREDENTIALS') is not None
+
+ if cls.has_real_credentials:
+ # Load real credentials from environment
+ import json
+ credentials_json = os.environ.get('GOOGLE_TEST_CREDENTIALS')
+ cls.test_credentials = json.loads(credentials_json)
+
+ # Set up session state with real credentials
+ st.session_state['google_credentials'] = cls.test_credentials
+ st.session_state['google_user_info'] = {
+ 'email': os.environ.get('GOOGLE_TEST_EMAIL', 'test@example.com'),
+ 'name': 'Test User'
+ }
+
+ def test_auth_flow(self):
+ """Test authentication flow."""
+ # Skip if no real credentials
+ if not self.has_real_credentials:
+ self.skipTest("No real credentials available for testing")
+
+ # Get auth URL
+ auth_url = get_google_auth_url()
+
+ # Verify URL format
+ self.assertTrue(auth_url.startswith('https://accounts.google.com/o/oauth2/auth'))
+ self.assertIn('client_id=', auth_url)
+ self.assertIn('redirect_uri=', auth_url)
+ self.assertIn('scope=', auth_url)
+
+ def test_search_console_access(self):
+ """Test Search Console API access."""
+ # Skip if no real credentials
+ if not self.has_real_credentials:
+ self.skipTest("No real credentials available for testing")
+
+ # Get site list
+ sites = get_site_list()
+
+ # Verify we got some sites
+ self.assertIsInstance(sites, list)
+
+ # If we have sites, test keyword data
+ if sites:
+ site_url = sites[0]['site_url']
+
+ # Get top keywords
+ keywords = get_top_keywords(site_url, days=30, limit=10)
+
+ # Verify we got keyword data
+ self.assertIsInstance(keywords, list)
+
+ # If we have keywords, test content opportunities
+ if keywords:
+ # Get content opportunities
+ opportunities = get_content_opportunities(site_url, min_impressions=10, max_position=50.0)
+
+ # Verify we got opportunities
+ self.assertIsInstance(opportunities, list)
+
+ @patch('lib.integrations.google.content_generator.generate_blog_post')
+ def test_content_generation(self, mock_generate_blog_post):
+ """Test content generation with GSC data."""
+ # Mock blog post generation
+ mock_generate_blog_post.return_value = {
+ 'title': 'Test Blog Post',
+ 'content': 'This is a test blog post.'
+ }
+
+ # Generate content
+ content = generate_seo_optimized_content(
+ keyword='test keyword',
+ site_url='https://example.com/',
+ content_type='blog',
+ length='medium'
+ )
+
+ # Verify content was generated
+ self.assertEqual(content['title'], 'Test Blog Post')
+ self.assertEqual(content['content'], 'This is a test blog post.')
+
+ # Verify prompt enhancements were passed
+ mock_generate_blog_post.assert_called_once()
+ call_args = mock_generate_blog_post.call_args[1]
+ self.assertEqual(call_args['keyword'], 'test keyword')
+ self.assertEqual(call_args['content_type'], 'blog')
+ self.assertEqual(call_args['length'], 'medium')
+ self.assertIn('prompt_enhancements', call_args)
+
+if __name__ == '__main__':
+ unittest.main()
+```
+
+#### Step 3: End-to-End Testing
+
+1. **Create Manual Test Plan**
+
+Create a new file at `/workspace/AI-Writer/tests/integrations/google/manual_test_plan.md`:
+
+```markdown
+# Google Integration Manual Test Plan
+
+This document outlines the manual testing procedures for the Google integration in AI-Writer.
+
+## Prerequisites
+
+- Google account with access to Search Console
+- Website verified in Google Search Console
+- AI-Writer development environment set up
+
+## Test Cases
+
+### 1. Authentication Flow
+
+#### 1.1 Google Login
+
+1. Navigate to the Google Integration page
+2. Click "Sign in with Google"
+3. Complete Google authentication flow
+4. Verify user is redirected back to AI-Writer
+5. Verify user info is displayed correctly
+
+**Expected Result:** User is successfully authenticated and user info is displayed.
+
+#### 1.2 Logout
+
+1. Click "Sign Out" button
+2. Verify user is logged out
+3. Verify login button is displayed again
+
+**Expected Result:** User is successfully logged out.
+
+### 2. Search Console Access
+
+#### 2.1 Site Selection
+
+1. Authenticate with Google
+2. Verify site selector displays user's verified sites
+3. Select a site from the dropdown
+4. Verify Search Console dashboard loads
+
+**Expected Result:** User's verified sites are displayed and can be selected.
+
+#### 2.2 Top Keywords
+
+1. Select a site
+2. Navigate to "Top Keywords" tab
+3. Adjust time period slider
+4. Verify keyword data loads and updates
+
+**Expected Result:** Keyword data is displayed and updates when time period changes.
+
+#### 2.3 Top Pages
+
+1. Select a site
+2. Navigate to "Top Pages" tab
+3. Adjust time period slider
+4. Verify page data loads and updates
+
+**Expected Result:** Page data is displayed and updates when time period changes.
+
+#### 2.4 Keyword Research
+
+1. Select a site
+2. Navigate to "Keyword Research" tab
+3. Enter a keyword
+4. Verify keyword insights are displayed
+5. Verify performance charts are displayed
+6. Verify ranking pages are displayed
+
+**Expected Result:** Keyword insights and related data are displayed correctly.
+
+### 3. Content Planning
+
+#### 3.1 Content Opportunities
+
+1. Navigate to Content Planning
+2. Select "Content Opportunities" tab
+3. Adjust filters
+4. Click "Find Opportunities"
+5. Verify opportunities are displayed
+6. Select a keyword and click "Create New Content"
+7. Verify redirection to content creation
+
+**Expected Result:** Content opportunities are found and can be used for content creation.
+
+#### 3.2 Topic Clusters
+
+1. Navigate to Content Planning
+2. Select "Topic Clusters" tab
+3. Adjust cluster settings
+4. Click "Generate Topic Clusters"
+5. Verify clusters are displayed
+6. Expand a cluster to view keywords
+7. Click "Create Cluster Content"
+8. Verify redirection to content creation
+
+**Expected Result:** Topic clusters are generated and can be used for content creation.
+
+#### 3.3 Content Calendar
+
+1. Navigate to Content Planning
+2. Select "Content Calendar" tab
+3. Adjust calendar settings
+4. Click "Generate Content Calendar"
+5. Verify calendar is displayed
+6. Verify export functionality works
+
+**Expected Result:** Content calendar is generated and can be exported.
+
+### 4. Content Creation
+
+#### 4.1 SEO-Optimized Content
+
+1. Select a keyword from opportunities
+2. Click "Create New Content"
+3. Verify GSC data is incorporated in the prompt
+4. Generate content
+5. Verify content includes related keywords
+6. Verify content addresses search intent
+
+**Expected Result:** Generated content is optimized based on GSC data.
+
+#### 4.2 Content Optimization
+
+1. Select a keyword
+2. Click "Optimize Existing Content"
+3. Enter or paste existing content
+4. Click "Analyze Content"
+5. Verify optimization suggestions are displayed
+6. Apply suggestions
+7. Verify content improvements
+
+**Expected Result:** Content optimization suggestions are provided and can be applied.
+
+## Test Execution Checklist
+
+| Test Case | Status | Notes |
+|-----------|--------|-------|
+| 1.1 Google Login | | |
+| 1.2 Logout | | |
+| 2.1 Site Selection | | |
+| 2.2 Top Keywords | | |
+| 2.3 Top Pages | | |
+| 2.4 Keyword Research | | |
+| 3.1 Content Opportunities | | |
+| 3.2 Topic Clusters | | |
+| 3.3 Content Calendar | | |
+| 4.1 SEO-Optimized Content | | |
+| 4.2 Content Optimization | | |
+```
+
+#### Step 4: Automated Testing Setup
+
+1. **Create GitHub Actions Workflow**
+
+Create a new file at `/.github/workflows/google_integration_tests.yml`:
+
+```yaml
+name: Google Integration Tests
+
+on:
+ push:
+ branches: [ main ]
+ paths:
+ - 'lib/integrations/google/**'
+ - 'tests/integrations/google/**'
+ pull_request:
+ branches: [ main ]
+ paths:
+ - 'lib/integrations/google/**'
+ - 'tests/integrations/google/**'
+ workflow_dispatch:
+
+jobs:
+ unit-tests:
+ runs-on: ubuntu-latest
+
+ steps:
+ - uses: actions/checkout@v3
+
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.10'
+
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+ pip install pytest pytest-cov
+
+ - name: Run unit tests
+ run: |
+ pytest tests/integrations/google/test_auth.py tests/integrations/google/test_search_console.py -v
+
+ - name: Generate coverage report
+ run: |
+ pytest --cov=lib.integrations.google tests/integrations/google/ --cov-report=xml
+
+ - name: Upload coverage to Codecov
+ uses: codecov/codecov-action@v3
+ with:
+ file: ./coverage.xml
+ flags: google-integration
+ fail_ci_if_error: false
+
+ integration-tests:
+ runs-on: ubuntu-latest
+ if: github.event_name == 'workflow_dispatch'
+
+ steps:
+ - uses: actions/checkout@v3
+
+ - name: Set up Python
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.10'
+
+ - name: Install dependencies
+ run: |
+ python -m pip install --upgrade pip
+ if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+ pip install pytest
+
+ - name: Run integration tests
+ env:
+ GOOGLE_TEST_CREDENTIALS: ${{ secrets.GOOGLE_TEST_CREDENTIALS }}
+ GOOGLE_TEST_EMAIL: ${{ secrets.GOOGLE_TEST_EMAIL }}
+ run: |
+ pytest tests/integrations/google/test_integration.py -v
+```
+
+## Deployment Plan
+
+### Phase 1: Development Environment
+
+1. **Initial Setup**
+ - Set up Google Cloud Project
+ - Configure OAuth consent screen
+ - Create OAuth credentials
+ - Enable required APIs
+ - Implement authentication module
+
+2. **Local Testing**
+ - Test authentication flow
+ - Test Search Console API access
+ - Test content generation with GSC data
+ - Run unit tests
+
+### Phase 2: Staging Environment
+
+1. **Configuration**
+ - Create staging Google Cloud Project
+ - Configure OAuth for staging domain
+ - Update redirect URIs
+ - Set up staging environment variables
+
+2. **Integration Testing**
+ - Deploy to staging environment
+ - Test end-to-end flow
+ - Verify data accuracy
+ - Test with multiple users and sites
+
+3. **Performance Testing**
+ - Test with large datasets
+ - Optimize API calls
+ - Implement caching if needed
+ - Monitor API usage limits
+
+### Phase 3: Production Deployment
+
+1. **Final Configuration**
+ - Create production Google Cloud Project
+ - Configure OAuth for production domain
+ - Set up production environment variables
+ - Verify API quotas and limits
+
+2. **Deployment**
+ - Deploy to production
+ - Monitor for errors
+ - Verify authentication flow
+ - Test with real user accounts
+
+3. **Post-Deployment**
+ - Monitor API usage
+ - Collect user feedback
+ - Address any issues
+ - Document the integration
+
+## User Documentation
+
+### User Guide
+
+Create a user guide that explains:
+
+1. How to connect Google account
+2. How to access Search Console data
+3. How to use content opportunities
+4. How to create SEO-optimized content
+5. How to use the content calendar
+
+### Video Tutorial
+
+Create a video tutorial demonstrating:
+
+1. Initial setup process
+2. Finding content opportunities
+3. Creating content with GSC data
+4. Optimizing existing content
+5. Planning content with the calendar
+
+## Conclusion
+
+This implementation plan provides a comprehensive approach to integrating Google login and Google Search Console with AI-Writer. By following these steps, you'll enhance the platform with real user insights for content creation, improving the relevance and effectiveness of generated content.
+
+The integration will provide significant value to users by:
+
+1. Leveraging real search data for content creation
+2. Identifying content opportunities based on actual performance
+3. Optimizing content for keywords that matter to the user's audience
+4. Creating a data-driven content strategy
+5. Measuring content performance over time
+
+This feature will differentiate AI-Writer from competitors by providing a closed-loop system for content creation, optimization, and performance tracking.
\ No newline at end of file
diff --git a/Roadmap TBDs/ONBOARDING_IMPROVEMENTS.md b/Roadmap TBDs/ONBOARDING_IMPROVEMENTS.md
new file mode 100644
index 00000000..88099474
--- /dev/null
+++ b/Roadmap TBDs/ONBOARDING_IMPROVEMENTS.md
@@ -0,0 +1,631 @@
+# Onboarding Process Improvements
+
+This document outlines a comprehensive plan to improve the user onboarding experience in AI-Writer, focusing on the API key management and initial setup process.
+
+## Current Issues
+
+After analyzing the current onboarding process in `utils.api_key_manager`, several issues were identified:
+
+### User Experience Issues
+
+- **Complex Multi-step Process**: The onboarding is split across multiple steps without clear indication of progress or purpose
+- **Confusing Navigation**: Users can get lost between steps with no clear path forward
+- **Required vs. Optional**: No clear distinction between required and optional API keys
+- **No Skip Option**: Users must go through all steps even if some are not relevant to them
+- **Limited Guidance**: Insufficient contextual help for users unfamiliar with API keys
+
+### Technical Issues
+
+- **Inconsistent State Management**: Wizard state is initialized in multiple places
+- **Basic Validation**: API keys are only checked for non-emptiness, not actual validity
+- **Environment Variable Handling**: Not robust across different environments
+- **Error Handling**: Inconsistent error handling and user feedback
+- **No Testing Mechanism**: No way to test API keys during setup
+
+### UI/Design Issues
+
+- **Inconsistent Styling**: Visual inconsistency across different components
+- **Poor Mobile Experience**: Limited responsiveness for mobile users
+- **Visual Hierarchy**: Lack of clear visual distinction for important elements
+- **Help Text Visibility**: Instructions and help text are not prominent enough
+
+## Proposed Improvements
+
+### 1. Redesigned Onboarding Flow
+
+#### Welcome Screen
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β β
+β Welcome to AI-Writer! π β
+β β
+β Let's get you set up in just a few minutes. β
+β β
+β What would you like to do? β
+β β
+β β Quick Start (minimal setup) β
+β β Complete Setup (all features) β
+β β Import Configuration β
+β β
+β [Start Setup] β
+β β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+```
+
+#### API Key Setup Screen
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β β
+β Step 1 of 4: Connect AI Models β
+β ββββββββββββββ β
+β β
+β Required (choose at least one): β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β β OpenAI API Key β β
+β β [β’β’β’β’β’β’β’β’β’β’β’β’β’β’β’β’β’β’] β β
+β β β Validated successfully! β β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β β Google Gemini API Key β β
+β β [ ] β β
+β β βΉοΈ Not configured (optional) β β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β
+β Optional: β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β Anthropic API Key (Coming Soon) β β
+β β [ ] β β
+β βββββββββββββββββββββββββββββββββββββββββββββββ β
+β β
+β [Skip Optional] [Test Keys] [Continue β] β
+β β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+```
+
+#### Progress Summary Screen
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β β
+β Setup Complete! π β
+β β
+β Here's your configuration: β
+β β
+β AI Models: β
+β β OpenAI API - Connected β
+β β Google Gemini - Not configured β
+β β
+β Research Tools: β
+β β Tavily Search - Connected β
+β β Serper API - Not configured β
+β β
+β Publishing: β
+β β WordPress - Not configured β
+β β
+β You can change these settings anytime from β
+β the Settings menu. β
+β β
+β [Edit Configuration] [Start Using AI-Writer] β
+β β
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+```
+
+### 2. Technical Improvements
+
+#### Unified State Management
+
+```python
+# Create a dedicated state manager class
+class OnboardingStateManager:
+ """Manages the state of the onboarding process."""
+
+ def __init__(self):
+ """Initialize the onboarding state."""
+ if 'onboarding_state' not in st.session_state:
+ st.session_state.onboarding_state = {
+ 'current_step': 1,
+ 'total_steps': 4,
+ 'completed_steps': set(),
+ 'api_keys': {},
+ 'validated_keys': {},
+ 'setup_mode': 'quick_start', # or 'complete'
+ 'setup_complete': False
+ }
+
+ def get_state(self):
+ """Get the current onboarding state."""
+ return st.session_state.onboarding_state
+
+ def update_state(self, updates):
+ """Update the onboarding state."""
+ st.session_state.onboarding_state.update(updates)
+
+ def next_step(self):
+ """Move to the next step."""
+ current = st.session_state.onboarding_state['current_step']
+ total = st.session_state.onboarding_state['total_steps']
+
+ if current < total:
+ st.session_state.onboarding_state['current_step'] += 1
+ st.session_state.onboarding_state['completed_steps'].add(current)
+
+ def previous_step(self):
+ """Move to the previous step."""
+ if st.session_state.onboarding_state['current_step'] > 1:
+ st.session_state.onboarding_state['current_step'] -= 1
+
+ def skip_to_step(self, step):
+ """Skip to a specific step."""
+ if 1 <= step <= st.session_state.onboarding_state['total_steps']:
+ st.session_state.onboarding_state['current_step'] = step
+
+ def mark_complete(self):
+ """Mark the onboarding as complete."""
+ st.session_state.onboarding_state['setup_complete'] = True
+```
+
+#### Enhanced API Key Validation
+
+```python
+async def validate_api_key(service, key):
+ """Validate an API key by making a test request."""
+ try:
+ if service == "openai":
+ # Test OpenAI API key with a minimal request
+ import openai
+ client = openai.OpenAI(api_key=key)
+ response = await client.models.list()
+ return {"valid": True, "models": [model.id for model in response.data[:5]]}
+
+ elif service == "gemini":
+ # Test Google Gemini API key
+ import google.generativeai as genai
+ genai.configure(api_key=key)
+ models = genai.list_models()
+ return {"valid": True, "models": [model.name for model in models]}
+
+ elif service == "tavily":
+ # Test Tavily API key
+ import requests
+ response = requests.get(
+ "https://api.tavily.com/health",
+ headers={"x-api-key": key}
+ )
+ if response.status_code == 200:
+ return {"valid": True, "status": "healthy"}
+ else:
+ return {"valid": False, "error": f"Status code: {response.status_code}"}
+
+ # Add more services as needed
+
+ except Exception as e:
+ return {"valid": False, "error": str(e)}
+```
+
+#### Secure API Key Storage
+
+```python
+def save_api_keys(keys_dict):
+ """Save API keys securely."""
+ try:
+ # 1. Save to .env file
+ env_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), '.env')
+
+ # Read existing .env file
+ env_contents = {}
+ if os.path.exists(env_path):
+ with open(env_path, 'r') as f:
+ for line in f:
+ if '=' in line:
+ key, value = line.strip().split('=', 1)
+ env_contents[key] = value
+
+ # Update with new keys
+ for key_name, key_value in keys_dict.items():
+ if key_value: # Only save non-empty keys
+ env_key = f"{key_name.upper()}_API_KEY"
+ env_contents[env_key] = key_value
+
+ # Write back to .env file
+ with open(env_path, 'w') as f:
+ for key, value in env_contents.items():
+ f.write(f"{key}={value}\n")
+
+ # 2. Also store in session state for immediate use
+ for key_name, key_value in keys_dict.items():
+ if key_value:
+ st.session_state[f"{key_name}_api_key"] = key_value
+
+ # 3. Set environment variables for current session
+ for key_name, key_value in keys_dict.items():
+ if key_value:
+ os.environ[f"{key_name.upper()}_API_KEY"] = key_value
+
+ return True
+ except Exception as e:
+ logger.error(f"Error saving API keys: {str(e)}")
+ return False
+```
+
+### 3. UI/UX Improvements
+
+#### Responsive Design
+
+```python
+def render_responsive_layout():
+ """Render a responsive layout that works on mobile and desktop."""
+ # Check viewport width
+ st.markdown("""
+
+
+
+ """, unsafe_allow_html=True)
+```
+
+#### Visual Hierarchy for Required vs Optional
+
+```python
+def render_api_key_input(label, key_name, required=False, help_text=""):
+ """Render an API key input with clear visual hierarchy."""
+
+ # Add required indicator if needed
+ display_label = f"{label} {'*' if required else '(optional)'}"
+
+ # Get existing value from session state or environment
+ existing_value = st.session_state.get(f"{key_name}_api_key", "") or os.getenv(f"{key_name.upper()}_API_KEY", "")
+
+ # Render the input with appropriate styling
+ st.markdown(f"""
+
+
+
{help_text}
+
+ """, unsafe_allow_html=True)
+
+ # The actual input field
+ value = st.text_input(
+ label="", # Empty because we use custom label above
+ value=existing_value,
+ type="password",
+ key=f"input_{key_name}",
+ label_visibility="collapsed"
+ )
+
+ # Validation status
+ if value:
+ is_valid = key_name in st.session_state.get("validated_keys", {})
+ if is_valid:
+ st.success(f"β {label} validated successfully")
+ else:
+ st.info(f"β οΈ {label} not validated yet")
+
+ return value
+```
+
+#### Interactive Help and Tooltips
+
+```python
+def render_help_section(service):
+ """Render an interactive help section for getting API keys."""
+
+ help_content = {
+ "openai": {
+ "title": "How to get your OpenAI API key",
+ "steps": [
+ "Go to [OpenAI's website](https://platform.openai.com)",
+ "Sign up or log in to your account",
+ "Navigate to the API section",
+ "Click 'Create new secret key'",
+ "Copy the generated key and paste it here"
+ ],
+ "note": "Keep your API key secure and never share it publicly.",
+ "pricing": "$0.002 per 1K tokens for GPT-3.5, $0.06 per 1K tokens for GPT-4",
+ "link": "https://platform.openai.com/account/api-keys"
+ },
+ "gemini": {
+ "title": "How to get your Google Gemini API key",
+ "steps": [
+ "Visit [Google AI Studio](https://makersuite.google.com/app/apikey)",
+ "Sign in with your Google account",
+ "Click 'Create API key'",
+ "Copy the generated key and paste it here"
+ ],
+ "note": "Make sure to enable the Gemini API in your Google Cloud Console.",
+ "pricing": "Free tier available, then $0.0025 per 1K tokens",
+ "link": "https://makersuite.google.com/app/apikey"
+ }
+ # Add more services as needed
+ }
+
+ if service in help_content:
+ content = help_content[service]
+
+ with st.expander(f"π {content['title']}", expanded=False):
+ st.markdown("**Step-by-step guide:**")
+ for i, step in enumerate(content["steps"], 1):
+ st.markdown(f"{i}. {step}")
+
+ st.markdown(f"**Note:** {content['note']}")
+ st.markdown(f"**Pricing:** {content['pricing']}")
+ st.markdown(f"[Get your API key here]({content['link']})")
+```
+
+### 4. Implementation Plan
+
+#### Phase 1: Core Improvements
+
+1. **Create Unified State Manager**
+ - Implement the `OnboardingStateManager` class
+ - Refactor existing code to use the new state manager
+ - Add proper state persistence
+
+2. **Enhance API Key Validation**
+ - Implement real validation for each service
+ - Add visual feedback for validation status
+ - Create a "Test All Keys" function
+
+3. **Improve Navigation**
+ - Redesign step indicator with clear labels
+ - Add skip options for optional steps
+ - Implement a "Quick Start" mode
+
+#### Phase 2: UI/UX Enhancements
+
+1. **Redesign Input Components**
+ - Create clear visual hierarchy
+ - Add responsive design for mobile
+ - Implement interactive help sections
+
+2. **Create Summary Screens**
+ - Add welcome screen with setup options
+ - Implement completion summary screen
+ - Add configuration export/import
+
+3. **Enhance Visual Design**
+ - Update color scheme for better accessibility
+ - Add animations for transitions
+ - Implement progress indicators
+
+#### Phase 3: Advanced Features
+
+1. **Guided Tours**
+ - Add interactive tutorials
+ - Create contextual help popups
+ - Implement feature discovery
+
+2. **Smart Defaults**
+ - Suggest configurations based on user needs
+ - Implement templates for common use cases
+ - Add recommended settings
+
+3. **Troubleshooting Assistance**
+ - Add automatic error detection
+ - Create guided troubleshooting flows
+ - Implement self-healing for common issues
+
+## Code Implementation Examples
+
+### Welcome Screen Component
+
+```python
+def render_welcome_screen():
+ """Render the welcome screen for onboarding."""
+ st.markdown("""
+
+
Welcome to AI-Writer! π
+
Let's get you set up in just a few minutes.
+
+ """, unsafe_allow_html=True)
+
+ # Setup mode selection
+ setup_mode = st.radio(
+ "What would you like to do?",
+ options=["Quick Start (minimal setup)",
+ "Complete Setup (all features)",
+ "Import Configuration"],
+ index=0,
+ key="setup_mode_selection"
+ )
+
+ # Store the selection in state
+ if "onboarding_state" in st.session_state:
+ st.session_state.onboarding_state["setup_mode"] = setup_mode.split(" ")[0].lower()
+
+ # Start button
+ if st.button("Start Setup", use_container_width=True, type="primary"):
+ if "onboarding_state" in st.session_state:
+ st.session_state.onboarding_state["current_step"] = 1
+ st.rerun()
+```
+
+### API Key Manager Component
+
+```python
+def render_api_key_manager():
+ """Render the improved API key manager."""
+ # Get state manager
+ state_manager = OnboardingStateManager()
+ state = state_manager.get_state()
+
+ # Render step indicator
+ render_step_indicator(state["current_step"], state["total_steps"])
+
+ # Render appropriate step based on current_step
+ if state["current_step"] == 1:
+ render_ai_providers_step(state_manager)
+ elif state["current_step"] == 2:
+ render_research_tools_step(state_manager)
+ elif state["current_step"] == 3:
+ render_publishing_step(state_manager)
+ elif state["current_step"] == 4:
+ render_summary_step(state_manager)
+
+ # Render navigation buttons
+ render_navigation_buttons(state_manager)
+```
+
+### Improved AI Providers Step
+
+```python
+def render_ai_providers_step(state_manager):
+ """Render the improved AI providers setup step."""
+ st.markdown("## Step 1: Connect AI Models")
+ st.markdown("Configure the AI models you want to use for content generation.")
+
+ # Create tabs for required vs optional
+ tab1, tab2 = st.tabs(["Required (at least one)", "Optional Models"])
+
+ with tab1:
+ col1, col2 = st.columns(2)
+
+ with col1:
+ # OpenAI
+ openai_key = render_api_key_input(
+ "OpenAI API Key",
+ "openai",
+ required=True,
+ help_text="Powers GPT-3.5 and GPT-4 models"
+ )
+ render_help_section("openai")
+
+ with col2:
+ # Google Gemini
+ gemini_key = render_api_key_input(
+ "Google Gemini API Key",
+ "gemini",
+ required=False,
+ help_text="Powers Gemini Pro models"
+ )
+ render_help_section("gemini")
+
+ with tab2:
+ col1, col2 = st.columns(2)
+
+ with col1:
+ # Anthropic
+ anthropic_key = render_api_key_input(
+ "Anthropic API Key",
+ "anthropic",
+ required=False,
+ help_text="Powers Claude models"
+ )
+ render_help_section("anthropic")
+
+ with col2:
+ # Mistral
+ mistral_key = render_api_key_input(
+ "Mistral API Key",
+ "mistral",
+ required=False,
+ help_text="Powers Mistral models"
+ )
+ render_help_section("mistral")
+
+ # Test keys button
+ if st.button("Test API Keys", use_container_width=True):
+ with st.spinner("Testing API keys..."):
+ # Test each provided key
+ results = {}
+ if openai_key:
+ results["openai"] = asyncio.run(validate_api_key("openai", openai_key))
+ if gemini_key:
+ results["gemini"] = asyncio.run(validate_api_key("gemini", gemini_key))
+ if anthropic_key:
+ results["anthropic"] = asyncio.run(validate_api_key("anthropic", anthropic_key))
+ if mistral_key:
+ results["mistral"] = asyncio.run(validate_api_key("mistral", mistral_key))
+
+ # Store validation results
+ state_manager.update_state({"validated_keys": results})
+
+ # Display results
+ for service, result in results.items():
+ if result.get("valid", False):
+ st.success(f"β {service.title()} API key is valid")
+ else:
+ st.error(f"β {service.title()} API key is invalid: {result.get('error', 'Unknown error')}")
+
+ # Save keys to state
+ api_keys = {
+ "openai": openai_key,
+ "gemini": gemini_key,
+ "anthropic": anthropic_key,
+ "mistral": mistral_key
+ }
+ state_manager.update_state({"api_keys": api_keys})
+
+ # Check if we have at least one valid key
+ has_valid_key = any([
+ openai_key and state_manager.get_state().get("validated_keys", {}).get("openai", {}).get("valid", False),
+ gemini_key and state_manager.get_state().get("validated_keys", {}).get("gemini", {}).get("valid", False)
+ ])
+
+ if not has_valid_key and (openai_key or gemini_key):
+ st.warning("Please test your API keys before continuing")
+```
+
+## Benefits of Improved Onboarding
+
+1. **Increased User Retention**
+ - Smoother onboarding leads to higher completion rates
+ - Clear guidance reduces frustration and abandonment
+ - Faster time-to-value improves user satisfaction
+
+2. **Reduced Support Burden**
+ - Better self-service options decrease support tickets
+ - Clearer instructions prevent common setup issues
+ - Automated validation catches problems early
+
+3. **Higher Feature Adoption**
+ - Users understand available features better
+ - Guided setup encourages exploration of capabilities
+ - Contextual help improves feature discovery
+
+4. **Improved User Experience**
+ - Consistent design creates a professional impression
+ - Responsive layout works across all devices
+ - Intuitive navigation reduces cognitive load
+
+5. **Better Data Quality**
+ - Proper validation ensures working API keys
+ - Clear requirements improve data completeness
+ - Structured setup leads to better configuration
+
+## Implementation Timeline
+
+- **Week 1**: Design and prototype core improvements
+- **Week 2**: Implement unified state management and API validation
+- **Week 3**: Develop UI components and responsive design
+- **Week 4**: Create welcome and summary screens
+- **Week 5**: Add help content and contextual assistance
+- **Week 6**: Testing, refinement, and documentation
+
+## Conclusion
+
+The proposed improvements to the onboarding process will significantly enhance the user experience for new AI-Writer users. By implementing a more intuitive, guided, and responsive setup flow, we can increase user retention, reduce support needs, and help users get value from the platform faster.
+
+These changes represent a comprehensive overhaul of the current system, addressing both technical and user experience issues while maintaining compatibility with the existing codebase.
\ No newline at end of file
diff --git a/Roadmap TBDs/ROADMAP.md b/Roadmap TBDs/ROADMAP.md
new file mode 100644
index 00000000..e35573e2
--- /dev/null
+++ b/Roadmap TBDs/ROADMAP.md
@@ -0,0 +1,250 @@
+# AI-Writer Public Roadmap
+
+This roadmap outlines the planned features and improvements for the AI-Writer platform. It provides transparency into our development priorities and gives users insight into upcoming capabilities.
+
+## π¦ Roadmap Status Indicators
+
+- π’ **In Progress**: Currently being developed
+- π‘ **Planned**: Scheduled for upcoming development cycles
+- π΅ **Researching**: Under investigation and evaluation
+- β **Completed**: Released and available
+
+## ποΈ Q2 2025 (April - June)
+
+### Core Platform
+
+- π’ **Performance Optimization**
+ - Reduce content generation time by 30%
+ - Optimize memory usage for large content pieces
+ - Implement caching for frequently used research data
+
+- π‘ **Multi-language Support**
+ - Add support for Spanish, French, and German content generation
+ - Implement language-specific research capabilities
+ - Create language-specific SEO optimization
+
+- π‘ **User Interface Refresh**
+ - Redesign main dashboard for improved usability
+ - Implement dark mode
+ - Add customizable workspace layouts
+
+### AI Writers
+
+- π’ **Enhanced Blog Writer**
+ - Add support for more blog formats (listicles, how-to guides, etc.)
+ - Implement advanced outline generation
+ - Add competitor content analysis
+
+- π‘ **AI Script Writer**
+ - Create specialized writer for video scripts
+ - Support multiple video formats (YouTube, TikTok, Instagram)
+ - Add scene breakdown and shot suggestions
+
+- π‘ **Technical Content Writer**
+ - Specialized writer for technical documentation
+ - Code snippet generation and formatting
+ - Technical accuracy verification
+
+### Research & SEO
+
+- π’ **Advanced Web Research**
+ - Implement multi-source research aggregation
+ - Add research depth controls
+ - Improve citation and source tracking
+
+- π‘ **Semantic SEO Tools**
+ - Entity-based content optimization
+ - Topic cluster mapping
+ - Natural language query optimization
+
+- π΅ **Competitive Analysis Tools**
+ - Analyze top-ranking content for target keywords
+ - Identify content gaps and opportunities
+ - Generate differentiation strategies
+
+## ποΈ Q3 2025 (July - September)
+
+### Core Platform
+
+- π‘ **Collaboration Features**
+ - Multi-user editing capabilities
+ - Role-based access control
+ - Comment and feedback system
+
+- π‘ **Content Versioning**
+ - Track content revisions
+ - Compare different versions
+ - Restore previous versions
+
+- π΅ **Analytics Dashboard**
+ - Content performance tracking
+ - Usage statistics and insights
+ - AI model performance metrics
+
+### AI Writers
+
+- π‘ **E-commerce Content Suite**
+ - Enhanced product description generator
+ - Category page content creator
+ - Product comparison generator
+
+- π‘ **AI Newsletter Writer**
+ - Email newsletter templates
+ - Subscriber segmentation support
+ - A/B testing headline generator
+
+- π΅ **Interactive Content Generator**
+ - Quiz and poll creator
+ - Interactive calculator generator
+ - Decision tree content builder
+
+### Research & SEO
+
+- π‘ **AI-Powered Content Audit**
+ - Analyze existing content
+ - Identify improvement opportunities
+ - Generate update recommendations
+
+- π‘ **Local SEO Tools**
+ - Location-based content optimization
+ - Local business schema generator
+ - Regional keyword research
+
+- π΅ **Content Distribution Planner**
+ - Channel-specific content adaptation
+ - Publishing schedule optimizer
+ - Cross-platform content strategy
+
+## ποΈ Q4 2025 (October - December)
+
+### Core Platform
+
+- π‘ **API Expansion**
+ - Comprehensive REST API
+ - Webhook integrations
+ - Developer documentation and SDKs
+
+- π‘ **Enterprise Features**
+ - SSO integration
+ - Advanced security controls
+ - Custom branding options
+
+- π΅ **AI Workflow Automation**
+ - Custom workflow builder
+ - Scheduled content generation
+ - Conditional content processing
+
+### AI Writers
+
+- π‘ **AI Book Writer**
+ - Long-form content organization
+ - Chapter planning and generation
+ - Book formatting and structure
+
+- π‘ **AI Course Creator**
+ - Educational content generator
+ - Lesson plan development
+ - Quiz and assessment creator
+
+- π΅ **Multimedia Content Generator**
+ - Integrated image generation
+ - Infographic creator
+ - Audio content generator
+
+### Research & SEO
+
+- π‘ **AI Research Assistant**
+ - Conversational research interface
+ - Deep research capabilities
+ - Research summarization and extraction
+
+- π‘ **International SEO Tools**
+ - Multi-language keyword research
+ - International content optimization
+ - Hreflang tag generator
+
+- π΅ **Predictive Content Performance**
+ - AI-powered performance prediction
+ - Content improvement recommendations
+ - Trend analysis and forecasting
+
+## ποΈ 2026 and Beyond
+
+### Core Platform
+
+- π΅ **NextJS React Application**
+ - Complete frontend rebuild
+ - Enhanced performance and responsiveness
+ - Progressive web app capabilities
+
+- π΅ **AI Agent Ecosystem**
+ - Specialized AI agents for different tasks
+ - Agent collaboration framework
+ - Custom agent creation
+
+- π΅ **Advanced Personalization**
+ - User behavior-based recommendations
+ - Personalized content generation
+ - Learning from user preferences
+
+### AI Writers
+
+- π΅ **Multimodal Content Creation**
+ - Integrated text, image, and video generation
+ - Cross-format content consistency
+ - Single-prompt multi-format generation
+
+- π΅ **Industry-Specific Writers**
+ - Legal content generator
+ - Medical content writer
+ - Financial content creator
+
+- π΅ **Real-time Collaborative Writing**
+ - Multi-user simultaneous editing
+ - AI-assisted collaboration
+ - Role-based collaborative workflows
+
+### Research & SEO
+
+- π΅ **Real-time Content Optimization**
+ - Live SEO feedback during writing
+ - Instant research integration
+ - Dynamic content suggestions
+
+- π΅ **Comprehensive Analytics Suite**
+ - Advanced content performance tracking
+ - Conversion attribution
+ - ROI calculation and reporting
+
+- π΅ **AI-Driven Content Strategy**
+ - Content gap analysis
+ - Opportunity identification
+ - Automated content planning
+
+## β Recently Completed
+
+- β **Google Gemini Integration** - Added support for Google's Gemini Pro model
+- β **AI News Article Writer** - Specialized writer for news content with citation support
+- β **ChromaDB Vector Storage** - Implemented vector database for semantic search capabilities
+- β **Tavily AI Research Integration** - Added support for AI-powered web research
+- β **Streamlit UI Improvements** - Enhanced user interface with better navigation and controls
+
+## π€ Community Contributions
+
+We welcome community contributions to the AI-Writer platform! If you're interested in contributing to any of the features on our roadmap or have ideas for new features, please:
+
+1. Check our [Contributing Guidelines](CONTRIBUTING.md)
+2. Open an issue to discuss your proposed feature or improvement
+3. Submit a pull request with your implementation
+
+## π Feedback
+
+Your feedback is essential in shaping the future of AI-Writer. If you have feature requests, suggestions, or feedback on existing features, please:
+
+- Open an issue on GitHub
+- Join our [community forum](https://alwrity.com)
+- Contact us directly at info@alwrity.com
+
+---
+
+*Note: This roadmap is subject to change based on user feedback, technological developments, and strategic priorities. Last updated: April 18, 2025*
\ No newline at end of file
diff --git a/Roadmap TBDs/TWITTER_IMPLEMENTATION_PLAN.md b/Roadmap TBDs/TWITTER_IMPLEMENTATION_PLAN.md
new file mode 100644
index 00000000..e43052b4
--- /dev/null
+++ b/Roadmap TBDs/TWITTER_IMPLEMENTATION_PLAN.md
@@ -0,0 +1,4018 @@
+# Twitter Integration Implementation Plan
+
+This document outlines a comprehensive plan to implement and enhance the Twitter features in AI-Writer, transforming the "coming soon" features into fully functional components using Tweepy and user-provided API keys.
+
+## Current State Analysis
+
+The current Twitter functionality in AI-Writer includes:
+- A Twitter dashboard with multiple planned features
+- Only the "Smart Tweet Generator" is currently active
+- The tweet generator creates content but lacks real Twitter integration
+- Performance metrics are simulated rather than based on real data
+- No actual posting, scheduling, or analytics capabilities
+
+## Implementation Strategy
+
+We'll implement the Twitter features in multiple phases, focusing on delivering value incrementally while building toward a comprehensive Twitter management solution.
+
+### Phase 1: Twitter Authentication & Basic Integration
+
+#### 1.1 Twitter API Authentication System
+
+Create a secure system for users to connect their Twitter accounts:
+
+```python
+# lib/integrations/twitter/auth.py
+
+import tweepy
+import streamlit as st
+from typing import Dict, Optional, Any
+import os
+import json
+from pathlib import Path
+import time
+from loguru import logger
+
+# Configuration paths
+CONFIG_DIR = Path(__file__).parent.parent.parent.parent / 'config' / 'twitter'
+CONFIG_DIR.mkdir(exist_ok=True, parents=True)
+
+def get_twitter_auth_url() -> str:
+ """Generate Twitter OAuth URL for user authentication."""
+ try:
+ # Get API keys from session state or environment
+ consumer_key = st.session_state.get('twitter_consumer_key', os.getenv('TWITTER_CONSUMER_KEY', ''))
+ consumer_secret = st.session_state.get('twitter_consumer_secret', os.getenv('TWITTER_CONSUMER_SECRET', ''))
+
+ if not consumer_key or not consumer_secret:
+ logger.error("Twitter API keys not found")
+ return ""
+
+ # Initialize OAuth handler
+ oauth1_user_handler = tweepy.OAuth1UserHandler(
+ consumer_key, consumer_secret,
+ callback="http://localhost:8501/twitter/callback"
+ )
+
+ # Get authorization URL
+ auth_url = oauth1_user_handler.get_authorization_url()
+
+ # Store OAuth handler in session state
+ st.session_state.twitter_oauth_handler = oauth1_user_handler
+
+ return auth_url
+ except Exception as e:
+ logger.error(f"Error generating Twitter auth URL: {str(e)}")
+ return ""
+
+def handle_twitter_callback(oauth_verifier: str) -> bool:
+ """Handle Twitter OAuth callback."""
+ try:
+ # Get OAuth handler from session state
+ oauth_handler = st.session_state.get('twitter_oauth_handler')
+ if not oauth_handler:
+ logger.error("OAuth handler not found in session state")
+ return False
+
+ # Get access tokens
+ access_token, access_token_secret = oauth_handler.get_access_token(oauth_verifier)
+
+ # Store tokens in session state
+ st.session_state.twitter_access_token = access_token
+ st.session_state.twitter_access_token_secret = access_token_secret
+
+ # Create API client
+ api = create_twitter_api()
+
+ # Get user info
+ user = api.verify_credentials()
+
+ # Store user info
+ st.session_state.twitter_user = {
+ 'id': user.id,
+ 'screen_name': user.screen_name,
+ 'name': user.name,
+ 'profile_image_url': user.profile_image_url,
+ 'followers_count': user.followers_count,
+ 'friends_count': user.friends_count
+ }
+
+ # Save credentials to file
+ save_twitter_credentials(
+ user.id,
+ access_token,
+ access_token_secret
+ )
+
+ return True
+ except Exception as e:
+ logger.error(f"Error handling Twitter callback: {str(e)}")
+ return False
+
+def create_twitter_api() -> tweepy.API:
+ """Create Twitter API client."""
+ try:
+ # Get API keys and tokens
+ consumer_key = st.session_state.get('twitter_consumer_key', os.getenv('TWITTER_CONSUMER_KEY', ''))
+ consumer_secret = st.session_state.get('twitter_consumer_secret', os.getenv('TWITTER_CONSUMER_SECRET', ''))
+ access_token = st.session_state.get('twitter_access_token', '')
+ access_token_secret = st.session_state.get('twitter_access_token_secret', '')
+
+ # Create auth handler
+ auth = tweepy.OAuth1UserHandler(
+ consumer_key, consumer_secret,
+ access_token, access_token_secret
+ )
+
+ # Create API client
+ api = tweepy.API(auth)
+
+ return api
+ except Exception as e:
+ logger.error(f"Error creating Twitter API client: {str(e)}")
+ raise
+
+def create_twitter_client() -> tweepy.Client:
+ """Create Twitter API v2 client."""
+ try:
+ # Get API keys and tokens
+ consumer_key = st.session_state.get('twitter_consumer_key', os.getenv('TWITTER_CONSUMER_KEY', ''))
+ consumer_secret = st.session_state.get('twitter_consumer_secret', os.getenv('TWITTER_CONSUMER_SECRET', ''))
+ access_token = st.session_state.get('twitter_access_token', '')
+ access_token_secret = st.session_state.get('twitter_access_token_secret', '')
+ bearer_token = st.session_state.get('twitter_bearer_token', os.getenv('TWITTER_BEARER_TOKEN', ''))
+
+ # Create client
+ client = tweepy.Client(
+ bearer_token=bearer_token,
+ consumer_key=consumer_key,
+ consumer_secret=consumer_secret,
+ access_token=access_token,
+ access_token_secret=access_token_secret
+ )
+
+ return client
+ except Exception as e:
+ logger.error(f"Error creating Twitter client: {str(e)}")
+ raise
+
+def save_twitter_credentials(user_id: int, access_token: str, access_token_secret: str) -> bool:
+ """Save Twitter credentials to file."""
+ try:
+ # Create user-specific credentials file
+ creds_file = CONFIG_DIR / f"{user_id}.json"
+
+ # Save credentials
+ with open(creds_file, 'w') as f:
+ json.dump({
+ 'access_token': access_token,
+ 'access_token_secret': access_token_secret,
+ 'timestamp': time.time()
+ }, f)
+
+ return True
+ except Exception as e:
+ logger.error(f"Error saving Twitter credentials: {str(e)}")
+ return False
+
+def load_twitter_credentials(user_id: int) -> Dict[str, str]:
+ """Load Twitter credentials from file."""
+ try:
+ # Get user-specific credentials file
+ creds_file = CONFIG_DIR / f"{user_id}.json"
+
+ # Check if file exists
+ if not creds_file.exists():
+ logger.warning(f"No credentials found for user {user_id}")
+ return {}
+
+ # Load credentials
+ with open(creds_file, 'r') as f:
+ creds = json.load(f)
+
+ # Update session state
+ st.session_state.twitter_access_token = creds['access_token']
+ st.session_state.twitter_access_token_secret = creds['access_token_secret']
+
+ return creds
+ except Exception as e:
+ logger.error(f"Error loading Twitter credentials: {str(e)}")
+ return {}
+
+def is_authenticated() -> bool:
+ """Check if user is authenticated with Twitter."""
+ return (
+ 'twitter_access_token' in st.session_state and
+ 'twitter_access_token_secret' in st.session_state and
+ 'twitter_user' in st.session_state
+ )
+
+def logout() -> None:
+ """Log out user from Twitter."""
+ if 'twitter_access_token' in st.session_state:
+ del st.session_state.twitter_access_token
+ if 'twitter_access_token_secret' in st.session_state:
+ del st.session_state.twitter_access_token_secret
+ if 'twitter_user' in st.session_state:
+ del st.session_state.twitter_user
+ if 'twitter_oauth_handler' in st.session_state:
+ del st.session_state.twitter_oauth_handler
+```
+
+#### 1.2 Twitter API Key Management UI
+
+Create a UI for users to enter and manage their Twitter API keys:
+
+```python
+# lib/integrations/twitter/api_key_manager.py
+
+import streamlit as st
+from typing import Dict, Optional
+import os
+from loguru import logger
+
+def render_twitter_api_key_manager() -> None:
+ """Render Twitter API key management UI."""
+ st.markdown("## Twitter API Configuration")
+ st.markdown("""
+ To use Twitter integration features, you need to provide your Twitter API keys.
+ You can get these by creating a Twitter Developer account and setting up a project.
+ """)
+
+ # Create tabs for different authentication methods
+ tab1, tab2 = st.tabs(["Basic Authentication", "Advanced Settings"])
+
+ with tab1:
+ # Get existing keys from session state or environment
+ consumer_key = st.session_state.get('twitter_consumer_key', os.getenv('TWITTER_CONSUMER_KEY', ''))
+ consumer_secret = st.session_state.get('twitter_consumer_secret', os.getenv('TWITTER_CONSUMER_SECRET', ''))
+ bearer_token = st.session_state.get('twitter_bearer_token', os.getenv('TWITTER_BEARER_TOKEN', ''))
+
+ # Input fields for API keys
+ new_consumer_key = st.text_input(
+ "Consumer Key (API Key)",
+ value=consumer_key,
+ type="password",
+ help="Your Twitter API consumer key"
+ )
+
+ new_consumer_secret = st.text_input(
+ "Consumer Secret (API Secret)",
+ value=consumer_secret,
+ type="password",
+ help="Your Twitter API consumer secret"
+ )
+
+ new_bearer_token = st.text_input(
+ "Bearer Token",
+ value=bearer_token,
+ type="password",
+ help="Your Twitter API bearer token"
+ )
+
+ # Save button
+ if st.button("Save API Keys", use_container_width=True):
+ # Update session state
+ st.session_state.twitter_consumer_key = new_consumer_key
+ st.session_state.twitter_consumer_secret = new_consumer_secret
+ st.session_state.twitter_bearer_token = new_bearer_token
+
+ # Show success message
+ st.success("Twitter API keys saved successfully!")
+
+ with tab2:
+ st.markdown("### Advanced API Settings")
+
+ # Callback URL
+ st.text_input(
+ "Callback URL",
+ value="http://localhost:8501/twitter/callback",
+ disabled=True,
+ help="Use this URL in your Twitter Developer Portal"
+ )
+
+ # API usage limits
+ st.info("""
+ **Twitter API Rate Limits:**
+ - Standard API: 500,000 tweets/month
+ - Essential API: 10,000 tweets/month
+
+ Make sure your Twitter Developer account has the appropriate access level for your needs.
+ """)
+
+ # Help resources
+ st.markdown("""
+ **Need help?**
+ - [Twitter Developer Portal](https://developer.twitter.com)
+ - [API Documentation](https://developer.twitter.com/en/docs)
+ - [Rate Limits](https://developer.twitter.com/en/docs/twitter-api/rate-limits)
+ """)
+```
+
+#### 1.3 Twitter Account Connection UI
+
+Create a UI for users to connect their Twitter accounts:
+
+```python
+# lib/integrations/twitter/account_manager.py
+
+import streamlit as st
+from typing import Dict, Optional
+from .auth import get_twitter_auth_url, is_authenticated, logout, create_twitter_api
+import tweepy
+from loguru import logger
+
+def render_twitter_account_manager() -> None:
+ """Render Twitter account management UI."""
+ st.markdown("## Twitter Account")
+
+ # Check if API keys are configured
+ if not st.session_state.get('twitter_consumer_key') or not st.session_state.get('twitter_consumer_secret'):
+ st.warning("Please configure your Twitter API keys first.")
+ return
+
+ # Check if user is authenticated
+ if is_authenticated():
+ # Get user info
+ user = st.session_state.twitter_user
+
+ # Display user info
+ col1, col2, col3 = st.columns([1, 3, 1])
+
+ with col1:
+ st.image(user['profile_image_url'], width=80)
+
+ with col2:
+ st.markdown(f"**{user['name']}** (@{user['screen_name']})")
+ st.markdown(f"Followers: {user['followers_count']} | Following: {user['friends_count']}")
+
+ with col3:
+ if st.button("Disconnect", key="disconnect_twitter"):
+ logout()
+ st.experimental_rerun()
+
+ # Account status
+ st.success("β Your Twitter account is connected and ready to use.")
+
+ # Account actions
+ st.markdown("### Quick Actions")
+ col1, col2 = st.columns(2)
+
+ with col1:
+ if st.button("View Profile", use_container_width=True):
+ try:
+ # Get API client
+ api = create_twitter_api()
+
+ # Get user timeline
+ tweets = api.user_timeline(count=5)
+
+ # Display tweets
+ st.markdown("#### Recent Tweets")
+ for tweet in tweets:
+ st.markdown(f"""
+
+
{tweet.text}
+ Posted on {tweet.created_at.strftime('%Y-%m-%d %H:%M')}
+
+ """, unsafe_allow_html=True)
+ except Exception as e:
+ logger.error(f"Error fetching tweets: {str(e)}")
+ st.error(f"Error fetching tweets: {str(e)}")
+
+ with col2:
+ if st.button("Check API Limits", use_container_width=True):
+ try:
+ # Get API client
+ api = create_twitter_api()
+
+ # Get rate limit status
+ limits = api.rate_limit_status()
+
+ # Display limits
+ st.markdown("#### API Rate Limits")
+
+ # Timeline limits
+ timeline_limit = limits['resources']['statuses']['/statuses/user_timeline']
+ st.markdown(f"**Timeline:** {timeline_limit['remaining']}/{timeline_limit['limit']} requests remaining")
+
+ # Search limits
+ search_limit = limits['resources']['search']['/search/tweets']
+ st.markdown(f"**Search:** {search_limit['remaining']}/{search_limit['limit']} requests remaining")
+
+ # Tweet posting limits
+ st.markdown("**Tweet Posting:** Limited by your Twitter API access level")
+ except Exception as e:
+ logger.error(f"Error checking API limits: {str(e)}")
+ st.error(f"Error checking API limits: {str(e)}")
+ else:
+ # Get auth URL
+ auth_url = get_twitter_auth_url()
+
+ if auth_url:
+ # Display connect button
+ st.markdown("""
+
+
+ """, unsafe_allow_html=True)
+
+if __name__ == "__main__":
+ run_dashboard()
+```
+
+## Testing Plan
+
+### Unit Testing
+
+1. **Authentication Tests**
+ - Test Twitter API key validation
+ - Test OAuth flow
+ - Test token storage and retrieval
+
+2. **API Integration Tests**
+ - Test tweet posting
+ - Test timeline retrieval
+ - Test hashtag search
+ - Test media upload
+
+3. **Feature Tests**
+ - Test tweet generation
+ - Test performance prediction
+ - Test content calendar creation
+ - Test image generation
+
+### Integration Testing
+
+1. **End-to-End Flow Tests**
+ - Test complete user journey from authentication to posting
+ - Test data flow between components
+ - Test error handling and recovery
+
+2. **Cross-Feature Tests**
+ - Test integration between tweet generator and performance predictor
+ - Test integration between content calendar and tweet scheduler
+ - Test integration between analytics and content recommendations
+
+### User Acceptance Testing
+
+1. **Usability Tests**
+ - Test with real users to gather feedback
+ - Evaluate UI/UX design
+ - Measure time to complete common tasks
+
+2. **Performance Tests**
+ - Test with large datasets
+ - Measure response times
+ - Identify bottlenecks
+
+## Implementation Timeline
+
+### Phase 1: Twitter Authentication & Basic Integration (2 weeks)
+- Week 1: Set up Twitter API authentication system
+- Week 2: Create account connection UI and API key management
+
+### Phase 2: Enhanced Tweet Generator (2 weeks)
+- Week 3: Implement real Twitter data integration
+- Week 4: Develop tweet performance predictor
+
+### Phase 3: Content Strategy Tools (3 weeks)
+- Week 5: Implement content calendar generator
+- Week 6-7: Develop hashtag strategy manager
+
+### Phase 4: Visual Content Creation (2 weeks)
+- Week 8-9: Implement image generator for quotes, tweets, and infographics
+
+### Phase 5: Analytics & Optimization (2 weeks)
+- Week 10-11: Implement performance analytics dashboard
+
+### Phase 6: Integration and Dashboard Updates (1 week)
+- Week 12: Update Twitter dashboard and integrate all features
+
+## Conclusion
+
+This implementation plan provides a comprehensive approach to enhancing the Twitter features in AI-Writer. By leveraging the Tweepy library and user-provided API keys, we can transform the "coming soon" features into fully functional components that provide real value to users.
+
+The phased approach allows for incremental delivery of features, with each phase building on the previous one. This ensures that users can start benefiting from the enhancements early, while more advanced features are developed.
+
+Key benefits of this implementation:
+
+1. **Real Twitter Integration**: Users can connect their Twitter accounts and interact directly with the platform.
+2. **Data-Driven Insights**: Performance analytics and predictions based on real Twitter data.
+3. **Comprehensive Content Strategy**: Tools for planning, creating, and optimizing Twitter content.
+4. **Visual Content Creation**: Easy-to-use tools for creating engaging visual content.
+5. **Streamlined Workflow**: Integrated dashboard for managing all Twitter activities.
+
+By following this plan, AI-Writer will stand out against competitors by offering a complete Twitter content creation and management solution that leverages real data and AI to optimize performance.
\ No newline at end of file
diff --git a/lib/workspace/Dockerfile b/lib/workspace/Dockerfile
deleted file mode 100644
index 1bcbd6f7..00000000
--- a/lib/workspace/Dockerfile
+++ /dev/null
@@ -1,13 +0,0 @@
-FROM python:3.12
-
-WORKDIR /app
-
-COPY requirements.txt .
-
-RUN pip install -r requirements.txt
-
-COPY . .
-
-EXPOSE 8501
-
-CMD ["streamlit", "run", "alwrity.py"]
diff --git a/lib/workspace/docker-compose.yml b/lib/workspace/docker-compose.yml
deleted file mode 100644
index c2baee11..00000000
--- a/lib/workspace/docker-compose.yml
+++ /dev/null
@@ -1,11 +0,0 @@
-version: '3.8'
-
-services:
- app:
- image: python:3.12
- working_dir: /app
- volumes:
- - .:/app
- command: bash -c "pip install -r requirements.txt && streamlit run alwrity.py"
- ports:
- - "8501:8501"
+