kunthawat/ALwrity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Alpha Subscription Implementation Plan

Browse Source
This commit is contained in:
ajaysi
2025-09-24 16:08:24 +05:30
parent 580282baa1
commit ac307683e0
57 changed files with 0 additions and 22102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

268
GSC_INTEGRATION_README.md
View File

@@ -1,268 +0,0 @@
# Google Search Console (GSC) Integration for ALwrity
This document describes the complete Google Search Console integration implemented for ALwrity, allowing users to connect their GSC accounts and fetch real website analytics data.
## 🚀 Features
### Backend Features
- **OAuth2 Authentication**: Secure Google OAuth2 flow for GSC access
- **User Credential Management**: Encrypted storage of user OAuth tokens
- **Data Caching**: SQLite-based caching system for GSC data
- **Multi-user Support**: Each user can connect their own GSC account
- **Real-time Analytics**: Fetch live search analytics, sitemaps, and site data
- **Comprehensive Logging**: Detailed logging throughout the system
### Frontend Features
- **GSC Login Button**: Seamless OAuth connection flow
- **Status Management**: Real-time connection status display
- **Popup Authentication**: Secure OAuth flow in popup window
- **Error Handling**: Comprehensive error management and user feedback
- **Responsive UI**: Material-UI components matching existing dashboard style
## 📁 File Structure
### Backend Files
```
backend/
├── services/
│ └── gsc_service.py # Core GSC service with OAuth and data management
├── routers/
│ └── gsc_auth.py # FastAPI router for GSC endpoints
├── middleware/
│ └── auth_middleware.py # Clerk authentication middleware
├── gsc_credentials.json # Google OAuth2 client credentials
├── env_template.txt # Environment variables template
└── requirements.txt # Updated with GSC dependencies
```
### Frontend Files
```
frontend/src/
├── api/
│ └── gsc.ts # GSC API client
├── components/SEODashboard/components/
│ ├── GSCLoginButton.tsx # GSC connection UI component
│ └── GSCAuthCallback.tsx # OAuth callback handler
├── env_template.txt # Frontend environment template
└── package.json # Updated with Clerk dependencies
```
## 🔧 API Endpoints
### GSC Authentication & Management
- `GET /gsc/auth/url` - Get OAuth authorization URL
- `GET /gsc/callback` - Handle OAuth callback
- `GET /gsc/status` - Check GSC connection status
- `DELETE /gsc/disconnect` - Revoke GSC access
### GSC Data Retrieval
- `GET /gsc/sites` - Get user's GSC sites
- `POST /gsc/analytics` - Fetch search analytics data
- `GET /gsc/sitemaps/{site_url}` - Get sitemaps for a site
- `GET /gsc/health` - Health check endpoint
## 🗄️ Database Schema
### GSC Credentials Table
```sql
CREATE TABLE gsc_credentials (
user_id TEXT PRIMARY KEY,
credentials_json TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
```
### GSC Data Cache Table
```sql
CREATE TABLE gsc_data_cache (
id INTEGER PRIMARY KEY AUTOINCREMENT,
user_id TEXT NOT NULL,
site_url TEXT NOT NULL,
data_type TEXT NOT NULL,
data_json TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
expires_at TIMESTAMP NOT NULL,
FOREIGN KEY (user_id) REFERENCES gsc_credentials (user_id)
);
```
## 🔐 Authentication Flow
1. **User clicks "Connect GSC"** → Frontend requests OAuth URL
2. **Backend generates OAuth URL** → Returns Google authorization URL
3. **User authorizes in popup** → Google redirects to callback
4. **Backend handles callback** → Exchanges code for tokens
5. **Credentials stored securely** → User can now access GSC data
6. **Real data replaces mock data** → Dashboard shows live analytics
## 🛠️ Setup Instructions
### 1. Backend Setup
1. **Install Dependencies**:
```bash
cd backend
pip install -r requirements.txt
```
2. **Configure Environment**:
```bash
cp env_template.txt .env
# Edit .env with your actual values
```
3. **Google OAuth Setup**:
- Copy your Google OAuth credentials to `gsc_credentials.json`
- Ensure redirect URIs include both backend and frontend URLs
4. **Start Backend**:
```bash
python app.py
```
### 2. Frontend Setup
1. **Install Dependencies**:
```bash
cd frontend
npm install
```
2. **Configure Environment**:
```bash
cp env_template.txt .env
# Edit .env with your actual values
```
3. **Start Frontend**:
```bash
npm start
```
## 🔑 Environment Variables
### Backend (.env)
```env
# Clerk Authentication
CLERK_SECRET_KEY=your_clerk_secret_key_here
# Google Search Console
GSC_REDIRECT_URI=http://localhost:8000/gsc/callback
# Development Settings
DISABLE_AUTH=false
```
### Frontend (.env)
```env
# Clerk Authentication
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=your_clerk_publishable_key_here
# CopilotKit
REACT_APP_COPILOTKIT_API_KEY=your_copilotkit_api_key_here
```
## 📊 Data Types Retrieved
### Search Analytics
- **Clicks**: Number of clicks from search results
- **Impressions**: Number of times site appeared in search
- **CTR**: Click-through rate percentage
- **Position**: Average position in search results
### Site Information
- **Site URLs**: List of verified sites in GSC
- **Permission Levels**: User's access level for each site
### Sitemaps
- **Sitemap Paths**: URLs of submitted sitemaps
- **Submission Dates**: When sitemaps were last submitted
- **Index Status**: Which pages are indexed
## 🔒 Security Features
- **OAuth2 Security**: Google's secure authorization protocol
- **Token Encryption**: Credentials stored securely in database
- **User Isolation**: Each user's data is completely separate
- **Token Refresh**: Automatic token refresh when expired
- **Access Revocation**: Users can disconnect at any time
## 🧪 Testing
### Backend Testing
```bash
cd backend
python -m pytest test_gsc_*.py
```
### Frontend Testing
```bash
cd frontend
npm test
```
### Integration Testing
1. Start both backend and frontend servers
2. Navigate to SEO Dashboard
3. Click "Connect GSC"
4. Complete OAuth flow
5. Verify real data appears in dashboard
## 🐛 Troubleshooting
### Common Issues
1. **"Not Found" Error**:
- Check API endpoint paths match between frontend and backend
- Ensure backend server is running
2. **"Not Authenticated" Error**:
- Verify Clerk API keys are correct
- Check environment variables are loaded
3. **OAuth Popup Blocked**:
- Allow popups for localhost
- Check browser popup settings
4. **GSC Data Not Loading**:
- Verify Google OAuth credentials
- Check user has verified sites in GSC
- Review backend logs for errors
## 📈 Performance Optimizations
- **Data Caching**: GSC data cached for 1 hour to reduce API calls
- **Lazy Loading**: Components load data only when needed
- **Error Boundaries**: Graceful error handling prevents crashes
- **Connection Pooling**: Efficient database connections
## 🔄 Future Enhancements
- **Real-time Updates**: WebSocket-based live data updates
- **Advanced Analytics**: More detailed GSC metrics and insights
- **Bulk Operations**: Analyze multiple sites simultaneously
- **Export Features**: Export GSC data to CSV/Excel
- **Scheduled Reports**: Automated GSC reports via email
## 📝 Logging
The system includes comprehensive logging at all levels:
- **Backend**: Detailed logs for OAuth flow, data retrieval, and errors
- **Frontend**: Console logs for API calls and user interactions
- **Database**: Query logging for debugging data issues
## 🤝 Contributing
When contributing to the GSC integration:
1. Follow existing code patterns and style
2. Add comprehensive logging for new features
3. Include error handling for all API calls
4. Update tests for any new functionality
5. Document any new environment variables or setup steps
## 📄 License
This GSC integration is part of the ALwrity project and follows the same licensing terms.

639
ToBeMigrated/ai_writers/ai_blog_writer/ai_blog_generator.py
View File

@@ -1,639 +0,0 @@
import os
import streamlit as st
from loguru import logger
from lib.utils.voice_processing import record_voice
from lib.ai_writers.ai_blog_writer.blog_writer_styles import apply_blog_writer_styles
from lib.ai_writers.ai_blog_writer.ai_blog_generator_utils import (
CONFIG_PATH,
load_config,
get_search_params_from_config,
get_blog_characteristics_from_config,
get_blog_images_from_config,
get_llm_options_from_config,
process_input,
handle_content_generation
)
apply_blog_writer_styles()
def display_input_section():
"""Display the input section with text area, file upload, and voice recording options."""
# Main container with columns for better organization
col1, col2, col3 = st.columns([2, 1.5, 0.5])
# First column: Keywords input
with col1:
st.markdown("### 📌 Content Source")
user_input = st.text_area(
'Power your content with keywords or a website URL',
help='Provide keywords, a blog title, YouTube link, or web URL to generate targeted content.',
placeholder="Examples:\n- Keywords: AI tools, digital marketing\n- Blog Title: The Future of AI in Marketing\n- YouTube Link: https://youtube.com/...\n- Web URL: https://example.com/...",
height=150
)
# Second column: File uploader
with col2:
st.markdown("### 📁 File Upload")
uploaded_file = st.file_uploader(
"Add files to enhance your content",
type=["txt", "pdf", "docx", "jpg", "jpeg", "png", "mp3", "wav", "mp4", "mkv", "avi"],
help='Upload documents, images, or media files to incorporate additional information in your blog.'
)
# Third column: Voice input
with col3:
st.markdown("### 🎤 Voice")
audio_input = record_voice()
if audio_input:
st.success("Voice recorded!")
return user_input, uploaded_file, audio_input
def display_content_type_selection(inside_expander=False):
"""Display the content type selection section and return the selected type.
Args:
inside_expander (bool): If True, adjust heading levels for display inside an expander.
"""
# Content options in a cleaner layout
if not inside_expander:
st.markdown("### 🔧 Content Configuration")
st.markdown("#### Select Content Type")
else:
st.markdown("#### Content Type")
# Content type selection with better UI
content_type = st.radio(
"Choose the format and length of your blog content",
["Standard Blog Post", "Comprehensive Long-form", "AI Agent Team (Beta)"],
horizontal=True,
help="Standard: 800-1200 words | Long-form: 1500+ words | AI Agent: Experimental multi-perspective content"
)
# Map the friendly content type names to the original options
content_type_map = {
"Standard Blog Post": "Normal-length content",
"Comprehensive Long-form": "Long-form content",
"AI Agent Team (Beta)": "Experimental - AI Agents team"
}
return content_type, content_type_map[content_type]
def display_content_characteristics_tab():
"""Display the Content Characteristics tab and return the selected options."""
st.markdown("#### Blog Content Characteristics")
# Load default values from configuration
config_blog_chars = get_blog_characteristics_from_config()
# Blog length
blog_length = st.number_input(
"Blog Length (words)",
min_value=500,
max_value=5000,
value=int(config_blog_chars.get("blog_length", 2000)),
step=100,
help="Target word count for your blog post"
)
# Blog tone
tone_options = ["Professional", "Casual", "Formal", "Conversational", "Authoritative", "Friendly"]
default_tone = config_blog_chars.get("blog_tone", "Professional")
default_tone_index = tone_options.index(default_tone) if default_tone in tone_options else 0
blog_tone = st.selectbox(
"Blog Tone",
options=tone_options,
index=default_tone_index,
help="The overall tone and style of your blog content"
)
# Blog demographic
demographic_options = ["Professional", "General", "Technical", "Beginner", "Expert", "Student"]
default_demo = config_blog_chars.get("blog_demographic", "Professional")
default_demo_index = demographic_options.index(default_demo) if default_demo in demographic_options else 0
blog_demographic = st.selectbox(
"Target Audience",
options=demographic_options,
index=default_demo_index,
help="Who your blog content is primarily written for"
)
# Blog type
type_options = ["Informational", "How-to", "List", "Review", "Tutorial", "Opinion"]
default_type = config_blog_chars.get("blog_type", "Informational")
default_type_index = type_options.index(default_type) if default_type in type_options else 0
blog_type = st.selectbox(
"Blog Type",
options=type_options,
index=default_type_index,
help="The format and purpose of your blog content"
)
# Blog language
language_options = ["English", "Spanish", "French", "German", "Italian", "Portuguese"]
default_lang = config_blog_chars.get("blog_language", "English")
default_lang_index = language_options.index(default_lang) if default_lang in language_options else 0
blog_language = st.selectbox(
"Blog Language",
options=language_options,
index=default_lang_index,
help="The language your blog will be written in"
)
# Blog output format
format_options = ["markdown", "html", "plain text"]
default_format = config_blog_chars.get("blog_output_format", "markdown").lower()
default_format_index = format_options.index(default_format) if default_format in format_options else 0
blog_output_format = st.selectbox(
"Output Format",
options=format_options,
index=default_format_index,
help="The format in which the blog content will be generated"
)
# Show current configuration source
if os.path.exists(CONFIG_PATH):
st.success(f"✅ Using blog characteristics from configuration file")
else:
st.info("ℹ️ Using default blog characteristics (no configuration file found)")
return {
"blog_length": blog_length,
"blog_tone": blog_tone,
"blog_demographic": blog_demographic,
"blog_type": blog_type,
"blog_language": blog_language,
"blog_output_format": blog_output_format
}
def display_content_analysis_tab():
"""Display the Content & Analysis Options tab and return the selected options."""
st.markdown("#### Content & Analysis Options")
# Create two columns for better organization
col1, col2 = st.columns(2)
with col1:
st.markdown("**Content Enhancements**")
create_seo_tags = st.checkbox(
'✅ Generate SEO metadata',
value=True,
help='Create schema markup, meta tags, and social media metadata'
)
generate_social_media = st.checkbox(
'✅ Create social media posts',
value=False,
help="Generate matching social content for Facebook, Twitter, and LinkedIn"
)
add_table_of_contents = st.checkbox(
'✅ Add table of contents',
value=True,
help="Include an auto-generated table of contents at the beginning of the blog"
)
with col2:
st.markdown("**Analysis & Improvement**")
content_analysis = st.checkbox(
'✅ Perform content analysis',
value=False,
help="Include proofreading, readability score, and improvement suggestions"
)
enhance_readability = st.checkbox(
'✅ Enhance readability',
value=True,
help="Optimize sentence structure and vocabulary for better readability"
)
fact_checking = st.checkbox(
'✅ Basic fact verification',
value=False,
help="Verify key facts from multiple sources when possible"
)
st.markdown("---")
st.markdown("**Formatting Options**")
# Create two columns for formatting options
fmt_col1, fmt_col2 = st.columns(2)
with fmt_col1:
section_headings = st.checkbox(
'✅ Use section headings',
value=True,
help="Include clear section headings throughout the blog"
)
include_lists = st.checkbox(
'✅ Use bullet points and lists',
value=True,
help="Format appropriate content as bullet points or numbered lists"
)
with fmt_col2:
include_quotes = st.checkbox(
'✅ Include relevant quotes',
value=False,
help="Add expert quotes or important statements as blockquotes"
)
use_subheadings = st.checkbox(
'✅ Use subheadings',
value=True,
help="Break down sections with descriptive subheadings"
)
return {
"create_seo_tags": create_seo_tags,
"generate_social_media": generate_social_media,
"add_table_of_contents": add_table_of_contents,
"content_analysis": content_analysis,
"enhance_readability": enhance_readability,
"fact_checking": fact_checking,
"section_headings": section_headings,
"include_lists": include_lists,
"include_quotes": include_quotes,
"use_subheadings": use_subheadings
}
def display_blog_images_tab():
"""Display the Blog Images Details tab and return the selected options."""
st.markdown("#### Blog Images Settings")
# Load default values from configuration
config_images = get_blog_images_from_config()
# Image generation model selection
model_options = ["stable-diffusion", "dall-e", "midjourney", "imagen"]
default_model = config_images.get("image_model", "stable-diffusion")
default_model_index = model_options.index(default_model) if default_model in model_options else 0
image_model = st.selectbox(
"Image Generation Model",
options=model_options,
index=default_model_index,
help="AI model used to generate blog images"
)
# Number of blog images
num_images = st.number_input(
"Number of Blog Images",
min_value=0,
max_value=10,
value=config_images.get("num_images", 1),
step=1,
help="Number of images to generate for the blog"
)
# Image style
style_options = ["Realistic", "Artistic", "Cartoon", "Minimalist", "Corporate", "Vibrant"]
default_style = config_images.get("image_style", "Realistic")
default_style_index = style_options.index(default_style) if default_style in style_options else 0
image_style = st.selectbox(
"Image Style",
options=style_options,
index=default_style_index,
help="Visual style of the generated images"
)
# Additional image options
st.markdown("**Additional Image Options**")
col1, col2 = st.columns(2)
with col1:
generate_featured = st.checkbox(
'✅ Generate featured image',
value=True,
help="Create a featured header image for the blog"
)
add_captions = st.checkbox(
'✅ Add image captions',
value=True,
help="Generate descriptive captions for each image"
)
with col2:
use_alt_text = st.checkbox(
'✅ Generate alt text',
value=True,
help="Create accessibility alt text for all images"
)
optimize_images = st.checkbox(
'✅ Optimize image placement',
value=True,
help="Intelligently place images throughout the content"
)
# Show current configuration source
if os.path.exists(CONFIG_PATH):
st.success(f"✅ Using image settings from configuration file")
else:
st.info("ℹ️ Using default image settings (no configuration file found)")
return {
"image_model": image_model,
"num_images": num_images,
"image_style": image_style,
"generate_featured": generate_featured,
"add_captions": add_captions,
"use_alt_text": use_alt_text,
"optimize_placement": optimize_images
}
def display_llm_options_tab():
"""Display the LLM Options tab and return the selected options."""
st.markdown("#### Language Model Settings")
# Load default values from configuration
config_llm = get_llm_options_from_config()
# LLM provider selection
provider_options = ["google", "openai", "anthropic", "local"]
default_provider = config_llm.get("provider", "google")
default_provider_index = provider_options.index(default_provider) if default_provider in provider_options else 0
llm_provider = st.selectbox(
"AI Provider",
options=provider_options,
index=default_provider_index,
help="The AI provider to use for content generation"
)
# Model selection (dynamic based on provider)
if llm_provider == "google":
model_options = ["gemini-1.5-flash-latest", "gemini-1.5-pro-latest", "gemini-pro"]
elif llm_provider == "openai":
model_options = ["gpt-4o", "gpt-4-turbo", "gpt-3.5-turbo"]
elif llm_provider == "anthropic":
model_options = ["claude-3-opus", "claude-3-sonnet", "claude-3-haiku"]
else:
model_options = ["llama-3-70b", "mistral-large", "local-model"]
default_model = config_llm.get("model", "gemini-1.5-flash-latest")
default_model_index = 0
if default_model in model_options:
default_model_index = model_options.index(default_model)
llm_model = st.selectbox(
"AI Model",
options=model_options,
index=default_model_index,
help="The specific AI model to use for content generation"
)
# Create two columns for temperature and max tokens
col1, col2 = st.columns(2)
with col1:
# Temperature setting
temperature = st.slider(
"Temperature",
min_value=0.0,
max_value=1.0,
value=config_llm.get("temperature", 0.7),
step=0.1,
help="Controls randomness: lower values are more deterministic, higher values more creative"
)
with col2:
# Max tokens
max_tokens = st.number_input(
"Max Tokens",
min_value=1000,
max_value=32000,
value=config_llm.get("max_tokens", 4000),
step=1000,
help="Maximum length of generated content (in tokens)"
)
# Advanced LLM options
st.markdown("---")
st.markdown("**Advanced LLM Options**")
show_advanced_llm = st.checkbox("Show advanced LLM parameters", value=False)
advanced_params = {}
if show_advanced_llm:
# Top-p (nucleus sampling)
top_p = st.slider(
"Top-p (Nucleus Sampling)",
min_value=0.1,
max_value=1.0,
value=0.9,
step=0.1,
help="Controls diversity via nucleus sampling: 1.0 considers all tokens, lower values restrict to more likely tokens"
)
# Top-k
top_k = st.slider(
"Top-k",
min_value=1,
max_value=100,
value=40,
step=1,
help="Controls diversity by limiting to top k tokens: higher values allow more diversity"
)
# Presence penalty
presence_penalty = st.slider(
"Presence Penalty",
min_value=-2.0,
max_value=2.0,
value=0.0,
step=0.1,
help="Penalizes repeated tokens: positive values discourage repetition"
)
advanced_params = {
"top_p": top_p,
"top_k": top_k,
"presence_penalty": presence_penalty
}
# Show current configuration source
if os.path.exists(CONFIG_PATH):
st.success(f"✅ Using LLM settings from configuration file")
else:
st.info("ℹ️ Using default LLM settings (no configuration file found)")
return {
"provider": llm_provider,
"model": llm_model,
"temperature": temperature,
"max_tokens": max_tokens,
**advanced_params
}
def display_search_settings_tab():
"""Display the Search Settings tab and return the selected options."""
st.markdown("#### AI Search Configuration")
st.markdown("Control how the AI researches your topic")
# Load default values from configuration
config_search_params = get_search_params_from_config()
# Number of search results
max_results = st.slider(
"Maximum Results",
min_value=5,
max_value=30,
value=config_search_params.get("max_results", 10),
step=5,
help="Maximum number of search results to use for research"
)
# Search depth
search_depth = st.radio(
"Search Depth",
options=["basic", "advanced"],
index=0,
horizontal=True,
help="Basic: Faster but less comprehensive. Advanced: More thorough but slower."
)
# Include domains
include_domains = st.text_input(
"Include Domains (Optional)",
value="",
help="Comma-separated list of domains to prioritize in search (e.g., wikipedia.org,nih.gov)"
)
# Time range - use value from config
time_options = ["day", "week", "month", "year", "all"]
default_time_index = time_options.index(config_search_params.get("time_range", "year")) if config_search_params.get("time_range", "year") in time_options else 3 # Default to "year" (index 3)
time_range = st.select_slider(
"Time Range",
options=time_options,
value=time_options[default_time_index],
help="Limit search results to a specific time period"
)
# Show current configuration source
if os.path.exists(CONFIG_PATH):
st.success(f"✅ Using search defaults from configuration file")
else:
st.info("ℹ️ Using default search settings (no configuration file found)")
# Replace expander with checkbox for configuration display
show_config = st.checkbox("Show configuration details", value=False)
if show_config:
st.markdown("""
**Configuration File Location**
Search parameters are loaded from the main configuration file at:
`lib/workspace/alwrity_config/main_config.json`
You can modify this file to change the default search settings.
""")
if os.path.exists(CONFIG_PATH):
try:
with open(CONFIG_PATH, 'r') as f:
config_content = f.read()
st.code(config_content, language="json")
except:
st.warning("Could not read configuration file")
st.info("These settings control how the AI performs web research for your content. More thorough searches may take longer but produce better results.")
# Process include_domains from string to list if provided
domains_list = []
if include_domains:
domains_list = [domain.strip() for domain in include_domains.split(",") if domain.strip()]
return {
"max_results": max_results,
"search_depth": search_depth,
"time_range": time_range,
"include_domains": domains_list
}
def display_advanced_options():
"""Display all advanced options tabs and return the selected configurations."""
with st.expander("⚙️ Advanced Options for Personalization, Analysis, Images, LLM, and Search", expanded=False):
content_type, selected_content_type = display_content_type_selection(inside_expander=True)
tabs = st.tabs(["Personalization", "Analysis Options", "Blog Images Details", "LLM Options", "Search Settings"])
with tabs[0]: # Content Characteristics
blog_params = display_content_characteristics_tab()
with tabs[1]: # Combined Content & Analysis Options
content_analysis_params = display_content_analysis_tab()
with tabs[2]: # Blog Images Details
image_params = display_blog_images_tab()
with tabs[3]: # LLM Options
llm_params = display_llm_options_tab()
with tabs[4]: # Search Settings
search_params = display_search_settings_tab()
return content_type, selected_content_type, blog_params, content_analysis_params, image_params, llm_params, search_params
def blog_from_keyword():
"""Input blog keywords, research and write a factual blog with enhanced UI."""
# Get user inputs
user_input, uploaded_file, audio_input = display_input_section()
# Display advanced options and get configurations
content_type, selected_content_type, blog_params, content_analysis_params, image_params, llm_params, search_params = display_advanced_options()
# Generate button with icon and clearer purpose
st.markdown("") # Add spacing
generate_pressed = st.button("✨ Generate Blog Content", use_container_width=True)
# Processing logic
if generate_pressed:
st.empty()
if not uploaded_file and not user_input and not audio_input:
st.error("Please provide at least one input source (keywords, file, or voice recording)")
st.stop()
input_type = process_input(user_input, uploaded_file)
# Use the utility function to handle content generation
handle_content_generation(input_type, user_input, uploaded_file, search_params, blog_params, selected_content_type)
def ai_blog_writer_page():
"""Render the AI Blog Writer page with enhanced styling."""
logger.info("Rendering AI Blog Writer page")
# Apply shared blog writer styles
apply_blog_writer_styles()
# Back button with icon
if st.button("← Back to Dashboard", key="back_to_dashboard"):
logger.info("User clicked back button, returning to ai writer dashboard")
st.query_params.clear()
st.rerun()
# Enhanced header with icon
st.markdown("""
<div class="page-header">
<h1>✍️ AI Blog Writer</h1>
<p>Create engaging, SEO-optimized blog content with AI assistance. Our advanced algorithms help you generate high-quality, relevant articles for any topic or niche.</p>
</div>
""", unsafe_allow_html=True)
# Call the blog generator function with enhanced UI
logger.info("Calling blog_from_keyword function")
blog_from_keyword()
logger.info("Finished rendering AI Blog Writer page")

867
ToBeMigrated/ai_writers/ai_blog_writer/ai_blog_generator_utils.py
View File

@@ -1,867 +0,0 @@
import re
import os
import json
import asyncio
from loguru import logger
import PyPDF2
import streamlit as st
import tiktoken
import openai
from datetime import datetime
from lib.gpt_providers.text_generation.main_text_generation import llm_text_gen
# Remove the circular import
# from lib.ai_writers.ai_blog_writer.keywords_to_blog_streamlit import write_blog_from_keywords
from lib.ai_writers.speech_to_blog.main_audio_to_blog import generate_audio_blog
from lib.ai_writers.long_form_ai_writer import long_form_generator
from lib.ai_writers.web_url_ai_writer import blog_from_url
from lib.ai_writers.image_ai_writer import blog_from_image
from .blog_from_google_serp import write_blog_google_serp
from lib.blog_metadata.get_blog_metadata import blog_metadata
from lib.gpt_providers.text_to_image_generation.main_generate_image_from_prompt import generate_image
# Constants
CONFIG_PATH = os.path.join("lib", "workspace", "alwrity_config", "main_config.json")
DEFAULT_CONFIG = {
"Search Engine Parameters": {
"Geographic Location": "us",
"Search Language": "en",
"Number of Results": 10,
"Time Range": "year"
}
}
# Function to load configuration from JSON file
def load_config():
"""Load configuration from the main config JSON file."""
try:
if os.path.exists(CONFIG_PATH):
with open(CONFIG_PATH, 'r') as f:
config = json.load(f)
logger.info(f"Loaded configuration from {CONFIG_PATH}")
return config
else:
logger.warning(f"Configuration file not found at {CONFIG_PATH}, using defaults")
return DEFAULT_CONFIG
except Exception as e:
logger.error(f"Error loading configuration: {str(e)}")
return DEFAULT_CONFIG
# Function to get search parameters from config
def get_search_params_from_config():
"""Extract search parameters from the main configuration."""
config = load_config()
search_params = config.get("Search Engine Parameters", {})
# Map config values to expected parameter names
result = {
"max_results": search_params.get("Number of Results", 10),
"time_range": search_params.get("Time Range", "year").lower(),
"geo": search_params.get("Geographic Location", "us"),
"language": search_params.get("Search Language", "en")
}
# Normalize time_range to match our options
time_map = {
"day": "day",
"week": "week",
"month": "month",
"year": "year",
"anytime": "all",
"all": "all"
}
result["time_range"] = time_map.get(result["time_range"].lower(), "year")
logger.info(f"Using search parameters from config: {result}")
return result
# Function to get blog content characteristics from config
def get_blog_characteristics_from_config():
"""Extract blog content characteristics from the main configuration."""
config = load_config()
blog_characteristics = config.get("Blog Content Characteristics", {})
# Map config values to expected parameter names
result = {
"blog_length": blog_characteristics.get("Blog Length", "2000"),
"blog_tone": blog_characteristics.get("Blog Tone", "Professional"),
"blog_demographic": blog_characteristics.get("Blog Demographic", "Professional"),
"blog_type": blog_characteristics.get("Blog Type", "Informational"),
"blog_language": blog_characteristics.get("Blog Language", "English"),
"blog_output_format": blog_characteristics.get("Blog Output Format", "markdown")
}
logger.info(f"Using blog characteristics from config: {result}")
return result
# Function to get blog image details from config
def get_blog_images_from_config():
"""Extract blog image details from the main configuration."""
config = load_config()
blog_images = config.get("Blog Images Details", {})
# Map config values to expected parameter names
result = {
"image_model": blog_images.get("Image Generation Model", "stable-diffusion"),
"num_images": int(blog_images.get("Number of Blog Images", 1)),
"image_style": blog_images.get("Image Style", "Realistic")
}
logger.info(f"Using blog image details from config: {result}")
return result
# Function to get LLM options from config
def get_llm_options_from_config():
"""Extract LLM options from the main configuration."""
config = load_config()
llm_options = config.get("LLM Options", {})
# Map config values to expected parameter names
result = {
"provider": llm_options.get("GPT Provider", "google"),
"model": llm_options.get("Model", "gemini-1.5-flash-latest"),
"temperature": float(llm_options.get("Temperature", 0.7)),
"max_tokens": int(llm_options.get("Max Tokens", 4000))
}
logger.info(f"Using LLM options from config: {result}")
return result
# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
tokens = tokenizer.encode(text)
"""Yield successive n-sized chunks from text."""
i = 0
while i < len(tokens):
# Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
j = min(i + int(1.5 * n), len(tokens))
while j > i + int(0.5 * n):
# Decode the tokens and check for full stop or newline
chunk = tokenizer.decode(tokens[i:j])
if chunk.endswith(".") or chunk.endswith("\n"):
break
j -= 1
# If no end of sentence found, use n tokens as the chunk size
if j == i + int(0.5 * n):
j = min(i + n, len(tokens))
yield tokens[i:j]
i = j
def extract_chunk(document, template_prompt):
""" Chunking for large documents, exceed context window"""
prompt = template_prompt.replace('<document>', document)
try:
response = llm_text_gen(prompt)
return response
except Exception as err:
logger.error(f"Failed to get response from LLM: {err}")
raise
def blog_from_pdf(pdf_text):
"""
Load in a long PDF and extract key information.
Chunk up document and process each chunk, then combine them.
"""
template_prompt=f'''Extract key pieces of information from the given document.
When you extract a key piece of information, include the closest page number.
Ex: Extracted Information (Page number)
\n\nDocument: \"\"\"<document>\"\"\"\n\n'''
# Initialize tokenizer
tokenizer = tiktoken.get_encoding("cl100k_base")
results = []
chunks = create_chunks(pdf_text, 1000, tokenizer)
text_chunks = [tokenizer.decode(chunk) for chunk in chunks]
for chunk in text_chunks:
try:
results.append(extract_chunk(chunk, template_prompt))
except Exception as e:
logger.error(f"Error processing chunk: {e}")
# Continue with other chunks even if one fails
continue
return results
# Input validation functions
def is_youtube_link(text):
"""Check if text is a valid YouTube link."""
if text is not None:
youtube_regex = re.compile(r'(https?://)?(www\.)?(youtube|youtu|youtube-nocookie)\.(com|be)/(watch\?v=|embed/|v/|.+\?v=)?([^&=%\?]{11})')
return youtube_regex.match(text)
return False
def is_web_link(text):
"""Check if text is a valid web link."""
if text is not None:
web_regex = re.compile(r'(https?://)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)')
return web_regex.match(text)
return False
def process_input(input_text, uploaded_file):
"""
Determine the type of input provided by the user.
Args:
input_text (str): The text input from the user
uploaded_file: The file uploaded by the user
Returns:
str: The determined input type ("youtube_url", "web_url", "keywords", "PDF_file", "image_file", "audio_file", "video_file", or None)
"""
# Process text input
if input_text:
if is_youtube_link(input_text):
if input_text.startswith("https://www.youtube.com/") or input_text.startswith("http://www.youtube.com/"):
return "youtube_url"
else:
st.error("Invalid YouTube URL. Please enter a valid URL.")
return None
elif is_web_link(input_text):
return "web_url"
else:
return "keywords"
# Process file input
if uploaded_file is not None:
file_details = {"filename": uploaded_file.name, "filetype": uploaded_file.type}
st.write(file_details)
# Handle different file types
if uploaded_file.type.startswith("text/"):
content = uploaded_file.read().decode("utf-8")
st.text(content)
return "text_file"
elif uploaded_file.type == "application/pdf":
return "PDF_file"
elif uploaded_file.type in ["application/vnd.openxmlformats-officedocument.wordprocessingml.document", "application/msword"]:
st.write("Word document uploaded. Add your DOCX processing logic here.")
return "word_file"
elif uploaded_file.type.startswith("image/"):
st.image(uploaded_file)
return "image_file"
elif uploaded_file.type.startswith("audio/"):
st.audio(uploaded_file)
return "audio_file"
elif uploaded_file.type.startswith("video/"):
st.video(uploaded_file)
return "video_file"
return None
# Content processing functions
def process_keywords_input(user_input, search_params, blog_params, selected_content_type):
"""Process keywords input and generate content based on the selected options."""
if not user_input or len(user_input.split()) < 2:
st.error('Please provide at least two keywords for best results')
return False
# Check for dialog states and handle them directly
if st.session_state.get("show_title_dialog", False):
st.warning("Please use the main function to handle title refinement dialog")
# Clear the dialog state to avoid getting stuck
st.session_state.show_title_dialog = False
return False
if st.session_state.get("show_meta_dialog", False):
st.warning("Please use the main function to handle meta description refinement dialog")
# Clear the dialog state to avoid getting stuck
st.session_state.show_meta_dialog = False
return False
if st.session_state.get("show_snippet_dialog", False):
st.warning("Please use the main function to handle structured data dialog")
# Clear the dialog state to avoid getting stuck
st.session_state.show_snippet_dialog = False
return False
try:
if selected_content_type == "Normal-length content":
st.subheader("Your Generated Blog Post")
logger.info(f"Generating standard blog post with parameters: {blog_params}")
# Use a direct approach to generate blog content to avoid nested expanders
# Instead of importing write_blog_from_keywords which contains many expanders
try:
# Show simplified progress UI
progress_container = st.container()
with progress_container:
progress_bar = st.progress(0)
status_text = st.empty()
# Step 1: Initialize and show progress
status_text.info("Initializing blog generation...")
progress_bar.progress(0.1)
# Initialize parameters
from .blog_ai_research_utils import initialize_parameters
search_params, blog_params = initialize_parameters(search_params, blog_params)
# Step 2: Research phase
status_text.info("Researching your topic...")
progress_bar.progress(0.2)
# Perform research using direct function calls
from .blog_ai_research_utils import do_google_serp_search, do_tavily_ai_search
# Do Google search
status_text.info("Searching Google for relevant information...")
google_result = do_google_serp_search(user_input, max_results=search_params.get("max_results", 10))
google_success = google_result and 'results' in google_result and google_result['results']
progress_bar.progress(0.4)
# Do Tavily search if needed
tavily_result = None
tavily_success = False
if not google_success:
status_text.info("Performing additional research with Tavily...")
tavily_result, _, _ = do_tavily_ai_search(
user_input,
max_results=search_params.get("max_results", 10),
search_depth=search_params.get("search_depth", "basic")
)
tavily_success = tavily_result is not None
progress_bar.progress(0.5)
# Step 3: Generate content
status_text.info("Generating blog content...")
progress_bar.progress(0.6)
# Generate content based on search results
from .blog_from_google_serp import write_blog_google_serp
if google_success:
blog_content = write_blog_google_serp(user_input, google_result['results'], blog_params=blog_params)
elif tavily_success:
blog_content = write_blog_google_serp(user_input, tavily_result, blog_params=blog_params)
else:
status_text.error("Failed to gather research data. Please try again.")
return False
# Step 4: Generate metadata and image
status_text.info("Adding metadata and final touches...")
progress_bar.progress(0.8)
# Import functions from keywords_to_blog_streamlit
from .keywords_to_blog_streamlit import generate_audio_version
# Define a simple update_progress function for compatibility
def simple_update_progress(step, total, message):
status_text.info(message)
progress_bar.progress(step / total)
# Generate metadata and image
# Import only essential functions needed for core processing
from .ai_blog_generator_utils import generate_blog_metadata, generate_blog_image
try:
# Create a proper status object
with st.status("Generating metadata and image...", expanded=True) as status:
# Generate metadata
blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug = generate_blog_metadata(
blog_content, user_input, status)
# Generate featured image if metadata is available
generated_image_filepath = None
if blog_title and blog_meta_desc:
generated_image_filepath = generate_blog_image(
blog_title, blog_meta_desc, blog_content, status, blog_tags)
# Save blog content to file
saved_blog_to_file = None
from ...blog_postprocessing.save_blog_to_file import save_blog_to_file
if blog_title and blog_meta_desc:
saved_blog_to_file = save_blog_to_file(
blog_content, blog_title, blog_meta_desc, blog_tags,
blog_categories, generated_image_filepath)
# Create metadata dictionary with string conversions for table display
metadata = {
"blog_title": blog_title or "",
"blog_meta_desc": blog_meta_desc or "",
"blog_tags": ", ".join(blog_tags) if isinstance(blog_tags, list) else str(blog_tags or ""),
"blog_categories": ", ".join(blog_categories) if isinstance(blog_categories, list) else str(blog_categories or ""),
"blog_hashtags": blog_hashtags or "",
"blog_slug": blog_slug or ""
}
except Exception as e:
logger.error(f"Error generating metadata or image: {e}")
metadata = {
"blog_title": "Generated Blog",
"blog_meta_desc": "",
"blog_tags": "",
"blog_categories": "",
"blog_hashtags": "",
"blog_slug": ""
}
generated_image_filepath = None
saved_blog_to_file = None
# Clear progress indicators
progress_bar.empty()
status_text.empty()
# Final message
final_message = st.empty()
final_message.success("Blog generation complete!")
# Display blog content first (without using expanders)
st.markdown("## Content")
st.markdown(blog_content)
# Show file save information if available
if saved_blog_to_file:
st.success(f"✅ Blog saved to: {saved_blog_to_file}")
# Add the audio generation button
st.markdown("---")
audio_col1, audio_col2 = st.columns([1, 3])
with audio_col1:
generate_audio_button = st.button("🔊 Generate Audio Version", use_container_width=True)
with audio_col2:
if generate_audio_button:
generate_audio_version(blog_content)
# Display metadata success message
if metadata["blog_title"]:
st.success(f"✅ Generated metadata for: {metadata['blog_title']}")
# Display metadata table (without nesting expanders)
st.markdown("---")
st.subheader("🏷️ Blog SEO Metadata")
st.table({
"Metadata": ["Blog Title", "Meta Description", "Tags", "Categories", "Hashtags", "Slug"],
"Value": [
metadata["blog_title"],
metadata["blog_meta_desc"],
metadata["blog_tags"],
metadata["blog_categories"],
metadata["blog_hashtags"],
metadata["blog_slug"]
]
})
# Display image if available
if generated_image_filepath:
st.subheader("🖼️ Featured Image")
st.image(generated_image_filepath, caption=metadata["blog_title"] or "Featured Image", use_column_width=True)
# Add regenerate button
if st.button("🔄 Regenerate Image", key="regenerate_image_simplified"):
# Use the function directly to avoid any nested expanders
new_image_path = regenerate_blog_image(
metadata["blog_title"],
metadata["blog_meta_desc"],
blog_content,
metadata["blog_tags"]
)
if new_image_path:
st.success("✅ Image regenerated successfully!")
st.image(new_image_path, caption=metadata["blog_title"], use_column_width=True)
else:
st.subheader("🖼️ Featured Image")
st.info("No image was generated. Try regenerating the blog.")
# Add refinement buttons directly, without using helper functions
col1, col2 = st.columns(2)
with col1:
if st.button("🔄 Refine Blog Title", key="refine_title_simplified", use_container_width=True):
st.session_state.show_title_dialog = True
st.rerun()
with col2:
if st.button("🔄 Refine Meta Description", key="refine_meta_simplified", use_container_width=True):
st.session_state.show_meta_dialog = True
st.rerun()
# Add structured data section directly, without using helper functions
st.markdown("---")
st.markdown("### Get Structured Data")
structured_data_col1, structured_data_col2 = st.columns([3, 1])
with structured_data_col1:
st.info("Rich snippets boost visibility and click-through rates in search results.")
with structured_data_col2:
if st.button("📊 Generate Rich Snippet", key="snippet_simplified", use_container_width=True):
st.session_state.show_snippet_dialog = True
st.rerun()
# Clear the success message after a delay
import time
time.sleep(3)
final_message.empty()
return True
except Exception as inner_err:
logger.error(f"Error in simplified blog generation: {inner_err}")
st.error(f"Failed to generate blog content: {inner_err}")
return False
elif selected_content_type == "Long-form content":
logger.info(f"Generating long-form content with parameters: {blog_params}")
# Ensure all blog parameters are properly passed to long-form generator
long_form_generator(
user_input,
search_params=search_params,
blog_params=blog_params
)
# Show success message briefly then clear it
success_msg = st.empty()
success_msg.success(f"Successfully generated long-form content for: {user_input}")
# Clear the message after 3 seconds
import time
time.sleep(3)
success_msg.empty()
return True
else:
info_msg = st.empty()
info_msg.info("AI Agent Team feature is coming soon! This will provide multi-perspective content with different AI experts collaborating on your blog.")
return False
except Exception as err:
logger.error(f"An error occurred while generating content: {err}")
st.error(f"An error occurred while generating content: {err}")
return False
def process_pdf_input(uploaded_file):
"""Process a PDF file and generate content."""
# Replace expander with a container to avoid nested expanders
pdf_container = st.container()
with pdf_container:
st.subheader("Processing PDF Document")
pdf_reader = PyPDF2.PdfReader(uploaded_file)
text = ""
combined_result = ""
# Show progress with better UI
progress_text = st.empty()
progress_bar = st.progress(0)
total_pages = len(pdf_reader.pages)
for page_num, page in enumerate(pdf_reader.pages):
progress_text.text(f"Processing page {page_num+1}/{total_pages}")
text += page.extract_text()
text = text.replace("\n", " ")
text = re.sub(r"(\w)([A-Z])", r"\1 \2", text)
results = blog_from_pdf(text)
progress_percent = (page_num + 1) / total_pages
progress_bar.progress(progress_percent)
combined_result += str(results[-1])
progress_text.empty()
progress_bar.empty()
st.subheader("Generated Content from PDF")
st.markdown(combined_result)
return True
def process_youtube_or_audio(user_input):
"""Process a YouTube URL or audio file and generate content."""
if not generate_audio_blog(user_input):
return False
return True
def process_web_url(user_input):
"""Process a web URL and generate content."""
blog_from_url(user_input)
return True
def process_image_input(user_input, uploaded_file):
"""Process an image file and generate content."""
blog_from_image(user_input, uploaded_file)
return True
def handle_content_generation(input_type, user_input, uploaded_file, search_params, blog_params, selected_content_type):
"""
Handle content generation based on the input type.
Args:
input_type: The type of input ("youtube_url", "web_url", etc.)
user_input: The text input from the user
uploaded_file: The uploaded file (if any)
search_params: Search parameters
blog_params: Blog content parameters
selected_content_type: The selected content type
Returns:
bool: True if content generation was successful, False otherwise
"""
# Create a status placeholder instead of a permanent message
status_message = st.empty()
status_message.info("Crafting your blog content... Please wait.")
try:
if input_type == "keywords":
result = process_keywords_input(user_input, search_params, blog_params, selected_content_type)
# Clear the status message when done
status_message.empty()
return result
elif input_type == "youtube_url" or input_type == "audio_file":
result = process_youtube_or_audio(user_input)
status_message.empty()
return result
elif input_type == "web_url":
result = process_web_url(user_input)
status_message.empty()
return result
elif input_type == "image_file":
result = process_image_input(user_input, uploaded_file)
status_message.empty()
return result
elif input_type == "PDF_file":
result = process_pdf_input(uploaded_file)
status_message.empty()
return result
else:
status_message.empty()
st.error(f"Unsupported input type: {input_type}")
return False
except Exception as e:
status_message.empty()
st.error(f"An error occurred during content generation: {str(e)}")
return False
def generate_blog_content(search_keywords, google_search_result, tavily_search_result,
google_search_success, tavily_search_success, blog_params, status):
"""
Generate blog content using either Google or Tavily search results.
Args:
search_keywords (str): Search keywords
google_search_result: Results from Google search
tavily_search_result: Results from Tavily search
google_search_success (bool): Whether Google search was successful
tavily_search_success (bool): Whether Tavily search was successful
blog_params (dict): Blog parameters
status: Streamlit status object
Returns:
str: Generated blog content or None if generation failed
"""
# Check if both searches failed - if so, stop the process
if not google_search_success and not tavily_search_success:
st.error("⛔ Both Google SERP and Tavily AI searches failed. Unable to generate blog content.")
st.warning("Please check your API keys in the environment settings and try again.")
return None
# Try Google results first if available
if google_search_success and 'results' in google_search_result:
try:
status.update(label=f"✏️ Writing blog from Google Search results...")
# Pass blog parameters to the blog writing function
blog_style_info = f"""
Length: {blog_params.get('blog_length')} words
Tone: {blog_params.get('blog_tone')}
Target Audience: {blog_params.get('blog_demographic')}
Blog Type: {blog_params.get('blog_type')}
Language: {blog_params.get('blog_language')}
"""
status.update(label=f"✏️ Writing {blog_params.get('blog_tone')} {blog_params.get('blog_type')} blog for {blog_params.get('blog_demographic')} audience...")
blog_markdown_str = write_blog_google_serp(search_keywords, google_search_result['results'], blog_params=blog_params)
status.update(label="✅ Generated content from Google search results", state="complete")
return blog_markdown_str
except Exception as err:
status.update(label=f"❌ Failed to generate content from Google results: {str(err)}", state="error")
st.error(f"Failed to generate content from Google results: {err}")
logger.error(f"Failed to process Google search results: {err}")
# If Google failed or had no results, try Tavily
if tavily_search_success and tavily_search_result:
try:
status.update(label=f"✏️ Writing blog from Tavily search results...")
status.update(label=f"✏️ Writing {blog_params.get('blog_tone')} {blog_params.get('blog_type')} blog for {blog_params.get('blog_demographic')} audience...")
blog_markdown_str = write_blog_google_serp(search_keywords, tavily_search_result, blog_params=blog_params)
status.update(label="✅ Generated content from Tavily search results", state="complete")
return blog_markdown_str
except Exception as err:
status.update(label=f"❌ Failed to generate content from Tavily results: {str(err)}", state="error")
st.error(f"Failed to generate content from Tavily results: {err}")
logger.error(f"Failed to process Tavily search results: {err}")
# If we still don't have content, show error
st.error("⛔ Failed to generate any blog content from the research results.")
return None
def generate_blog_metadata(blog_markdown_str, search_keywords, status):
"""
Generate metadata for the blog content.
Args:
blog_markdown_str (str): Blog content
search_keywords (str): Original search keywords
status: Streamlit status object
Returns:
tuple: (blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug)
"""
status.update(label="🔍 Generating title, meta description, tags, categories, hashtags, and slug...")
try:
# Get all 6 metadata values from blog_metadata
blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug = asyncio.run(blog_metadata(blog_markdown_str))
status.update(label="✅ Generated blog metadata successfully")
return blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug
except Exception as err:
st.error(f"Failed to get blog metadata: {err}")
logger.error(f"Failed to get blog metadata: {err}")
status.update(label="❌ Failed to get blog metadata", state="error")
return None, None, None, None, None, None
def generate_blog_image(blog_title, blog_meta_desc, blog_markdown_str, status, blog_tags=None):
"""
Generate a featured image for the blog.
Args:
blog_title (str): Blog title
blog_meta_desc (str): Blog meta description
blog_markdown_str (str): Blog content
status: Streamlit status object
blog_tags (list, optional): Blog tags to use for image prompt enhancement
Returns:
str: Path to the generated image or None if generation failed
"""
try:
status.update(label="🖼️ Generating featured image for blog...")
# Create a better prompt for image generation
if blog_title and blog_meta_desc:
# If we have both title and description, use them
text_to_image = f"{blog_title}: {blog_meta_desc}"
elif blog_title:
# If we only have title, use it
text_to_image = blog_title
elif blog_meta_desc:
# If we only have description, use it
text_to_image = blog_meta_desc
else:
# Fallback to first 200 chars of content
text_to_image = blog_markdown_str[:200]
# Ensure the prompt is of reasonable length
if len(text_to_image) > 300:
text_to_image = text_to_image[:300]
# Log the prompt being used
logger.info(f"Generating image with prompt: {text_to_image}")
status.update(label=f"🖼️ Creating image with prompt: \"{text_to_image[:50]}...\"")
# Extract blog tags if available
blog_tags_list = blog_tags if isinstance(blog_tags, list) else []
# Attempt image generation with all available parameters
generated_image_filepath = generate_image(
user_prompt=text_to_image,
title=blog_title,
description=blog_meta_desc,
tags=blog_tags_list,
content=blog_markdown_str[:2000] # Limit content length to avoid too large payloads
)
# If first attempt failed, try with a simplified prompt
if not generated_image_filepath:
logger.warning("First image generation attempt failed, trying with simplified prompt")
status.update(label="⚠️ First image attempt failed, trying again with simplified prompt...")
# Create a simpler prompt
simplified_prompt = " ".join(text_to_image.split()[:10])
generated_image_filepath = generate_image(
user_prompt=simplified_prompt,
title=blog_title,
description=blog_meta_desc,
tags=blog_tags_list,
content=blog_markdown_str[:1000] # Use even shorter content for the retry
)
if generated_image_filepath:
status.update(label="✅ Successfully generated featured image")
return generated_image_filepath
else:
status.update(label="❌ Image generation failed - no image created", state="error")
return None
except Exception as err:
st.warning(f"Failed in Image generation: {err}")
logger.error(f"Failed in Image generation: {err}")
status.update(label="❌ Image generation failed - no image created", state="error")
return None
def regenerate_blog_image(blog_title, blog_meta_desc, blog_markdown_str, blog_tags=None):
"""
Regenerate a blog image on demand.
Args:
blog_title (str): Blog title
blog_meta_desc (str): Blog meta description
blog_markdown_str (str): Blog content
blog_tags (list, optional): Blog tags to use for image prompt enhancement
Returns:
str: Path to the generated image or None if generation failed
"""
with st.status("Regenerating image...", expanded=True) as status:
try:
# Use keywords from title or description
if blog_title:
keywords = " ".join(blog_title.split()[:6])
prompt = f"Blog illustration for: {keywords}"
elif blog_meta_desc:
keywords = " ".join(blog_meta_desc.split()[:6])
prompt = f"Blog illustration for: {keywords}"
else:
keywords = blog_markdown_str.split()[:50]
prompt = f"Blog illustration based on: {' '.join(keywords[:6])}"
status.update(label=f"🖼️ Generating new image with prompt: \"{prompt}\"")
# Extract any tags if available - will be passed as empty list otherwise
blog_tags_list = blog_tags if isinstance(blog_tags, list) else []
# Generate the image with all parameters
generated_image_filepath = generate_image(
user_prompt=prompt,
title=blog_title,
description=blog_meta_desc,
tags=blog_tags_list,
content=blog_markdown_str[:2000] # Limit content length to avoid too large payloads
)
if generated_image_filepath:
status.update(label="✅ Successfully generated new image", state="complete")
return generated_image_filepath
else:
status.update(label="❌ Image regeneration failed", state="error")
return None
except Exception as err:
st.error(f"Failed to regenerate image: {err}")
logger.error(f"Image regeneration error: {err}")
status.update(label="❌ Image regeneration failed", state="error")
return None

420
ToBeMigrated/ai_writers/ai_blog_writer/blog_ai_research_utils.py
View File

@@ -1,420 +0,0 @@
import sys
import os
import streamlit as st
from loguru import logger
from dotenv import load_dotenv
from pathlib import Path
import time
# Load environment variables
load_dotenv(Path('../../../.env'))
# Import necessary modules
from ...ai_web_researcher.gpt_online_researcher import (
do_google_serp_search as gpt_do_google_serp_search,
do_tavily_ai_search as gpt_do_tavily_ai_search
)
from ...ai_web_researcher.tavily_ai_search import do_tavily_ai_search as tavily_direct_search
def initialize_parameters(search_params=None, blog_params=None):
"""
Initialize and validate search and blog parameters with defaults.
Args:
search_params (dict, optional): Search parameters
blog_params (dict, optional): Blog parameters
Returns:
tuple: (search_params, blog_params) with defaults applied
"""
# Initialize search params if not provided
if search_params is None:
search_params = {}
# Initialize blog params if not provided
if blog_params is None:
blog_params = {}
# Provide default values only for missing keys
# This ensures we don't override values that were intentionally set to 0 or other falsy values
if "max_results" not in search_params:
search_params["max_results"] = 10
if "search_depth" not in search_params:
search_params["search_depth"] = "basic"
if "time_range" not in search_params:
search_params["time_range"] = "year"
if "include_domains" not in search_params:
search_params["include_domains"] = []
# Provide default values only for missing blog parameter keys
if "blog_length" not in blog_params:
blog_params["blog_length"] = 2000
if "blog_tone" not in blog_params:
blog_params["blog_tone"] = "Professional"
if "blog_demographic" not in blog_params:
blog_params["blog_demographic"] = "Professional"
if "blog_type" not in blog_params:
blog_params["blog_type"] = "Informational"
if "blog_language" not in blog_params:
blog_params["blog_language"] = "English"
if "blog_output_format" not in blog_params:
blog_params["blog_output_format"] = "markdown"
# Log the parameters for debugging
logger.info(f"Using search parameters: {search_params}")
logger.info(f"Using blog parameters: {blog_params}")
return search_params, blog_params
def perform_google_search(search_keywords, search_params, status, status_container, progress_bar):
"""
Perform Google SERP search for the given keywords.
Args:
search_keywords (str): Keywords to search for
search_params (dict): Search parameters
status: Streamlit status object
status_container: Streamlit container for status messages
progress_bar: Streamlit progress bar
Returns:
tuple: (google_search_result, g_titles, success_flag)
"""
def update_progress(message, progress=None, level="info"):
"""Helper function to update progress in Streamlit UI"""
if progress is not None:
progress_bar.progress(progress)
if level == "error":
status_container.error(f"🚫 {message}")
elif level == "warning":
status_container.warning(f"⚠️ {message}")
elif level == "success":
status_container.success(f"✅ {message}")
else:
status_container.info(f"🔄 {message}")
logger.debug(f"Progress update [{level}]: {message}")
try:
# Update the function call to include the required parameters and search_params
status.update(label=f"Starting Google SERP search for: {search_keywords}")
# Add search params to the Google SERP search
google_search_params = {
"max_results": search_params.get("max_results", 10)
}
# Include domains if provided
if search_params.get("include_domains"):
google_search_params["include_domains"] = search_params.get("include_domains")
google_search_result = do_google_serp_search(
search_keywords,
status_container=status_container,
update_progress=update_progress,
**google_search_params
)
if google_search_result and google_search_result.get('titles') and len(google_search_result.get('titles', [])) > 0:
status.update(label=f"✅ Finished with Google web for Search: {search_keywords}")
g_titles = google_search_result.get('titles', [])
return google_search_result, g_titles, True
else:
# Check if there's an error message in the result
if google_search_result and 'summary' in google_search_result and 'Error' in google_search_result['summary']:
error_msg = google_search_result['summary']
status.update(label=f"❌ Google search failed: {error_msg}", state="error")
st.error(f"Google SERP search failed: {error_msg}")
else:
status.update(label="❌ Failed to get Google SERP results. No valid data returned.", state="error")
st.error("Google SERP search failed to return valid results.")
return google_search_result, [], False
except Exception as err:
status.update(label=f"❌ Google search error: {str(err)}", state="error")
st.error(f"Google web research failed: {err}")
logger.error(f"Failed in Google web research: {err}")
return None, [], False
def perform_tavily_search(search_keywords, search_params, status):
"""
Perform Tavily AI search for the given keywords.
Args:
search_keywords (str): Keywords to search for
search_params (dict): Search parameters
status: Streamlit status object
Returns:
tuple: (tavily_search_result, success_flag)
"""
try:
status.update(label=f"🔍 Starting Tavily AI research: {search_keywords}")
# Pass the search parameters to Tavily
tavily_result_tuple = do_tavily_ai_search(
search_keywords,
max_results=search_params.get("max_results", 10),
search_depth=search_params.get("search_depth", "basic"),
include_domains=search_params.get("include_domains", []),
time_range=search_params.get("time_range", "year")
)
if tavily_result_tuple and len(tavily_result_tuple) == 3:
tavily_search_result, t_titles, t_answer = tavily_result_tuple
# If we have either titles or an answer, consider it a success
if (t_titles and len(t_titles) > 0) or (t_answer and len(t_answer) > 10):
status.update(label=f"✅ Finished Tavily AI Search on: {search_keywords}", state="complete")
return tavily_search_result, True
else:
status.update(label="❌ Tavily search returned empty results", state="error")
st.warning("Tavily search didn't find relevant information.")
return tavily_search_result, False
else:
status.update(label="❌ Tavily search returned incomplete results", state="error")
st.error("Tavily search failed to return valid results.")
return None, False
except Exception as err:
status.update(label=f"❌ Tavily search error: {str(err)}", state="error")
st.error(f"Failed in Tavily web research: {err}")
logger.error(f"Failed in Tavily web research: {err}")
return None, False
def do_google_serp_search(search_keywords, status_container=None, update_progress=None, **kwargs):
"""
Wrapper function to handle the parameter mismatch with the original function.
"""
try:
if status_container is None:
status_container = st.empty()
if update_progress is None:
def update_progress(message, progress=None, level="info"):
if level == "error":
status_container.error(message)
elif level == "warning":
status_container.warning(message)
else:
status_container.info(message)
# Create a fixed update_progress function that handles any progress type
def safe_update_progress(message, progress=None, level="info"):
try:
# Handle progress value of different types
if progress is not None:
if isinstance(progress, str):
# Try to convert string to float if it represents a number
try:
progress = float(progress)
except ValueError:
# If conversion fails, just log the message without updating progress
progress = None
# Call the original update_progress with sanitized values
update_progress(message, progress, level)
except Exception as err:
# If there's an error in the progress function, just log to console
logger.error(f"Error in progress update: {err}")
# Try one more time with just the message
try:
update_progress(message, None, level)
except:
pass
# Set default search parameters - fix the parameter to use 'max_results' not 'num_results'
search_params = {
"max_results": kwargs.get("max_results", 10),
"include_domains": kwargs.get("include_domains", []),
"search_depth": kwargs.get("search_depth", "basic")
}
# Update status to indicate we're checking API keys
status_container.info("🔑 Checking required API keys...")
# Call the original function with the required parameters
result = gpt_do_google_serp_search(search_keywords, status_container, safe_update_progress, **search_params)
return result
except Exception as e:
error_msg = str(e)
logger.error(f"Error in do_google_serp_search wrapper: {error_msg}")
# Check for common error patterns and display user-friendly messages
if "SERPER_API_KEY is missing" in error_msg:
status_container.error("🔑 Google search API key (SERPER_API_KEY) is missing. Please check your environment settings.")
st.error("Google SERP search failed: API key is missing. Using alternative methods.")
elif "Progress Value has invalid type" in error_msg:
# This is an internal error, log it but show a more user-friendly message
status_container.warning("⚠️ Internal progress tracking error. Continuing with search.")
else:
# For unknown errors, show the full error message
status_container.error(f"🚫 Google search error: {error_msg}")
st.error(f"Google SERP search failed: {error_msg}")
# Return a minimal result structure to prevent downstream errors
return {
'results': {},
'titles': [],
'summary': f"Error occurred during search: {error_msg}",
'stats': {
'organic_count': 0,
'questions_count': 0,
'related_count': 0
}
}
def do_tavily_ai_search(keywords, max_results=10, search_depth="basic", include_domains=None, time_range="year"):
"""
Wrapper function for Tavily search to handle parameter differences.
Args:
keywords (str): Keywords to search for
max_results (int): Maximum number of search results to return
search_depth (str): "basic" or "advanced" search depth
include_domains (list): List of domains to prioritize in search
time_range (str): Time range for results ("day", "week", "month", "year", "all")
"""
status_container = st.empty()
if include_domains is None:
include_domains = []
try:
# Show status message
status_container.info(f"🔍 Preparing Tavily AI search with {search_depth} depth...")
# FIXED: Ensure all parameters have correct types to prevent comparison errors
tavily_params = {
'max_results': int(max_results), # Explicitly convert to int
'search_depth': str(search_depth), # Ensure this is a string
'include_domains': include_domains,
'time_range': str(time_range)
}
# Log the parameters for debugging
logger.info(f"Tavily search parameters: {tavily_params}")
# Check for API key before making the request
tavily_api_key = os.environ.get("TAVILY_API_KEY")
if not tavily_api_key:
status_container.error("🔑 Tavily API key (TAVILY_API_KEY) is missing. Please check your environment settings.")
st.error("Tavily search failed: API key is missing. Using alternative methods.")
return None, [], "API key missing"
status_container.info(f"🔍 Searching with Tavily AI using {search_depth} depth for: {keywords}")
# Direct implementation without calling gpt_do_tavily_ai_search to avoid type issues
try:
# Call the function directly with correct parameter types
tavily_raw_results = tavily_direct_search(
keywords,
max_results=tavily_params['max_results'],
search_depth=tavily_params['search_depth'],
include_domains=tavily_params['include_domains'],
time_range=tavily_params['time_range']
)
# Extract the needed information
if isinstance(tavily_raw_results, tuple) and len(tavily_raw_results) == 3:
# If already in the right format, use it directly
return tavily_raw_results
# Process the results to extract titles and answer
t_results = tavily_raw_results
t_titles = []
t_answer = ""
# Extract titles from results if available
if isinstance(t_results, dict):
if 'results' in t_results and isinstance(t_results['results'], list):
t_titles = [r.get('title', '') for r in t_results['results']]
status_container.success(f"✅ Found {len(t_titles)} relevant articles")
if 'answer' in t_results:
t_answer = t_results['answer']
status_container.success("✅ Generated a summary answer")
return t_results, t_titles, t_answer
except ImportError:
# Fall back to the original function if direct import fails
status_container.warning("⚠️ Using fallback Tavily search method...")
logger.warning("Using fallback Tavily search method")
# FIXED: Alternative approach - wrap the call in try/except to handle type errors
try:
tavily_result = gpt_do_tavily_ai_search(keywords, **tavily_params)
# Format the result to match what the blog writer expects
if isinstance(tavily_result, tuple) and len(tavily_result) == 3:
status_container.success("✅ Tavily search completed successfully")
return tavily_result
# If not a tuple with expected values, try to extract what we need
t_results = tavily_result
# Extract titles and answer if available
t_titles = []
t_answer = ""
if isinstance(t_results, dict):
if 'results' in t_results and isinstance(t_results['results'], list):
t_titles = [r.get('title', '') for r in t_results['results']]
status_container.success(f"✅ Found {len(t_titles)} relevant articles")
if 'answer' in t_results:
t_answer = t_results['answer']
status_container.success("✅ Generated a summary answer")
return t_results, t_titles, t_answer
except TypeError as type_err:
# Handle the specific type error more gracefully
error_msg = str(type_err)
logger.error(f"Type error in Tavily search: {error_msg}")
if "'>' not supported" in error_msg:
status_container.error("🚫 Tavily search parameter type error. Trying alternative approach...")
# Try a simpler approach with minimal parameters
try:
# Call with only the keyword and fixed max_results
tavily_result = gpt_do_tavily_ai_search(keywords, max_results=10)
# Minimal processing to extract titles and answer
t_results = tavily_result
t_titles = []
t_answer = ""
if isinstance(t_results, dict):
if 'results' in t_results and isinstance(t_results['results'], list):
t_titles = [r.get('title', '') for r in t_results['results']]
if 'answer' in t_results:
t_answer = t_results['answer']
return t_results, t_titles, t_answer
except Exception as inner_err:
logger.error(f"Alternative Tavily approach also failed: {inner_err}")
raise
else:
# Re-raise other type errors
raise
except Exception as e:
error_msg = str(e)
logger.error(f"Error in do_tavily_ai_search wrapper: {error_msg}")
# Display user-friendly error message
status_container.error(f"🚫 Tavily search error: {error_msg}")
st.error(f"Tavily AI search failed: {error_msg}")
# Return empty results to prevent downstream errors
return None, [], f"Error: {error_msg}"
finally:
# Clear the status container after a delay
time.sleep(2)
status_container.empty()

199
ToBeMigrated/ai_writers/ai_blog_writer/blog_from_google_serp.py
View File

@@ -1,199 +0,0 @@
import os
import sys
import json
from pathlib import Path
from loguru import logger
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}"
)
from ...gpt_providers.text_generation.main_text_generation import llm_text_gen
def write_blog_google_serp(keywords, search_results, blog_params=None):
"""
Write a blog post using search results from Google SERP.
Args:
keywords (str): The keywords or topic for the blog
search_results (dict): Results from Google SERP search
blog_params (dict, optional): Blog content characteristics:
- blog_length: Target word count
- blog_tone: Content tone
- blog_demographic: Target audience
- blog_type: Type of blog post
- blog_language: Language for the blog
Returns:
str: The generated blog content in markdown format
"""
# If no blog parameters are provided, use defaults
if blog_params is None:
blog_params = {
"blog_length": 2000,
"blog_tone": "Professional",
"blog_demographic": "Professional",
"blog_type": "Informational",
"blog_language": "English"
}
# Ensure all parameters have default values
blog_length = blog_params.get("blog_length", 2000)
blog_tone = blog_params.get("blog_tone", "Professional")
blog_demographic = blog_params.get("blog_demographic", "Professional")
blog_type = blog_params.get("blog_type", "Informational")
blog_language = blog_params.get("blog_language", "English")
logger.info(f"Generating {blog_tone} {blog_type} blog of {blog_length} words for {blog_demographic} audience in {blog_language}")
try:
# Build a prompt based on search results
prompt_parts = [
f"You are a specialized blog writer who writes in a {blog_tone} tone for a {blog_demographic} audience. "
f"Create a {blog_type} blog post that is approximately {blog_length} words in {blog_language}.",
f"The blog should be about: {keywords}",
"Use the following search results to create an informative, accurate, and well-structured blog post:"
]
# Add organic search results
if 'organic' in search_results:
prompt_parts.append("\nSearch results:")
for i, result in enumerate(search_results['organic'][:5], 1):
title = result.get('title', 'No title')
snippet = result.get('snippet', 'No snippet')
prompt_parts.append(f"{i}. {title}: {snippet}")
# Add people also ask questions if available
if 'peopleAlsoAsk' in search_results and search_results['peopleAlsoAsk']:
prompt_parts.append("\nPeople also ask:")
for i, question in enumerate(search_results['peopleAlsoAsk'][:3], 1):
q_text = question.get('question', 'No question')
q_answer = question.get('answer', {}).get('snippet', 'No answer')
prompt_parts.append(f"{i}. Q: {q_text}\n A: {q_answer}")
# Add related searches if available
if 'relatedSearches' in search_results and search_results['relatedSearches']:
related = [item.get('query', '') for item in search_results['relatedSearches'][:5]]
if related:
prompt_parts.append("\nRelated topics to consider including:")
prompt_parts.append(", ".join(related))
# Add specific instructions based on blog_type
type_instructions = {
"Informational": "Focus on providing factual information and educating the reader about the topic.",
"How-to": "Include clear step-by-step instructions with actionable advice.",
"List": "Organize content into a numbered or bulleted list of points, tips, or examples.",
"Review": "Provide balanced analysis with pros and cons, and a clear conclusion or recommendation.",
"Tutorial": "Include detailed instructions with examples and explanations for each step.",
"Opinion": "Present a clear perspective supported by evidence, while acknowledging other viewpoints."
}
prompt_parts.append(f"\nSpecific instructions: {type_instructions.get(blog_type, '')}")
# Add formatting instructions
prompt_parts.append("""
Format the blog post in markdown with:
- A compelling title (# Title)
- An introduction that hooks the reader
- Well-structured sections with appropriate headings (## Headings)
- Bullet points or numbered lists where appropriate
- A conclusion summarizing key points
- Make sure all content is accurate, informative, and adds value to the reader.
- Include 2-3 subheadings to organize the content well.
- Be concise and to the point.
- Write in an engaging, reader-friendly style.
- Avoid using phrases like "According to the search results" or "Based on the information provided."
- Present information as direct knowledge.
""")
# Combine all prompt parts
full_prompt = "\n".join(prompt_parts)
# Generate the blog content using the prompt
response = llm_text_gen(full_prompt)
# Return the generated content
return response
except Exception as err:
logger.error(f"Error generating blog from search results: {err}")
raise
def improve_blog_intro(blog_content, blog_intro):
"""Combine the given online research and gpt blog content"""
prompt = f"""
You are a skilled content editor, tasked with creating an engaging peek into the blog post provided.
This peek should entice readers to delve into the full content.
Here's what you need to do:
1. **Replace the old blog introduction with the new one provided.**
2. **Craft a short and captivating summary of the key points and interesting takeaways from the blog.**
- Highlight what makes the blog unique and worth reading.
- This peek should be placed directly before the new introduction.
3. **Include the complete blog content, with the new introduction and the added peek.**
Do not provide explanations for your actions, simply present the edited blog content.
Blog Content: \"\"\"{blog_content}\"\"\"
Blog Introduction: \"\"\"{blog_intro}\"\"\"
"""
logger.info("Generating blog introduction from tavily answer.")
try:
response = llm_text_gen(prompt)
return response
except Exception as err:
logger.error(f"Exit: Failed to get response from LLM: {err}")
exit(1)
def blog_with_keywords(blog, keywords):
"""Combine the given online research and gpt blog content"""
prompt = f"""
You are Sarah, the Creative Content writer, writing up fresh ideas and crafts them with care.
She makes complex topics easy to understand and writes in a friendly tone that connects with everyone.
She excels at simplifying complex topics and communicates with charisma, making technical jargon come alive for her audience.
As an expert digital content writer, specializing in content optimization and SEO.
I will provide you with my 'blog content' and 'list of keywords' on the same topic.
Your task is to write an original blog, utilizing given keywords and blog content.
Your blog should be highly detailed and well formatted.
Blog content: '{blog}'
list of keywords: '{keywords}'
"""
try:
response = llm_text_gen(prompt)
return response
except Exception as err:
logger.error(f"blog_with_keywords: Failed to get response from LLM: {err}")
raise err
def blog_with_research(report, blog):
"""Combine the given online research and gpt blog content"""
prompt = f"""
As expert Creative Content writer, Your task is to update a blog post using the latest research.
Here's what you need to do:
1. **Read the outdated blog content and the new research report carefully.**
2. **Identify key insights and updates from the research report that should be incorporated into the blog post.**
3. **Rewrite sections of the blog post to reflect the new information, ensuring a smooth and natural flow.**
4. **Maintain the blog's original friendly and conversational tone throughout.**
Remember, your goal is to seamlessly blend the new information into the existing blog post, making it accurate and engaging for readers.
\n\n
Research Report: \"\"\"{report}\"\"\"
Blog Content: \"\"\"{blog}\"\"\"
"""
try:
response = llm_text_gen(prompt)
return response
except Exception as err:
logger.error(f"blog_with_research: Failed to get response from LLM: {err}")
raise err

252
ToBeMigrated/ai_writers/ai_blog_writer/blog_writer_styles.py
View File

@@ -1,252 +0,0 @@
import streamlit as st
def apply_blog_writer_styles():
st.markdown("""
<style>
/* Base UI improvements */
body, .main .block-container {
background: linear-gradient(135deg, #f0f4f8 0%, #d7e1ec 100%) !important;
min-height: 100vh;
color: #2c3e50;
font-family: 'Helvetica Neue', sans-serif;
}
/* Main layout improvements */
.main .block-container {
padding: 1rem 2rem 2rem 2rem !important;
max-width: 1200px;
margin: 0 auto;
}
/* Back button styling */
[data-testid="stButton"] > button:first-of-type {
background: #1976d2;
color: white;
border: none;
border-radius: 30px;
padding: 0.5rem 1.5rem;
font-weight: 600;
display: flex;
align-items: center;
justify-content: center;
transition: all 0.2s ease;
box-shadow: 0 2px 10px rgba(25, 118, 210, 0.2);
margin-bottom: 1.5rem;
}
[data-testid="stButton"] > button:first-of-type:hover {
background: #1565c0;
transform: translateY(-1px);
box-shadow: 0 4px 15px rgba(25, 118, 210, 0.3);
}
/* Header styling */
.blog-header, .page-header {
text-align: center;
margin-bottom: 2rem;
padding: 2rem 1.5rem;
background: linear-gradient(135deg, #ffffff 0%, #f5f7fa 100%);
border-radius: 16px;
box-shadow: 0 4px 20px rgba(0, 0, 0, 0.08);
border: 1px solid rgba(0, 0, 0, 0.05);
}
.blog-header h1, .page-header h1 {
font-size: 2.5em;
font-family: 'Helvetica Neue', sans-serif;
font-weight: 700;
color: #1976d2;
margin-bottom: 0.5rem;
}
.blog-header p, .page-header p {
font-size: 1.1em;
color: #546e7a;
max-width: 700px;
margin: 0 auto;
line-height: 1.6;
}
/* Input section styling */
.stTextArea textarea, .stTextInput input {
background: #ffffff;
border: 1px solid #e0e0e0;
border-radius: 8px;
padding: 0.75rem 1rem;
color: #2c3e50;
font-size: 1rem;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.04);
transition: all 0.2s ease;
}
.stTextArea textarea:focus, .stTextInput input:focus {
border: 1.5px solid #1976d2;
box-shadow: 0 2px 10px rgba(25, 118, 210, 0.12);
}
/* File uploader styling */
.stFileUploader > div {
background: #ffffff;
border: 2px dashed #cfd8dc;
border-radius: 10px;
padding: 1.5rem 1rem;
text-align: center;
transition: all 0.2s ease;
}
.stFileUploader > div:hover {
border-color: #1976d2;
background: rgba(25, 118, 210, 0.03);
}
/* Options expander styling */
.stExpander {
border-radius: 10px;
overflow: hidden;
margin: 1.5rem 0;
border: 1px solid #e0e0e0;
}
.stExpander > details {
background: #ffffff;
padding: 0.5rem;
}
.stExpander > details > summary {
padding: 0.75rem 1rem;
font-weight: 600;
color: #1976d2;
}
.stExpander > details > summary:hover {
color: #1565c0;
}
/* Checkbox styling */
.stCheckbox > div {
background: #ffffff;
padding: 0.75rem 1rem;
border-radius: 8px;
margin-bottom: 0.5rem;
border: 1px solid #f0f0f0;
}
.stCheckbox > div:hover {
background: rgba(25, 118, 210, 0.03);
}
.stCheckbox label {
font-weight: 500;
color: #455a64;
}
/* Radio button styling */
.stRadio > div {
background: #ffffff;
border-radius: 10px;
padding: 0.5rem;
overflow: hidden;
box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05);
border: 1px solid #f0f0f0;
}
.stRadio > div > div {
display: flex;
flex-wrap: wrap;
gap: 0.5rem;
}
.stRadio > div > div > label {
background: #f5f7fa;
padding: 0.6rem 1.2rem;
border-radius: 30px;
transition: all 0.2s ease;
text-align: center;
font-weight: 500;
color: #546e7a;
border: 1px solid #e0e0e0;
}
.stRadio > div > div > label:hover {
background: #e3f2fd;
color: #1976d2;
border-color: #bbdefb;
}
.stRadio > div > div > label[data-baseweb="radio"] input:checked + div {
background: #1976d2;
color: white;
border-color: #1976d2;
}
/* Generate button styling */
button[data-testid="baseButton-secondary"],
button[data-testid="baseButton-primary"] {
background: linear-gradient(45deg, #1976d2, #2196f3);
color: white;
border: none;
border-radius: 30px;
padding: 0.85rem 1.5rem;
font-weight: 600;
font-size: 1.1rem;
letter-spacing: 0.5px;
transition: all 0.3s ease;
box-shadow: 0 4px 15px rgba(25, 118, 210, 0.25);
text-transform: uppercase;
display: flex;
align-items: center;
justify-content: center;
gap: 0.5rem;
}
button[data-testid="baseButton-secondary"]:hover,
button[data-testid="baseButton-primary"]:hover {
background: linear-gradient(45deg, #1565c0, #1976d2);
transform: translateY(-2px);
box-shadow: 0 6px 20px rgba(25, 118, 210, 0.35);
}
/* Input labels */
.stTextArea label, .stTextInput label, .stFileUploader label {
font-weight: 600;
color: #455a64;
font-size: 1.05rem;
margin-bottom: 0.5rem;
}
/* Section headers */
.stMarkdown h3 {
color: #1976d2;
font-weight: 600;
font-size: 1.3rem;
margin: 1.5rem 0 0.75rem 0;
padding-bottom: 0.5rem;
border-bottom: 2px solid rgba(25, 118, 210, 0.1);
}
.stMarkdown h4 {
color: #455a64;
font-weight: 600;
font-size: 1.1rem;
margin: 1rem 0 0.5rem 0;
}
/* Column layout improvements */
[data-testid="column"] {
background: #ffffff;
border-radius: 12px;
padding: 1.2rem;
box-shadow: 0 2px 15px rgba(0, 0, 0, 0.04);
border: 1px solid rgba(0, 0, 0, 0.05);
margin: 0 0.5rem;
}
/* Success and error messages */
.stSuccess, .stInfo, .stError {
border-radius: 10px;
padding: 1rem;
margin: 1rem 0;
font-weight: 500;
}
</style>
""", unsafe_allow_html=True)

864
ToBeMigrated/ai_writers/ai_blog_writer/keywords_to_blog_streamlit.py
View File

@@ -1,864 +0,0 @@
import sys
import os
import asyncio
from textwrap import dedent
from pathlib import Path
from datetime import datetime
import streamlit as st
from gtts import gTTS
import base64
from dotenv import load_dotenv
import time
# Load environment variables
load_dotenv(Path('../../.env'))
# Logger setup
from loguru import logger
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}")
# Import other necessary modules
from ...ai_web_researcher.gpt_online_researcher import (
do_metaphor_ai_research, do_google_pytrends_analysis)
from .blog_from_google_serp import write_blog_google_serp, blog_with_research
from ...blog_metadata.get_blog_metadata import blog_metadata
from ...blog_postprocessing.save_blog_to_file import save_blog_to_file
from ...gpt_providers.text_to_image_generation.main_generate_image_from_prompt import generate_image
from ...ai_seo_tools.content_title_generator import generate_blog_titles
from ...ai_seo_tools.meta_desc_generator import generate_blog_metadesc
from ...ai_seo_tools.seo_structured_data import ai_structured_data
# Import search functions from the research utils module
from .blog_ai_research_utils import (
initialize_parameters,
perform_google_search,
perform_tavily_search,
do_google_serp_search,
do_tavily_ai_search
)
def save_blog_content(blog_markdown_str, blog_title, blog_meta_desc, blog_tags, blog_categories, generated_image_filepath, status, blog_hashtags=None, blog_slug=None):
"""
Save the blog content to a file.
Args:
blog_markdown_str (str): Blog content
blog_title (str): Blog title
blog_meta_desc (str): Blog meta description
blog_tags (list): Blog tags
blog_categories (list): Blog categories
generated_image_filepath (str): Path to the generated image
status: Streamlit status object
blog_hashtags (str, optional): Social media hashtags
blog_slug (str, optional): SEO-friendly URL slug
Returns:
str: Path to the saved file or None if saving failed
"""
try:
status.update(label="💾 Saving blog content to file...")
saved_blog_to_file = save_blog_to_file(blog_markdown_str, blog_title, blog_meta_desc,
blog_tags, blog_categories, generated_image_filepath)
status.update(label=f"✅ Saved the content to: {saved_blog_to_file}")
return saved_blog_to_file
except Exception as err:
st.error(f"Failed to save blog to file: {err}")
logger.error(f"Failed to save blog to file: {err}")
status.update(label="❌ Failed to save blog to file", state="error")
return None
def generate_audio_version(blog_markdown_str, status=None):
"""
Generate an audio version of the blog content.
Args:
blog_markdown_str (str): Blog content
status: Streamlit status object (optional)
Returns:
bool: True if audio generation was successful, False otherwise
"""
try:
if status:
status.update(label="🔊 Generating audio version of the blog...")
else:
st.info("🔊 Generating audio version...")
# Only generate audio for reasonable-sized blogs (to avoid errors with very large text)
if blog_markdown_str and len(blog_markdown_str) < 50000: # Max ~50KB of text
tts = gTTS(text=blog_markdown_str[:40000], lang='en', slow=False) # Use first 40K chars to be safe
tts.save("delete_me.mp3")
st.audio("delete_me.mp3")
st.download_button(
label="📥 Download Audio File",
data=open("delete_me.mp3", "rb").read(),
file_name="blog_audio.mp3",
mime="audio/mp3"
)
if status:
status.update(label="✅ Audio version generated successfully", state="complete")
else:
st.success("✅ Audio version generated successfully")
return True
else:
st.warning("Blog content too large for audio generation")
if status:
status.update(label="⚠️ Blog content too large for audio generation", state="complete")
return False
except Exception as err:
st.warning(f"Failed to generate audio version: {err}")
logger.error(f"Failed to generate audio version: {err}")
if status:
status.update(label="❌ Failed to generate audio version", state="error")
return False
# Helper functions for write_blog_from_keywords
def setup_progress_tracking():
"""Set up progress tracking elements for blog generation."""
# Create a placeholder for the final blog content
final_content_placeholder = st.empty()
# Create progress tracking
progress_placeholder = st.empty()
with progress_placeholder.container():
progress_bar = st.progress(0)
status_text = st.empty()
def update_progress(step, total_steps, message):
"""Update the progress bar and status message"""
progress_value = min(step / total_steps, 1.0)
progress_bar.progress(progress_value)
status_text.info(f"Step {step}/{total_steps}: {message}")
# When process is complete, clear the progress info
if step == total_steps:
import time
time.sleep(3) # Show the complete message for 3 seconds
progress_bar.empty()
status_text.empty()
return final_content_placeholder, progress_placeholder, progress_bar, status_text, update_progress
def perform_research_phase(search_keywords, search_params, update_progress):
"""
Perform the research phase of blog generation.
Args:
search_keywords (str): Keywords to research
search_params (dict): Search parameters
update_progress (function): Function to update progress
Returns:
tuple: Google search results, Tavily search results, success flags, and blog titles
"""
update_progress(1, 5, f"Starting web research on '{search_keywords}'")
logger.info(f"Researching and Writing Blog on keywords: {search_keywords}")
# Create a section header for the research phase
st.subheader("🔍 Web Research Progress")
# Use a collapsible expander for research details
with st.expander("Research Details", expanded=True):
example_blog_titles = []
# Create a status element for research updates
with st.status("Web research in progress...", expanded=True) as status:
status.update(label=f"📊 Performing web research on: {search_keywords}")
# Create status container and progress tracking for Google SERP
status_container = st.empty()
research_progress = st.progress(0)
# Google Search
status.update(label="🔍 Performing Google search...")
google_search_result, g_titles, google_search_success = perform_google_search(
search_keywords, search_params, status, status_container, research_progress
)
if g_titles:
example_blog_titles.append(g_titles)
status.update(label=f"✅ Google search complete - found {len(g_titles)} relevant resources")
else:
status.update(label="⚠️ Google search yielded limited results")
# Tavily Search
status.update(label="🔍 Performing Tavily AI search...")
tavily_search_result, tavily_search_success = perform_tavily_search(
search_keywords, search_params, status
)
if tavily_search_success:
status.update(label="✅ Tavily AI search complete", state="complete")
elif google_search_success:
status.update(label="⚠️ Tavily search had issues, but Google search was successful")
else:
status.update(label="❌ Both search methods encountered issues", state="error")
# Clear the progress indicators
status_container.empty()
research_progress.empty()
return google_search_result, tavily_search_result, google_search_success, tavily_search_success, example_blog_titles
def generate_content_phase(search_keywords, google_search_result, tavily_search_result,
google_search_success, tavily_search_success, blog_params, update_progress):
"""
Generate blog content from research results.
Args:
search_keywords (str): Keywords to research
google_search_result: Results from Google search
tavily_search_result: Results from Tavily search
google_search_success (bool): Whether Google search was successful
tavily_search_success (bool): Whether Tavily search was successful
blog_params (dict): Blog parameters
update_progress (function): Function to update progress
Returns:
str: Generated blog content or None if generation failed
"""
# Import content generation function here to avoid circular import
from .ai_blog_generator_utils import generate_blog_content
update_progress(2, 5, "Generating blog content from research")
# Create a section header for the content generation phase
st.subheader("✍️ Content Generation Progress")
# Use a collapsible expander for content generation details
with st.expander("Content Generation Details", expanded=True):
# Create a status element for content generation updates
with st.status("Content generation in progress...", expanded=True) as status:
if google_search_success:
source = "Google search results"
else:
source = "Tavily AI research"
status.update(label=f"📝 Creating {blog_params.get('blog_tone')} {blog_params.get('blog_type')} content for {blog_params.get('blog_demographic')} audience...")
blog_markdown_str = generate_blog_content(
search_keywords, google_search_result, tavily_search_result,
google_search_success, tavily_search_success, blog_params, status
)
if blog_markdown_str:
status.update(label=f"✅ Successfully generated ~{len(blog_markdown_str.split())} words of content using {source}", state="complete")
else:
status.update(label="❌ Content generation failed", state="error")
return blog_markdown_str
def generate_metadata_and_image(blog_markdown_str, search_keywords, blog_tags, update_progress):
"""
Generate metadata and featured image for the blog.
Args:
blog_markdown_str (str): Blog content
search_keywords (str): Keywords used for research
blog_tags (list): Blog tags
update_progress (function): Function to update progress
Returns:
tuple: Blog metadata and image filepath
"""
# Import metadata and image generation functions here to avoid circular import
from .ai_blog_generator_utils import generate_blog_metadata, generate_blog_image
update_progress(3, 5, "Generating SEO metadata and enhancements")
# Create a section header for the enhancement phase
st.subheader("🔍 SEO & Enhancement Progress")
# Use a collapsible expander for enhancement details
with st.expander("Enhancement Details", expanded=True):
blog_title = None
blog_meta_desc = None
blog_categories = None
blog_hashtags = None
blog_slug = None
generated_image_filepath = None
saved_blog_to_file = None
# Create a status element for enhancement updates
with st.status("Enhancing content...", expanded=True) as status:
# Generate metadata
status.update(label="🏷️ Generating SEO metadata (title, description, tags)...")
blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug = generate_blog_metadata(
blog_markdown_str, search_keywords, status
)
# Check if there are updated values in session state
if 'blog_title' in st.session_state:
blog_title = st.session_state.blog_title
status.update(label=f"✅ Using refined title: \"{blog_title}\"")
if 'blog_meta_desc' in st.session_state:
blog_meta_desc = st.session_state.blog_meta_desc
status.update(label=f"✅ Using refined meta description")
if blog_title and blog_meta_desc:
status.update(label=f"✅ Generated metadata: \"{blog_title}\"")
# Generate featured image
status.update(label="🖼️ Creating featured image...")
generated_image_filepath = generate_blog_image(
blog_title, blog_meta_desc, blog_markdown_str, status, blog_tags
)
# Save blog content to file
status.update(label="💾 Saving blog content...")
saved_blog_to_file = save_blog_content(
blog_markdown_str, blog_title, blog_meta_desc, blog_tags,
blog_categories, generated_image_filepath, status, blog_hashtags, blog_slug
)
status.update(label="✅ Content enhancement complete", state="complete")
else:
status.update(label="⚠️ Metadata generation had issues, using simplified format", state="warning")
# Add buttons for metadata refinement
create_metadata_refinement_ui()
# Add rich snippet section
create_structured_data_ui()
metadata = {
"blog_title": blog_title,
"blog_meta_desc": blog_meta_desc,
"blog_tags": blog_tags,
"blog_categories": blog_categories,
"blog_hashtags": blog_hashtags,
"blog_slug": blog_slug
}
return metadata, generated_image_filepath, saved_blog_to_file
def create_metadata_refinement_ui():
"""Create UI elements for refining blog metadata (title and meta description)."""
col1, col2 = st.columns(2)
with col1:
if st.button("🔄 Refine Blog Title", key="refine_title_main", use_container_width=True):
st.session_state.show_title_dialog = True
st.rerun()
with col2:
if st.button("🔄 Refine Meta Description", key="refine_meta_main", use_container_width=True):
st.session_state.show_meta_dialog = True
st.rerun()
def create_structured_data_ui():
"""Create UI elements for generating structured data."""
st.markdown("---")
structured_data_col1, structured_data_col2 = st.columns([3, 1])
with structured_data_col1:
# Educational popover explaining why rich snippets are important
with st.expander("ℹ️ Why Rich Snippets Are Important for SEO"):
st.markdown("""
### Rich Snippets: Boosting Your SEO and Click-Through Rates
**What are Rich Snippets?**
Rich snippets are enhanced search results that display additional information directly in search engine results pages (SERPs). They're created using structured data markup (JSON-LD) that helps search engines understand your content better.
**Why are they important?**
1. **Increased Visibility**: Rich snippets stand out in search results with stars, images, and additional information
2. **Higher Click-Through Rates (CTR)**: Studies show rich snippets can increase CTR by 30-150%
3. **Improved SEO**: They help search engines understand your content better, potentially improving rankings
4. **Enhanced User Experience**: Users can see key information before clicking, leading to more qualified traffic
5. **Mobile-Friendly**: Rich snippets are especially effective on mobile searches
**Common types of rich snippets include:**
- Articles/Blogs (with author, date, image)
- Products (with ratings, price, availability)
- Recipes (with cooking time, ratings, calories)
- Events (with date, location, ticket info)
- Local Business (with address, hours, ratings)
Adding structured data to your content is a powerful SEO technique that requires minimal effort but provides significant benefits.
""")
with structured_data_col2:
# Button to generate rich snippet
if st.button("📊 Generate Rich Snippet", key="snippet_main", use_container_width=True):
st.session_state.show_snippet_dialog = True
st.rerun()
def display_featured_image(blog_title, blog_meta_desc, blog_markdown_str, blog_tags, generated_image_filepath):
"""
Display the featured image with regeneration options.
Args:
blog_title (str): Blog title
blog_meta_desc (str): Blog meta description
blog_markdown_str (str): Blog content
blog_tags (list): Blog tags
generated_image_filepath (str): Path to the generated image
Returns:
str: Updated image filepath if regenerated, otherwise original filepath
"""
# Import image regeneration function here to avoid circular import
from .ai_blog_generator_utils import regenerate_blog_image
st.subheader("🖼️ Featured Image")
image_container = st.container()
# Display featured image
with image_container:
if generated_image_filepath:
st.image(generated_image_filepath, caption=blog_title or "Featured Image", use_column_width=True)
# Add regenerate button
if st.button("🔄 Regenerate Image", key="regenerate_image"):
new_image_path = regenerate_blog_image(blog_title, blog_meta_desc, blog_markdown_str, blog_tags)
if new_image_path:
return new_image_path
else:
st.info("No featured image was generated. Click below to generate one.")
if st.button("🖼️ Generate Image", key="generate_image"):
new_image_path = regenerate_blog_image(blog_title, blog_meta_desc, blog_markdown_str, blog_tags)
if new_image_path:
return new_image_path
return generated_image_filepath
def display_blog_content_and_audio(blog_markdown_str, saved_blog_to_file):
"""
Display the blog content and audio generation option.
Args:
blog_markdown_str (str): Blog content
saved_blog_to_file (str): Path to the saved blog file
"""
# Display blog content
st.markdown("## Content")
st.markdown(blog_markdown_str)
# Show file save information if available
if saved_blog_to_file:
st.success(f"✅ Blog saved to: {saved_blog_to_file}")
# Add the audio generation button
st.markdown("---")
audio_col1, audio_col2 = st.columns([1, 3])
with audio_col1:
generate_audio_button = st.button("🔊 Generate Audio Version", use_container_width=True)
with audio_col2:
if generate_audio_button:
generate_audio_version(blog_markdown_str)
def display_final_metadata_table(metadata, update_progress):
"""
Display the final metadata table and options.
Args:
metadata (dict): Blog metadata
update_progress (function): Function to update progress
"""
update_progress(4, 5, "Preparing final blog presentation")
st.markdown("---")
# Display metadata in a collapsible expander to save space
with st.expander("🏷️ Metadata", expanded=True):
st.table({
"Metadata": ["Blog Title", "Meta Description", "Tags", "Categories", "Hashtags", "Slug"],
"Value": [
metadata["blog_title"],
metadata["blog_meta_desc"],
metadata["blog_tags"],
metadata["blog_categories"],
metadata["blog_hashtags"],
metadata["blog_slug"]
]
})
# Add buttons in columns for refining metadata
create_metadata_refinement_ui()
# Add a row for structured data with a "Generate Rich Snippet" button
st.markdown("---")
st.markdown("### Get Structured Data")
# Add structured data UI
create_structured_data_ui()
# Create snippet generation dialog if button is clicked
if st.session_state.get("show_snippet_dialog", False):
display_structured_data_dialog(metadata["blog_title"], metadata["blog_tags"])
def display_structured_data_dialog(blog_title, blog_tags):
"""
Display the structured data generation dialog.
Args:
blog_title (str): Blog title
blog_tags (list): Blog tags
"""
with st.expander("Structured Data Generation Tool", expanded=True):
st.subheader("Generate Structured Data (Rich Snippets)")
# Close button at the top
if st.button("Close", key="close_structured_data"):
st.session_state.show_snippet_dialog = False
st.rerun()
# Simplified blog URL input
blog_url = st.text_input(
"Blog URL:",
placeholder="https://yourblog.com/your-article",
help="Enter the URL where this blog will be published"
)
# Auto-fill content type to "Article" since we're working with a blog
content_type = "Article"
st.info(f"Content Type: {content_type} (Auto-selected for blog content)")
# Form for additional article details
with st.form(key="structured_data_form"):
st.markdown("#### Article Details")
# Pre-fill with blog title and other metadata
article_title = st.text_input("Headline:", value=blog_title if blog_title else "")
article_author = st.text_input("Author:", value="")
article_date = st.date_input("Date Published:", value=datetime.now())
article_keywords = st.text_input("Keywords:", value=blog_tags if blog_tags else "")
submit_structured_data = st.form_submit_button("Generate JSON-LD")
if submit_structured_data:
if not blog_url:
st.error("Please enter a blog URL to generate structured data.")
else:
# Create details dictionary
details = {
"Headline": article_title,
"Author": article_author,
"Date Published": article_date,
"Keywords": article_keywords
}
# Call the imported ai_structured_data function or recreate its functionality
with st.spinner("Generating structured data..."):
# Import and use the function from the module directly
from ...ai_seo_tools.seo_structured_data import generate_json_data
# Generate the structured data
structured_data = generate_json_data(content_type, details, blog_url)
if structured_data:
st.success("✅ Structured data generated successfully!")
st.markdown("### Generated JSON-LD Code")
st.code(structured_data, language="json")
# Download button
st.download_button(
label="📥 Download JSON-LD",
data=structured_data,
file_name=f"{content_type}_structured_data.json",
mime="application/json",
)
# Implementation instructions
with st.expander("How to Implement This Code"):
st.markdown("""
### Adding this JSON-LD to your website:
1. **Copy the generated JSON-LD code** above
2. **Add it to the `<head>` section of your HTML** like this:
```html
<script type="application/ld+json">
[PASTE YOUR JSON-LD CODE HERE]
</script>
```
3. **Verify the implementation** using Google's Rich Results Test tool:
[https://search.google.com/test/rich-results](https://search.google.com/test/rich-results)
4. **Monitor your search appearance** in Google Search Console
""")
else:
st.error("Failed to generate structured data. Please check your inputs and try again.")
def display_title_refinement_dialog(blog_title, blog_tags):
"""
Display a dialog for refining the blog title.
Args:
blog_title (str): Current blog title
blog_tags (list): Blog tags for context
"""
with st.expander("Blog Title Refinement Tool", expanded=True):
st.subheader("Generate Better Blog Titles")
# Form for title generation
with st.form(key="title_generation_form"):
st.markdown("#### Title Generation Parameters")
# Pre-fill with blog tags if available
keywords = st.text_input("Target Keywords:",
value=blog_tags if blog_tags else "",
help="Enter primary keywords to target in the title")
blog_type = st.selectbox(
"Blog Type:",
["How-to Guide", "Tutorial", "List Post", "Informational", "Case Study", "Opinion Piece", "Review"],
index=0,
help="Select the type of blog you're creating"
)
search_intent = st.selectbox(
"Search Intent:",
["Informational", "Commercial", "Navigational", "Transactional"],
index=0,
help="Select the primary search intent your title should address"
)
language = st.selectbox(
"Language:",
["English", "Spanish", "French", "German", "Italian"],
index=0
)
submit_title = st.form_submit_button("Generate Title Suggestions")
if submit_title:
with st.spinner("Generating title suggestions..."):
# Import and use the function from the module
from ...ai_seo_tools.content_title_generator import generate_blog_titles
# Generate the titles
title_suggestions = generate_blog_titles(
target_keywords=keywords,
blog_type=blog_type,
search_intent=search_intent,
language=language
)
if title_suggestions:
st.success("✅ Generated title suggestions!")
# Display each title with an option to select it
st.markdown("### Select a Title or Modify")
selected_title = st.text_input(
"Selected or Modified Title:",
value=blog_title if blog_title else (title_suggestions[0] if title_suggestions else ""),
help="Select one of the suggested titles or modify it to your preference"
)
if st.button("Confirm Title"):
st.session_state.blog_title = selected_title
st.session_state.show_title_dialog = False
st.success(f"Title updated to: {selected_title}")
st.rerun()
# Display all suggestions
for i, title in enumerate(title_suggestions):
st.markdown(f"**Option {i+1}:** {title}")
else:
st.error("Failed to generate title suggestions. Please try different parameters.")
def display_meta_description_dialog(blog_meta_desc, blog_tags):
"""
Display a dialog for refining the meta description.
Args:
blog_meta_desc (str): Current meta description
blog_tags (list): Blog tags for context
"""
with st.expander("Meta Description Refinement Tool", expanded=True):
st.subheader("Generate Optimized Meta Descriptions")
# Form for meta description generation
with st.form(key="meta_desc_generation_form"):
st.markdown("#### Meta Description Parameters")
# Pre-fill with blog tags if available
keywords = st.text_input("Target Keywords:",
value=blog_tags if blog_tags else "",
help="Enter primary keywords to target in the meta description")
tone = st.selectbox(
"Tone:",
["Informative", "Engaging", "Professional", "Conversational", "Humorous", "Urgent"],
index=0,
help="Select the tone for your meta description"
)
search_intent = st.selectbox(
"Search Intent:",
["Informational", "Commercial", "Navigational", "Transactional"],
index=0,
help="Select the primary search intent your meta description should address"
)
language = st.selectbox(
"Language:",
["English", "Spanish", "French", "German", "Italian"],
index=0
)
submit_meta = st.form_submit_button("Generate Meta Description Suggestions")
if submit_meta:
with st.spinner("Generating meta description suggestions..."):
# Import and use the function from the module
from ...ai_seo_tools.meta_desc_generator import generate_blog_metadesc
# Generate the meta descriptions
meta_suggestions = generate_blog_metadesc(
target_keywords=keywords,
tone=tone,
search_intent=search_intent,
language=language
)
if meta_suggestions:
st.success("✅ Generated meta description suggestions!")
# Display each meta description with an option to select it
st.markdown("### Select a Meta Description or Modify")
selected_meta = st.text_area(
"Selected or Modified Meta Description:",
value=blog_meta_desc if blog_meta_desc else (meta_suggestions[0] if meta_suggestions else ""),
height=100,
help="Select one of the suggested meta descriptions or modify it to your preference"
)
if st.button("Confirm Meta Description"):
st.session_state.blog_meta_desc = selected_meta
st.session_state.show_meta_dialog = False
st.success(f"Meta description updated!")
st.rerun()
# Display all suggestions
for i, meta in enumerate(meta_suggestions):
st.markdown(f"**Option {i+1}:** {meta}")
else:
st.error("Failed to generate meta description suggestions. Please try different parameters.")
def write_blog_from_keywords(search_keywords, url=None, search_params=None, blog_params=None):
"""
This function will take a blog Topic to first generate sections for it
and then generate content for each section.
Args:
search_keywords (str): Keywords to research and write about
url (str, optional): Optional URL to use as a source
search_params (dict, optional): Dictionary of search parameters including:
- max_results: Maximum number of search results (default: 10)
- search_depth: "basic" or "advanced" search depth (default: "basic")
- include_domains: List of domains to prioritize in search
- time_range: Time range for results (default: "year")
blog_params (dict, optional): Dictionary of blog content characteristics including:
- blog_length: Target word count (default: 2000)
- blog_tone: Tone of the content (default: "Professional")
- blog_demographic: Target audience (default: "Professional")
- blog_type: Type of blog post (default: "Informational")
- blog_language: Language for the blog (default: "English")
- blog_output_format: Format for the blog (default: "markdown")
"""
# Check if we need to display any dialog boxes first
if st.session_state.get("show_title_dialog") and "blog_title" in st.session_state:
display_title_refinement_dialog(st.session_state.blog_title, None)
return None
if st.session_state.get("show_meta_dialog") and "blog_meta_desc" in st.session_state:
display_meta_description_dialog(st.session_state.blog_meta_desc, None)
return None
if st.session_state.get("show_snippet_dialog"):
# Get blog title and tags to pass to the dialog
blog_title = st.session_state.get("blog_title", "")
blog_tags = st.session_state.get("blog_tags", "")
display_structured_data_dialog(blog_title, blog_tags)
return None
# Initialize parameters with defaults
search_params, blog_params = initialize_parameters(search_params, blog_params)
# Set up progress tracking
final_content_placeholder, progress_placeholder, progress_bar, status_text, update_progress = setup_progress_tracking()
# STEP 1: Research phase
google_search_result, tavily_search_result, google_search_success, tavily_search_success, example_blog_titles = perform_research_phase(
search_keywords, search_params, update_progress
)
# Check if both searches failed - if so, stop the process
if not google_search_success and not tavily_search_success:
update_progress(5, 5, "Research failed")
progress_placeholder.error("⛔ Both Google SERP and Tavily AI searches failed. Unable to generate blog content.")
st.warning("Please check your API keys in the environment settings and try again.")
st.stop()
return None
# STEP 2: Content generation phase
blog_markdown_str = generate_content_phase(
search_keywords, google_search_result, tavily_search_result,
google_search_success, tavily_search_success, blog_params, update_progress
)
if not blog_markdown_str:
update_progress(5, 5, "Content generation failed")
progress_placeholder.error("⛔ Failed to generate blog content from research data.")
st.stop()
return None
# STEP 3: Metadata & enhancement phase
metadata, generated_image_filepath, saved_blog_to_file = generate_metadata_and_image(
blog_markdown_str, search_keywords, None, update_progress
)
# Display image with regeneration option
updated_image_filepath = display_featured_image(
metadata["blog_title"], metadata["blog_meta_desc"],
blog_markdown_str, metadata["blog_tags"], generated_image_filepath
)
if updated_image_filepath != generated_image_filepath:
generated_image_filepath = updated_image_filepath
st.rerun() # Refresh the page to show the new image
# Display blog content and audio option
display_blog_content_and_audio(blog_markdown_str, saved_blog_to_file)
# STEP 4: Final presentation
with final_content_placeholder.container():
display_final_metadata_table(metadata, update_progress)
# If there's a button click to generate a structured data snippet, handle it
if st.session_state.get("show_snippet_dialog", False):
display_structured_data_dialog(metadata["blog_title"], metadata["blog_tags"])
# Final progress update
update_progress(5, 5, "Blog generation complete!")
# Replace progress bar with success message
progress_placeholder.success("✅ Blog generation process completed successfully!")
return blog_markdown_str

1271
ToBeMigrated/ai_writers/ai_letter_writer/business_letters.py
View File

File diff suppressed because it is too large Load Diff

1135
ToBeMigrated/ai_writers/ai_letter_writer/cover_letters.py
View File

File diff suppressed because it is too large Load Diff

1184
ToBeMigrated/ai_writers/ai_letter_writer/formal_letters.py
View File

File diff suppressed because it is too large Load Diff

758
ToBeMigrated/ai_writers/ai_letter_writer/letter_templates.py
View File

@@ -1,758 +0,0 @@
"""
Letter Templates Module
This module provides structured templates and guidance for generating
different types and subtypes of letters.
Templates are defined as a nested dictionary containing 'structure' (list of sections)
and 'guidance' (a string) for each letter type and subtype.
"""
from typing import Dict, Any, List
# Define letter templates using a nested dictionary structure for better organization and lookup.
# The structure is {letter_type: {subtype: {template_details}}}
# 'default' subtype is used as a fallback if a specific subtype isn't found for a given type.
TEMPLATES: Dict[str, Dict[str, Dict[str, Any]]] = {
"personal": {
"congratulations": {
"structure": [
"Greeting",
"Express congratulations",
"Acknowledge the achievement",
"Share personal thoughts/memory (optional)",
"Look to the future/well wishes",
"Closing"
],
"guidance": "Be warm, sincere, and specific about the achievement. Express genuine happiness for the recipient. Keep the tone personal and friendly."
},
"thank_you": {
"structure": [
"Greeting",
"Express gratitude clearly",
"Specify what you are thankful for",
"Explain the impact or how you used it (optional)",
"Share a personal thought or memory (optional)",
"Offer reciprocation or look to the future",
"Closing"
],
"guidance": "Be specific about what you're thankful for and how it affected you. Express sincere appreciation. Personalize the message."
},
"sympathy": {
"structure": [
"Greeting",
"Express sympathy for the loss",
"Acknowledge the significance of the person/situation",
"Share a positive memory or quality (optional)",
"Offer specific support (optional)",
"Closing with comforting words"
],
"guidance": "Be gentle, compassionate, and sincere. Avoid clichés. Focus on offering genuine comfort and acknowledging the recipient's feelings."
},
"apology": {
"structure": [
"Greeting",
"Clearly state your apology",
"Acknowledge the specific mistake or action",
"Express understanding of the impact on the other person",
"Explain (briefly, without making excuses) what happened (optional)",
"Offer amends or suggest how to make things right",
"Assure it won't happen again",
"Closing"
],
"guidance": "Be sincere, take full responsibility for your actions, and focus on making things right. Avoid making excuses or blaming others."
},
"invitation": {
"structure": [
"Greeting",
"Clearly state the invitation",
"Provide full event details (What, When, Where)",
"Explain the significance or purpose (optional)",
"Mention who else might be there (optional)",
"Request RSVP (date and contact method)",
"Express anticipation",
"Closing"
],
"guidance": "Be clear and specific about the details (what, when, where, why). Make it easy for the person to respond."
},
"friendship": {
"structure": [
"Greeting",
"Express appreciation for the friendship",
"Share a recent memory or anecdote",
"Acknowledge the value of the relationship",
"Check in on them or share updates",
"Look to the future (getting together, etc.)",
"Closing"
],
"guidance": "Be warm, personal, and specific about what you value in the friendship. Share updates and show genuine interest."
},
"love": {
"structure": [
"Greeting (Terms of endearment)",
"Express depth of feelings",
"Share a cherished memory or moment",
"Describe specific qualities you love and appreciate",
"Reaffirm commitment or future hopes",
"Closing (Terms of endearment)"
],
"guidance": "Be sincere, personal, and specific about your feelings. Use sensory details and emotional language appropriate for your relationship."
},
"encouragement": {
"structure": [
"Greeting",
"Acknowledge the situation or challenge they face",
"Express belief in their abilities/strength",
"Offer specific words of encouragement or support",
"Remind them of past successes (optional)",
"Offer practical help (optional)",
"Look to the future with hope",
"Closing with support"
],
"guidance": "Be positive, supportive, and specific about the person's strengths and abilities. Offer genuine encouragement and belief in them."
},
"farewell": {
"structure": [
"Greeting",
"State the purpose (saying goodbye)",
"Express feelings about their departure (sadness, happiness for them)",
"Share a positive memory or highlight their contribution",
"Express good wishes for their future endeavors",
"Look to staying in touch (optional)",
"Closing"
],
"guidance": "Be warm, reflective, and forward-looking. Focus on positive memories and express genuine good wishes for their next steps."
},
# Default personal letter template if subtype is not found
"default": {
"structure": [
"Greeting",
"Introduction",
"Main content paragraphs",
"Closing thoughts",
"Signature"
],
"guidance": "Be personal, authentic, and appropriate for your relationship with the recipient. The tone is typically informal to semi-formal."
}
},
"formal": {
"application": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (if known)",
"Subject line (Clear and concise)",
"Salutation (Formal)",
"Introduction (State position applied for and where you saw it)",
"Body paragraphs (Highlight relevant skills and experience)",
"Closing paragraph (Reiterate interest, mention enclosed resume, call to action)",
"Complimentary close (Formal)",
"Signature (Typed name)",
"Enclosures (Mention if attaching resume/portfolio)"
],
"guidance": "Be professional, concise, and specific about your qualifications and genuine interest in the position. Tailor it to the specific job description."
},
"complaint": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state it's a complaint)",
"Salutation (Formal)",
"Introduction (State the purpose: complaint about X service/product)",
"Problem description (Provide specific details: date, time, location, product details, names if applicable)",
"Impact statement (Explain how the problem affected you)",
"Requested resolution (Clearly state what you want: refund, replacement, action)",
"Closing paragraph (Reference attached documents, state expectation for response)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, factual, and specific about the issue and your desired resolution. Maintain a respectful but firm tone. Include all relevant details."
},
"request": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state the request)",
"Salutation (Formal)",
"Introduction (State the purpose: making a request)",
"Request details (Clearly explain what you are requesting)",
"Justification (Explain why the request is necessary or beneficial)",
"Provide supporting information (optional)",
"Closing paragraph (Express gratitude for consideration, reiterate call to action)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, specific, and courteous about your request. Explain why it's important or beneficial to the recipient or organization."
},
"recommendation": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Letter of Recommendation for [Name])",
"Salutation (Formal)",
"Introduction (State your name, title, relationship to the recommendee, and for what purpose the letter is written)",
"Body paragraphs (Describe the recommendee's qualifications, skills, and achievements with specific examples)",
"Highlight relevant experiences and contributions",
"Closing recommendation (Summarize endorsement, strongly recommend the person)",
"Complimentary close (Formal)",
"Signature (Typed name and title)"
],
"guidance": "Be specific, positive, and credible. Use concrete examples and anecdotes to support your recommendation. Tailor it to the specific role/opportunity."
},
"resignation": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (Immediate supervisor/HR)",
"Subject line (Letter of Resignation - [Your Name])",
"Salutation (Formal)",
"Statement of resignation (Clearly state you are resigning)",
"Last day of employment (Specify the date)",
"Gratitude and reflection (Optional: Express thanks for the opportunity/experience)",
"Transition plan/Offer of assistance (Optional: Suggest how to ensure a smooth handover)",
"Closing paragraph (Express good wishes for the company's future)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, positive (if possible), and clear about your departure and last day. Maintain a good relationship."
},
"inquiry": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state the nature of the inquiry)",
"Salutation (Formal)",
"Introduction (State your purpose for writing - making an inquiry)",
"Inquiry details (Provide necessary context or background)",
"Specific questions (List your questions clearly, perhaps numbered)",
"Closing paragraph (Express gratitude for assistance, indicate when you need a response)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, specific, and courteous about your inquiry. Organize your questions logically for easy answering."
},
"authorization": {
"structure": [
"Sender's contact information (The grantor of authority)",
"Date",
"Recipient's contact information (The person/entity receiving the letter)",
"Subject line (Letter of Authorization)",
"Salutation (Formal)",
"Statement of authorization (Clearly state who is authorized)",
"Authorized person's details (Full name, ID if applicable)",
"Scope of authority (Precisely define what they are authorized to do)",
"Limitations (Specify any restrictions or conditions)",
"Duration of authorization (Start and end dates, if applicable)",
"Closing paragraph (State responsibility, express confidence)",
"Complimentary close (Formal)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and precise about who is authorized, what they can do, for how long, and under what conditions. This is a legal document."
},
"appeal": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (Appeals committee/relevant authority)",
"Subject line (Letter of Appeal - [Your Name] - [Subject of Appeal])",
"Salutation (Formal)",
"Introduction (State your name, the decision being appealed, and the date of the decision)",
"Grounds for appeal (Clearly state the reasons why you believe the decision is incorrect)",
"Provide supporting evidence (Reference attached documents: records, photos, etc.)",
"Explain mitigating circumstances (Optional)",
"Requested outcome (Clearly state what resolution you seek)",
"Closing paragraph (Express hope for reconsideration, gratitude for time)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be respectful, factual, and persuasive. Focus on valid grounds for appeal and provide clear, supporting evidence. Maintain a formal tone."
},
"introduction": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Introduction - [Your Name])",
"Salutation (Formal)",
"Introduction (Introduce yourself and the purpose of the letter)",
"Background information (Briefly describe your relevant background or expertise)",
"Reason for reaching out (Explain why you are introducing yourself to this specific person/entity)",
"Potential areas of collaboration or shared interest (Optional)",
"Call to action (Suggest a meeting, call, or further communication)",
"Closing paragraph (Express enthusiasm for potential connection)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, informative, and engaging. Clearly explain who you are, your expertise, and why you're reaching out to them specifically."
},
# Default formal letter template if subtype is not found
"default": {
"structure": [
"Sender's address",
"Date",
"Recipient's address",
"Subject line",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, clear, and concise. Use formal language and structure. The tone is typically formal."
}
},
"business": {
"sales": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Benefit-oriented)",
"Salutation",
"Attention-grabbing opening (Address a pain point or introduce a benefit)",
"Problem statement (Briefly describe the challenge the recipient faces)",
"Solution presentation (Introduce your product/service as the solution)",
"Benefits and features (Explain how your solution helps, focusing on benefits)",
"Social proof (Optional: Testimonials, case studies, data)",
"Call to action (Clearly state what you want them to do next)",
"Closing paragraph (Reiterate benefit, create urgency/incentive)",
"Complimentary close (Professional)",
"Signature (Typed name and title)",
"Enclosures (Optional: Brochure, pricing)"
],
"guidance": "Be persuasive, customer-focused, and clear about the value proposition. Focus on benefits, not just features. Make the call to action obvious."
},
"proposal": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Clear and descriptive)",
"Salutation",
"Introduction (State purpose: submitting a proposal)",
"Problem statement/Needs assessment (Demonstrate understanding of client's needs)",
"Proposed solution (Describe your solution in detail)",
"Implementation plan (Outline steps and timeline)",
"Costs and investment (Clearly state pricing and payment terms)",
"Benefits and ROI (Explain the value the client will receive)",
"Call to action (Suggest next steps: meeting, discussion)",
"Closing paragraph (Express enthusiasm, availability for questions)",
"Complimentary close (Professional)",
"Signature (Typed name and title)",
"Enclosures (Proposal document, appendix)"
],
"guidance": "Be clear, specific, and persuasive about your solution. Focus on the client's needs and the value you provide. Structure it logically."
},
"order": {
"structure": [
"Letterhead (Your company)",
"Date",
"Recipient's address (Supplier)",
"Subject line (Purchase Order - [PO Number])",
"Salutation",
"Introduction (Reference quote/agreement, state purpose: placing an order)",
"Order details (Item list with quantities, descriptions, unit prices, total)",
"Delivery requirements (Shipping address, requested delivery date, shipping method)",
"Payment terms (Reference agreed terms)",
"Closing paragraph (Express expectation for timely delivery)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and detailed about what you're ordering, quantities, delivery requirements, and payment terms. Include a purchase order number."
},
"quotation": {
"structure": [
"Letterhead (Your company)",
"Date",
"Recipient's address (Customer)",
"Subject line (Quotation for [Product/Service])",
"Salutation",
"Introduction (Reference inquiry, state purpose: providing a quotation)",
"Quotation details (List items/services, descriptions, unit prices, quantities, line totals)",
"Pricing breakdown (Mention taxes, discounts, fees separately)",
"Terms and conditions (Payment terms, delivery terms, warranty)",
"Validity period (State how long the quote is valid)",
"Next steps (How they can place an order)",
"Closing paragraph (Express hope to do business, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and transparent about pricing, terms, and what's included or excluded. Make it easy for the customer to understand and accept."
},
"acknowledgment": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Acknowledgment of [Received Item/Request])",
"Salutation",
"Acknowledgment statement (Clearly state what you have received or are acknowledging)",
"Details of what's being acknowledged (Reference number, date, brief description)",
"Confirm understanding (Optional: Briefly restate the request/issue to show understanding)",
"Next steps (Outline what will happen next, e.g., processing order, investigating issue)",
"Timeline (Provide an estimated timeframe if possible)",
"Closing paragraph (Express gratitude, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be prompt, clear, and specific about what you're acknowledging. Set clear expectations for next steps and timelines."
},
"collection": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Invoice [Invoice Number] - Payment Due)",
"Salutation",
"Introduction (Reference invoice number and due date)",
"Account status (Clearly state the outstanding amount)",
"Payment request (Politely request payment)",
"Payment options (Remind them how to pay)",
"Consequences of non-payment (Optional: Briefly mention late fees or further action, depending on letter stage)",
"Call to action (Request payment by a specific date)",
"Closing paragraph (Express hope for prompt payment, offer to discuss)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be firm but professional. Clearly state the amount due, due date, and payment options. The tone may vary depending on how overdue the payment is."
},
"adjustment": {
"structure": [
"Letterhead",
"Date",
"Recipient's address (Customer who made a complaint)",
"Subject line (Response to your inquiry - [Reference Number])",
"Salutation",
"Acknowledgment of complaint (Reference their communication and the issue)",
"Investigation findings (Explain the outcome of your investigation)",
"Adjustment offered (Clearly state the resolution: refund, replacement, credit, etc.)",
"Apology (Optional: Express regret for the inconvenience)",
"Preventive measures (Optional: Explain steps taken to prevent recurrence)",
"Closing paragraph (Express hope for continued business, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be responsive, empathetic, and solution-oriented. Clearly explain the adjustment and any preventive measures taken."
},
"credit": {
"structure": [
"Letterhead",
"Date",
"Recipient's address (Applicant)",
"Subject line (Credit Application Status - [Applicant Name])",
"Salutation",
"Introduction (Reference their credit application and the purpose of the letter)",
"Credit decision (Clearly state if credit is approved or denied)",
"If approved: Credit terms (Credit limit, payment terms, interest rates)",
"If denied: Reason for decision (Provide specific, compliant reasons)",
"Requirements (If approved: any further steps or documents needed)",
"Closing paragraph (If approved: Express welcome; If denied: Offer alternative options or appeals process)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and transparent about the credit decision, terms, limits, or reasons for denial. Ensure compliance with regulations if denying credit."
},
"follow_up": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Following up on [Previous Communication/Meeting])",
"Salutation",
"Reference to previous communication (Mention date, topic, or meeting)",
"Purpose of follow-up (Clearly state why you are writing again)",
"Action items/Next steps (Remind of agreed-upon actions or propose next steps)",
"Provide additional information (Optional)",
"Call to action (If applicable, e.g., request a response, schedule a meeting)",
"Closing paragraph (Reiterate interest, express anticipation)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and action-oriented. Reference previous communication and clearly state the purpose of your follow-up and desired outcome."
},
# Default business letter template if subtype is not found
"default": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, clear, and concise. Focus on the business purpose of your letter. The tone is typically formal to semi-formal."
}
},
"cover": {
"standard": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information (if known)",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation (Formal)",
"Introduction (State the position you are applying for, where you saw the advertisement, and a brief statement of enthusiasm)",
"Body paragraph 1 (Highlight skills and experience directly relevant to the job description - often 1-2 key qualifications)",
"Body paragraph 2 (Provide a specific example or anecdote demonstrating your abilities)",
"Body paragraph 3 (Connect your passion/goals to the company's mission/values - optional but effective)",
"Closing paragraph (Reiterate interest, mention enclosed resume, call to action)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, specific about your most relevant qualifications, and clear about your interest in the position. Tailor every cover letter to the specific job and company."
},
"career_change": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the position and acknowledge your career transition)",
"Body paragraph 1 (Highlight transferable skills from previous roles)",
"Body paragraph 2 (Explain your motivation for the career change and how your skills apply)",
"Body paragraph 3 (Demonstrate understanding of the new industry/role)",
"Closing paragraph (Reiterate enthusiasm, mention enclosed resume, call to action)",
"Complimentary close",
"Signature"
],
"guidance": "Focus on transferable skills and explain your career transition. Connect your past experience and new skills directly to the requirements of the target role."
},
"entry_level": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the position and your enthusiasm for the opportunity as a recent graduate/entrant)",
"Body paragraph 1 (Highlight relevant education, coursework, GPA if strong)",
"Body paragraph 2 (Describe relevant internships, projects, or volunteer experience)",
"Body paragraph 3 (Showcase soft skills: teamwork, communication, eagerness to learn)",
"Closing paragraph (Reiterate interest, mention attached resume, express availability for interview)",
"Complimentary close",
"Signature"
],
"guidance": "Emphasize education, relevant internships/projects, and transferable skills gained through academic or extracurricular activities. Show strong potential and enthusiasm."
},
"executive": {
"structure": [
"Your contact information",
"Date",
"Recipient's contact information (Senior Executive/Board Member)",
"Subject line (Executive Application - [Your Name] - [Position])",
"Salutation (Formal)",
"Introduction (State position applying for, brief summary of executive profile)",
"Body paragraph 1 (Highlight strategic leadership experience and key achievements)",
"Body paragraph 2 (Discuss relevant industry expertise and market insights)",
"Body paragraph 3 (Describe experience in driving growth, managing teams, achieving results)",
"Closing paragraph (Reiterate interest, express desire to discuss contribution to the organization)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Emphasize strategic leadership experience, significant achievements with measurable results, and industry expertise. Use a confident, authoritative, and forward-looking tone."
},
"creative": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Application - [Your Name] - [Creative Role])",
"Salutation",
"Creative introduction (Engaging hook related to the role or your passion)",
"Body paragraph 1 (Highlight relevant creative experience and skills)",
"Body paragraph 2 (Reference specific portfolio pieces or projects that showcase your style/abilities)",
"Body paragraph 3 (Describe your creative process or approach)",
"Closing paragraph (Reiterate enthusiasm, mention attached resume/portfolio link, call to action)",
"Complimentary close",
"Signature"
],
"guidance": "Use a more engaging and expressive style appropriate for a creative role while maintaining professionalism. Highlight specific creative achievements and link to your portfolio."
},
"technical": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Application - [Your Name] - [Technical Role])",
"Salutation (Formal)",
"Introduction (State position, source, and brief technical interest)",
"Body paragraph 1 (Highlight specific technical skills and proficiencies relevant to the job description)",
"Body paragraph 2 (Describe relevant technical projects or challenges you've solved)",
"Body paragraph 3 (Discuss problem-solving abilities and experience with relevant technologies)",
"Closing paragraph (Reiterate interest, mention attached resume, express availability for technical discussion/interview)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Focus on technical skills, relevant projects, and problem-solving abilities. Use appropriate technical terminology accurately."
},
"academic": {
"structure": [
"Your contact information",
"Date",
"Recipient's contact information (Search Committee Chair)",
"Subject line (Application for [Position] - [Your Name])",
"Salutation (Formal)",
"Introduction (State the position, the department, and express your strong interest)",
"Body paragraph 1 (Discuss your research experience, focus on key projects and contributions)",
"Body paragraph 2 (Describe your teaching philosophy and relevant teaching experience)",
"Body paragraph 3 (Mention publications, presentations, grants, and other scholarly contributions)",
"Closing paragraph (Reiterate enthusiasm for joining the faculty, express availability for interview/presentation)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Focus on research experience, teaching philosophy, publications, and contributions to the field. Use a scholarly and professional tone suitable for academia."
},
"remote": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Remote Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the remote position, source, and enthusiasm for remote work)",
"Body paragraph 1 (Highlight experience working remotely or independently)",
"Body paragraph 2 (Emphasize self-management, time management, and organizational skills required for remote work)",
"Body paragraph 3 (Describe strong written and verbal communication skills, essential for remote collaboration)",
"Closing paragraph (Reiterate interest in the remote role, mention attached resume, express availability for video interview)",
"Complimentary close",
"Signature"
],
"guidance": "Emphasize self-motivation, excellent communication skills (especially written), time management, and any prior experience working independently or in remote teams."
},
"referral": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Referral Application - [Your Name] - [Job Title] - Referred by [Referrer's Name])",
"Salutation",
"Referral introduction (Immediately state who referred you and for what position)",
"Body paragraph 1 (Briefly explain your connection to the referrer and how you learned about the role)",
"Body paragraph 2 (Highlight key qualifications relevant to the job description)",
"Body paragraph 3 (Express strong interest in the position and the company)",
"Closing paragraph (Reiterate enthusiasm, mention attached resume, express availability for interview)",
"Complimentary close",
"Signature"
],
"guidance": "Mention the referral prominently and early. Explain your connection to the referrer and how it aligns with your interest in the role. Still, ensure you highlight your own qualifications."
},
# Default cover letter template if subtype is not found
"default": {
"structure": [
"Contact information",
"Date",
"Recipient's information",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, specific about your qualifications, and clear about your interest in the position. Tailor your letter to the specific job and company."
}
},
# Overall default template if letter type is not recognized
"default": {
"structure": [
"Introduction",
"Body",
"Conclusion"
],
"guidance": "Be clear, concise, and appropriate for your audience and purpose. This is a generic structure."
}
}
def get_template_by_type(letter_type: str, subtype: str = "default") -> Dict[str, Any]:
"""
Get a template for a specific letter type and subtype using a dictionary lookup.
Args:
letter_type: Type of letter (e.g., "personal", "formal", "business", "cover").
subtype: Subtype of letter (e.g., "congratulations", "application", "sales").
Defaults to "default" if no subtype is specified.
Returns:
Template dictionary with 'structure' (List[str]) and 'guidance' (str).
Returns the default template if the letter type or subtype is not found,
ensuring the return structure is always consistent.
"""
# Get templates for the specific letter type, or the overall default templates
# .get() method is used for safe dictionary access with a default fallback
type_templates = TEMPLATES.get(letter_type, TEMPLATES["default"])
# Get the template for the specific subtype, or the default for that letter type
# Chain .get() calls to handle cases where subtype or the type's default is missing
template = type_templates.get(subtype, type_templates.get("default", TEMPLATES["default"]))
# Ensure the returned template always has 'structure' (as a list) and 'guidance' (as a string) keys.
# This adds robustness in case a template definition is incomplete.
if "structure" not in template or not isinstance(template["structure"], list):
# Fallback structure if missing or incorrect type
template["structure"] = ["Introduction", "Body", "Conclusion"]
# Update guidance to reflect that the structure was defaulted
template["guidance"] = "Generic template structure applied due to missing or invalid definition."
if "guidance" not in template or not isinstance(template["guidance"], str):
# Fallback guidance if missing or incorrect type
template["guidance"] = "Generic guidance applied due to missing or invalid definition."
return template
# Example usage (for testing purposes)
if __name__ == '__main__':
# Test cases to demonstrate functionality and default handling
print("--- Testing Letter Templates Module ---")
# Test a known personal letter subtype
personal_congrats = get_template_by_type("personal", "congratulations")
print("\nPersonal Congratulations Template:")
print(f"Structure: {personal_congrats['structure']}")
print(f"Guidance: {personal_congrats['guidance']}")
# Test a known formal letter subtype
formal_complaint = get_template_by_type("formal", "complaint")
print("\nFormal Complaint Template:")
print(f"Structure: {formal_complaint['structure']}")
print(f"Guidance: {formal_complaint['guidance']}")
# Test a known business letter subtype
business_sales = get_template_by_type("business", "sales")
print("\nBusiness Sales Template:")
print(f"Structure: {business_sales['structure']}")
print(f"Guidance: {business_sales['guidance']}")
# Test a known cover letter subtype
cover_entry_level = get_template_by_type("cover", "entry_level")
print("\nCover Entry Level Template:")
print(f"Structure: {cover_entry_level['structure']}")
print(f"Guidance: {cover_entry_level['guidance']}")
# Test an unknown letter type (should fallback to overall default)
unknown_type = get_template_by_type("unknown_type", "some_subtype")
print("\nUnknown Type Template (Should be Overall Default):")
print(f"Structure: {unknown_type['structure']}")
print(f"Guidance: {unknown_type['guidance']}")
# Test a known letter type but unknown subtype (should fallback to type's default)
personal_unknown_subtype = get_template_by_type("personal", "unknown_subtype")
print("\nPersonal Unknown Subtype Template (Should be Personal Default):")
print(f"Structure: {personal_unknown_subtype['structure']}")
print(f"Guidance: {personal_unknown_subtype['guidance']}")
# Test with only letter type (should use type's default)
formal_default = get_template_by_type("formal")
print("\nFormal Default Template (No Subtype Specified):")
print(f"Structure: {formal_default['structure']}")
print(f"Guidance: {formal_default['guidance']}")

236
ToBeMigrated/ai_writers/ai_letter_writer/main.py
View File

@@ -1,236 +0,0 @@
"""
AI Letter Writer - Main Module
This module provides a comprehensive interface for generating various types of letters
using AI assistance. It supports multiple letter formats, styles, and use cases.
It uses Streamlit for the user interface.
"""
import streamlit as st
# Assuming these modules exist in a package structure
from .letter_types import (
business_letters,
personal_letters,
formal_letters,
cover_letters,
recommendation_letters,
complaint_letters,
thank_you_letters,
invitation_letters
)
# Assuming these utility functions exist
from .utils.letter_formatter import format_letter
from .utils.letter_analyzer import analyze_letter_tone, check_formality
from .utils.letter_templates import get_template_by_type
# Define the letter types and their properties
LETTER_TYPES_CONFIG = [
{
"id": "business",
"name": "Business Letters",
"icon": "💼",
"description": "Professional correspondence for business contexts.",
"color": "#1E88E5", # Blue 600
"module": business_letters
},
{
"id": "personal",
"name": "Personal Letters",
"icon": "💌",
"description": "Heartfelt messages for friends and family.",
"color": "#43A047", # Green 600
"module": personal_letters
},
{
"id": "formal",
"name": "Formal Letters",
"icon": "📜",
"description": "Official correspondence for institutions and authorities.",
"color": "#5E35B1", # Deep Purple 600
"module": formal_letters
},
{
"id": "cover",
"name": "Cover Letters",
"icon": "📋",
"description": "Job application letters to showcase your qualifications.",
"color": "#FB8C00", # Orange 600
"module": cover_letters
},
{
"id": "recommendation",
"name": "Recommendation Letters",
"icon": "👍",
"description": "Endorse colleagues, students, or employees.",
"color": "#00ACC1", # Cyan 600
"module": recommendation_letters
},
{
"id": "complaint",
"name": "Complaint Letters",
"icon": "⚠️",
"description": "Address issues with products, services, or situations.",
"color": "#E53935", # Red 600
"module": complaint_letters
},
{
"id": "thank_you",
"name": "Thank You Letters",
"icon": "🙏",
"description": "Express gratitude for various occasions.",
"color": "#8E24AA", # Purple 600
"module": thank_you_letters
},
{
"id": "invitation",
"name": "Invitation Letters",
"icon": "🎉",
"description": "Invite people to events, interviews, or gatherings.",
"color": "#FFB300", # Amber 600
"module": invitation_letters
}
]
# Map letter type IDs to their modules for easy access
LETTER_MODULES_MAP = {config["id"]: config["module"] for config in LETTER_TYPES_CONFIG}
def initialize_session_state() -> None:
"""Initializes necessary Streamlit session state variables."""
if "letter_type" not in st.session_state:
st.session_state.letter_type = None
if "letter_subtype" not in st.session_state:
st.session_state.letter_subtype = None # Useful if a letter type has subtypes
if "generated_letter" not in st.session_state:
st.session_state.generated_letter = None
if "letter_metadata" not in st.session_state:
# Store information like sender, recipient, date, subject, tone, etc.
st.session_state.letter_metadata = {}
if "letter_input_data" not in st.session_state:
# Store user inputs for letter generation
st.session_state.letter_input_data = {}
def display_letter_type_selection() -> None:
"""Displays the letter type selection interface using a grid of styled containers with buttons."""
st.markdown("## Select Letter Type")
# Create a grid layout for the cards (3 columns)
cols = st.columns(3)
# Display each letter type as a card with a button below it
for i, letter_type_config in enumerate(LETTER_TYPES_CONFIG):
with cols[i % 3]:
# Use markdown to create a styled container for the card appearance
st.markdown(
f"""
<div style="
background-color: {letter_type_config['color']};
padding: 20px;
border-radius: 10px;
margin-bottom: 10px; /* Space between card content and button */
color: white;
min-height: 180px; /* Ensure consistent minimum height */
box-shadow: 0 4px 8px rgba(0, 0, 0, 0.2);
display: flex;
flex-direction: column;
justify-content: space-between; /* Distribute space within the card */
">
<h3 style="margin-top: 0; color: white;">{letter_type_config['icon']} {letter_type_config['name']}</h3>
<p style="color: white;">{letter_type_config['description']}</p>
</div>
""",
unsafe_allow_html=True
)
# Place the Streamlit button below the styled container
# Make the button expand to the width of the column for better alignment with the card
if st.button(
f"Select {letter_type_config['name']}",
key=f"btn_select_{letter_type_config['id']}", # Unique key for each button
use_container_width=True
):
st.session_state.letter_type = letter_type_config['id']
# Clear previous state data when selecting a new type
st.session_state.letter_subtype = None
st.session_state.generated_letter = None
st.session_state.letter_metadata = {}
st.session_state.letter_input_data = {}
st.rerun()
def display_letter_interface(letter_type_id: str) -> None:
"""
Displays the interface for the selected letter type by calling the
appropriate module's write function.
Args:
letter_type_id: The ID string of the selected letter type.
"""
module = LETTER_MODULES_MAP.get(letter_type_id)
if module:
try:
# Call the main function (e.g., write_letter or main) from the selected module
# Assuming the module has a function that renders its UI and handles generation
module.write_letter() # Assuming the function is named 'write_letter'
except AttributeError:
st.error(f"Module for '{letter_type_id}' does not have a 'write_letter' function.")
except Exception as e:
st.error(f"An error occurred while loading the interface for '{letter_type_id}': {e}")
else:
st.error(f"Letter type module '{letter_type_id}' not found in map.")
def write_letter() -> None:
"""Main function for the AI Letter Writer interface."""
# Page title and description
st.title("✉️ AI Letter Writer")
st.markdown("""
Create professional, personalized letters for any occasion. Select a letter type below to get started.
Our AI will help you craft the perfect letter with the right tone, structure, and content.
""")
# Initialize session state on first run
initialize_session_state()
# Back button logic - only show if a letter type is selected
if st.session_state.letter_type is not None:
if st.button("← Back to Letter Types"):
# Reset session state to return to selection
st.session_state.letter_type = None
st.session_state.letter_subtype = None
st.session_state.generated_letter = None
st.session_state.letter_metadata = {}
st.session_state.letter_input_data = {}
st.rerun() # Rerun to show the selection page
# Main navigation logic
if st.session_state.letter_type is None:
# Display letter type selection if no type is selected
display_letter_type_selection()
else:
# Display the interface for the selected letter type
display_letter_interface(st.session_state.letter_type)
# --- Placeholder for displaying generated letter and actions ---
# This part would typically be handled within the specific letter type modules
# after the letter is generated. However, if a common display is needed
# after returning from the module function, it would go here, but this
# requires the module function to somehow signal completion or store
# the generated letter in session state. The current structure expects
# the module's write_letter() to handle its entire lifecycle.
# Example of potentially displaying a generated letter after returning
# (This assumes the module updates st.session_state.generated_letter)
# if st.session_state.generated_letter:
# st.subheader("Generated Letter Preview")
# st.text_area("Your Letter", st.session_state.generated_letter, height=400)
# # Add options like copy, download, analyze, edit, etc.
if __name__ == "__main__":
# Run the main letter writing function when the script is executed
write_letter()

1121
ToBeMigrated/ai_writers/ai_letter_writer/personal_letter.py
View File

File diff suppressed because it is too large Load Diff

493
ToBeMigrated/ai_writers/ai_letter_writer/utils/letter_analyzer.py
View File

@@ -1,493 +0,0 @@
"""
Letter Analyzer Utility
This module provides functions for analyzing letter content, including tone,
formality, readability, and offering basic suggestions for improvement.
Note: The analysis methods provided here are simplified rule-based and
keyword-based approaches. For more sophisticated analysis in a production
environment, consider using advanced Natural Language Processing (NLP)
libraries and models.
"""
import re
from typing import Dict, Any, Tuple, List
def analyze_letter_tone(content: str) -> Dict[str, float]:
"""
Analyze the tone of a letter based on the presence of specific keywords
and phrases.
Args:
content: The letter content to analyze.
Returns:
Dictionary with tone scores (formal, friendly, assertive, etc.).
Scores are based on the frequency of matching patterns and capped at 1.0.
"""
# This is a simplified version using keyword matching.
# A more sophisticated approach would involve NLP libraries for sentiment and tone analysis.
# Initialize tone scores
# Scores are arbitrary counts normalized in a simple way
tone_scores = {
"formal": 0.0,
"friendly": 0.0,
"assertive": 0.0,
"respectful": 0.0,
"urgent": 0.0,
"apologetic": 0.0
}
# Define patterns for different tones (case-insensitive)
formal_patterns = [
r"\bI am writing to\b",
r"\bI would like to\b",
r"\bplease find\b",
r"\bregarding\b",
r"\bpursuant to\b",
r"\bhereby\b",
r"\bthus\b",
r"\btherefore\b",
r"\bfurthermore\b",
r"\bconsequently\b",
r"\bnevertheless\b",
r"\bmoreover\b",
r"\benclosed\b", # Added common formal word
r"\bherewith\b" # Added common formal word
]
friendly_patterns = [
r"\bhope you're well\b",
r"\bhope this finds you well\b",
r"\bgreat to hear\b",
r"\blooking forward\b",
r"\bthanks\b",
r"\bappreciate\b",
r"!", # Exclamation points often indicate friendly or excited tone
r"\bexcited\b",
r"\bgreat\b", # Common friendly adjective
r"\bnice\b" # Common friendly adjective
]
assertive_patterns = [
r"\brequire\b",
r"\bmust\b",
r"\bneed\b",
r"\bexpect\b",
r"\bdemand\b",
r"\binsist\b",
r"\bimmediately\b",
r"\baction\b", # Often used in assertive contexts
r"\bresolution\b" # Can imply assertion
]
respectful_patterns = [
r"\brespectfully\b",
r"\bhonored\b",
r"\bplease\b",
r"\bkindly\b",
r"\bgrateful\b",
r"\bthank you\b",
r"\bappreciate\b",
r"\bhumbly\b", # Added respectful word
r"\bapologies\b" # Can show respect for impact
]
urgent_patterns = [
r"\burgent\b",
r"\bas soon as possible\b",
r"\bASAP\b",
r"\bimmediately\b",
r"\bpressing\b",
r"\bcritical\b",
r"\bdeadline\b",
r"\bexpedite\b", # Added urgent word
r"\bpromptly\b" # Added urgent word
]
apologetic_patterns = [
r"\bapologize\b",
r"\bsorry\b",
r"\bregret\b",
r"\bmistake\b",
r"\berror\b",
r"\binconvenience\b",
r"\bfault\b", # Added apologetic word
r"\boversight\b" # Added apologetic word
]
# Count pattern matches and update scores (arbitrary weighting)
# A simple count multiplied by a factor acts as a basic indicator
for pattern in formal_patterns:
tone_scores["formal"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
for pattern in friendly_patterns:
tone_scores["friendly"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
for pattern in assertive_patterns:
tone_scores["assertive"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
for pattern in respectful_patterns:
tone_scores["respectful"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
for pattern in urgent_patterns:
tone_scores["urgent"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
for pattern in apologetic_patterns:
tone_scores["apologetic"] += len(re.findall(pattern, content, re.IGNORECASE)) * 0.2
# Cap scores at 1.0 (arbitrary capping)
# A more meaningful score might be relative frequency or use a proper model
for tone in tone_scores:
tone_scores[tone] = min(tone_scores[tone], 1.0)
return tone_scores
def check_formality(content: str) -> float:
"""
Check the formality level of a letter based on the presence of formal
vs. informal indicators and contractions.
Args:
content: The letter content to analyze.
Returns:
Formality score between 0.0 (very informal) and 1.0 (very formal).
Calculated as formal_count / (formal_count + informal_count).
"""
# This is a simplified version based on keyword counting.
# More accurate formality analysis would require advanced NLP techniques.
# Define formal and informal indicators (case-insensitive)
formal_indicators = [
r"\bDear\b",
r"\bSincerely\b",
r"\bRegards\b",
r"\bRespectfully\b",
r"\bI am writing to\b",
r"\bI would like to\b",
r"\bplease find\b",
r"\bregarding\b",
r"\bpursuant to\b",
r"\bhereby\b",
r"\bthus\b",
r"\btherefore\b",
r"\bfurthermore\b",
r"\bconsequently\b",
r"\bnevertheless\b",
r"\bmoreover\b",
r"\benclosed\b",
r"\bherewith\b",
r"\bsincerely yours\b", # Added
r"\bto whom it may concern\b" # Added
]
informal_indicators = [
r"\bHey\b",
r"\bHi\b",
r"\bWhat's up\b",
r"\bCheers\b",
r"\bThanks\b", # 'Thank you' is formal, 'Thanks' is informal
r"\bTake care\b",
r"\bSee you\b",
r"\bLater\b",
r"\bBye\b",
r"\bLove\b", # As a closing
r"\bXO\b",
r"!+", # Multiple exclamation points
r"\bawesome\b",
r"\bcool\b",
r"\bgreat\b",
r"\bnice\b",
r"\bbtw\b", # By the way
r"\bimo\b", # In my opinion
r"\blol\b" # Laugh out loud
]
# Define common contractions (case-insensitive)
contractions = [
r"\bdon't\b", r"\bcan't\b", r"\bwon't\b", r"\bshouldn't\b",
r"\bcouldn't\b", r"\bwouldn't\b", r"\bhasn't\b", r"\bhaven't\b",
r"\bisn't\b", r"\baren't\b", r"\bwasn't\b", r"\bweren't\b",
r"\bi'm\b", r"\byou're\b", r"\bhe's\b", r"\bshe's\b", r"\bit's\b",
r"\bwe're\b", r"\bthey're\b", r"\bi've\b", r"\byou've\b",
r"\bwe've\b", r"\bthey've\b", r"\bi'd\b", r"\byou'd\b",
r"\bhe'd\b", r"\bshe'd\b", r"\bit'd\b", r"\bwe'd\b", r"\bthey'd\b",
r"\bi'll\b", r"\byou'll\b", r"\bhe'll\b", r"\bshe'll\b", r"\bit'll\b",
r"\bwe'll\b", r"\bthey'll\b"
]
formal_count = 0
for pattern in formal_indicators:
formal_count += len(re.findall(pattern, content, re.IGNORECASE))
informal_count = 0
for pattern in informal_indicators:
informal_count += len(re.findall(pattern, content, re.IGNORECASE))
# Count contractions as informal indicators
for pattern in contractions:
informal_count += len(re.findall(pattern, content, re.IGNORECASE))
# Calculate formality score
total_indicators = formal_count + informal_count
if total_indicators == 0:
# If no indicators found, return a neutral score
return 0.5
# Score is the proportion of formal indicators
formality_score = formal_count / total_indicators
return formality_score
def count_syllables_simple(word: str) -> int:
"""
Counts syllables in a word using a simplified heuristic.
This method is not linguistically perfect but provides a basic estimate
for readability formulas.
Args:
word: The word string.
Returns:
Estimated syllable count.
"""
word = word.lower()
if len(word) <= 3:
# Assume short words have one syllable
return 1
# Remove common silent endings like 'e', 'es', 'ed'
if word.endswith(('es', 'ed')):
word = word[:-2]
elif word.endswith('e'):
word = word[:-1]
# Count vowel groups (consecutive vowels count as one syllable)
vowels = 'aeiouy'
count = 0
prev_is_vowel = False
for char in word:
is_vowel = char in vowels
if is_vowel and not prev_is_vowel:
count += 1
prev_is_vowel = is_vowel
# Ensure at least one syllable is counted
return max(1, count)
def get_readability_metrics(content: str) -> Dict[str, Any]:
"""
Calculate readability metrics for a letter using simplified methods
like Flesch Reading Ease.
Args:
content: The letter content to analyze.
Returns:
Dictionary with readability metrics: word_count, sentence_count,
avg_words_per_sentence, flesch_reading_ease, reading_level.
"""
# Split content into words and sentences using simple regex
words = re.findall(r'\b\w+\b', content)
# Split by common sentence terminators, handling potential multiple marks
sentences = re.split(r'[.!?]+\s*', content)
# Filter out empty strings resulting from the split (e.g., trailing punctuation)
sentences = [s for s in sentences if s.strip()]
word_count = len(words)
sentence_count = len(sentences)
syllable_count = sum(count_syllables_simple(word) for word in words)
if word_count == 0 or sentence_count == 0:
return {
"word_count": word_count,
"sentence_count": sentence_count,
"avg_words_per_sentence": 0.0,
"flesch_reading_ease": 0.0,
"reading_level": "N/A"
}
# Calculate average words per sentence
avg_words_per_sentence = word_count / sentence_count
# Calculate Flesch Reading Ease Score
# Formula: 206.835 - (1.015 * AvgWordsPerSentence) - (84.6 * AvgSyllablesPerWord)
# AvgSyllablesPerWord = syllable_count / word_count
avg_syllables_per_word = syllable_count / word_count if word_count > 0 else 0
flesch = 206.835 - (1.015 * avg_words_per_sentence) - (84.6 * avg_syllables_per_word)
# Clamp score between 0 and 100
flesch = max(0.0, min(100.0, flesch))
# Determine reading level based on Flesch score ranges
if flesch >= 90:
reading_level = "Very Easy (5th grade)"
elif flesch >= 80:
reading_level = "Easy (6th grade)"
elif flesch >= 70:
reading_level = "Fairly Easy (7th grade)"
elif flesch >= 60:
reading_level = "Standard (8th-9th grade)"
elif flesch >= 50:
reading_level = "Fairly Difficult (10th-12th grade)"
elif flesch >= 30:
reading_level = "Difficult (College)"
else:
reading_level = "Very Difficult (Graduate)"
return {
"word_count": word_count,
"sentence_count": sentence_count,
"avg_words_per_sentence": round(avg_words_per_sentence, 2), # Rounded for display
"flesch_reading_ease": round(flesch, 2), # Rounded for display
"reading_level": reading_level
}
def suggest_improvements(content: str, letter_type: str) -> List[str]:
"""
Suggest improvements for a letter based on its content, basic analysis,
and target letter type.
Args:
content: The letter content to analyze.
letter_type: The type of letter (e.g., "business", "cover", "personal").
Returns:
List of improvement suggestions strings.
"""
suggestions = []
words = re.findall(r'\b\w+\b', content)
word_count = len(words)
# Basic length check based on letter type
if letter_type in ["business", "formal"]:
if word_count < 100 and word_count > 10: # Avoid suggesting for very short placeholders
suggestions.append("Consider adding more details to make your letter more comprehensive.")
elif word_count > 600: # Increased max length slightly
suggestions.append("Your letter is quite long. Consider condensing it for better readability and focus.")
elif letter_type == "cover":
if word_count < 150 and word_count > 10: # Avoid suggesting for very short placeholders
suggestions.append("Your cover letter may be too brief. Consider highlighting more of your relevant qualifications.")
elif word_count > 500: # Increased max length slightly
suggestions.append("Your cover letter is quite long. Consider focusing on your most relevant qualifications and experiences.")
elif letter_type == "recommendation":
if word_count < 150 and word_count > 10:
suggestions.append("Consider adding more specific examples or anecdotes to strengthen the recommendation.")
elif word_count > 600:
suggestions.append("Your recommendation letter is quite long. Ensure it remains focused and impactful.")
# Check for overuse of "I" (simple count-based heuristic)
# Count "I" as a standalone word
i_count = len(re.findall(r"\bI\b", content))
# Avoid suggestion for very short content or content with few sentences
sentence_count = len(re.split(r'[.!?]+\s*', content.strip()))
if sentence_count > 2 and word_count > 50 and i_count > sentence_count * 1.5: # Suggest if 'I' count is significantly higher than sentence count
suggestions.append("Your letter contains many uses of 'I'. Consider rephrasing some sentences to focus more on the recipient or the subject matter.")
# Check for expression of gratitude (using common phrases)
gratitude_patterns = [r"\bthank you\b", r"\bgrateful\b", r"\bappreciate\b"]
has_gratitude = any(re.search(pattern, content, re.IGNORECASE) for pattern in gratitude_patterns)
# Suggest adding gratitude, but avoid for letter types where it might be less common (e.g., some complaint letters)
if not has_gratitude and letter_type not in ["complaint", "urgent"]:
suggestions.append("Consider expressing gratitude or appreciation somewhere in your letter.")
# Check for clear call to action (using common phrases)
# Phrases indicating desired action or next step
action_phrases = [
"look forward to", "please", "would appreciate", "request",
"hope to", "call me", "email me", "contact me", "schedule",
"arrange", "require action", "next steps"
]
has_call_to_action = any(phrase in content.lower() for phrase in action_phrases)
# Suggest adding a call to action for relevant letter types
if not has_call_to_action and letter_type in ["business", "cover", "complaint", "invitation"]:
suggestions.append("Consider adding a clear call to action or outlining the desired next steps.")
# Check for proper closing (using common phrases)
closing_patterns = [
r"\bSincerely\b", r"\bRegards\b", r"\bThank you\b", r"\bBest regards\b",
r"\bYours sincerely\b", r"\bYours faithfully\b", r"\bRespectfully\b",
r"\bBest wishes\b", r"\bKind regards\b"
]
# Check if any standard closing phrase is present, typically near the end
# A more robust check might look specifically at the last paragraph/lines
has_proper_closing = any(re.search(pattern, content[-200:], re.IGNORECASE) for pattern in closing_patterns) # Check last 200 chars
if not has_proper_closing and word_count > 20: # Avoid suggesting for very short snippets
suggestions.append("Consider adding a proper closing phrase (e.g., Sincerely, Regards) followed by your name.")
return suggestions
# Example usage (for testing purposes, not part of the module's core functionality)
if __name__ == '__main__':
sample_formal_letter = """
Dear Mr. Smith,
I am writing to follow up regarding the project proposal submitted on October 26, 2023.
We believe the proposed solution aligns well with your stated requirements.
Please find the revised budget document attached for your review.
We look forward to your feedback at your earliest convenience.
Sincerely,
Jane Doe
"""
sample_informal_letter = """
Hey John,
Hope you're doing well! Just wanted to quickly touch base about the party next week.
Excited to catch up with everyone! Let me know if you need any help setting up.
Thanks!
Best,
Alex
"""
sample_complaint_letter = """
To Whom It May Concern,
I am writing to complain about the faulty product I received on November 1, 2023 (Order #12345).
The device stopped working after only two days of use. I require a full refund or replacement immediately.
I expect a prompt response regarding this issue.
Sincerely,
Concerned Customer
"""
print("--- Analyzing Formal Letter ---")
tone = analyze_letter_tone(sample_formal_letter)
formality = check_formality(sample_formal_letter)
readability = get_readability_metrics(sample_formal_letter)
suggestions = suggest_improvements(sample_formal_letter, "business")
print(f"Tone: {tone}")
print(f"Formality: {formality:.2f}")
print(f"Readability: {readability}")
print(f"Suggestions: {suggestions}")
print("\n--- Analyzing Informal Letter ---")
tone = analyze_letter_tone(sample_informal_letter)
formality = check_formality(sample_informal_letter)
readability = get_readability_metrics(sample_informal_letter)
suggestions = suggest_improvements(sample_informal_letter, "personal")
print(f"Tone: {tone}")
print(f"Formality: {formality:.2f}")
print(f"Readability: {readability}")
print(f"Suggestions: {suggestions}")
print("\n--- Analyzing Complaint Letter ---")
tone = analyze_letter_tone(sample_complaint_letter)
formality = check_formality(sample_complaint_letter)
readability = get_readability_metrics(sample_complaint_letter)
suggestions = suggest_improvements(sample_complaint_letter, "complaint")
print(f"Tone: {tone}")
print(f"Formality: {formality:.2f}")
print(f"Readability: {readability}")
print(f"Suggestions: {suggestions}")

545
ToBeMigrated/ai_writers/ai_letter_writer/utils/letter_formatter.py
View File

@@ -1,545 +0,0 @@
"""
Letter Formatter Module
This module provides utilities for formatting letters and generating HTML
previews in different styles (Personal, Formal, Business, Cover).
The formatting functions here are primarily focused on generating HTML
for preview purposes, applying standard layout conventions for each letter type
using inline CSS styles.
"""
import re
from typing import Dict, Any
def format_letter(content: str, metadata: Dict[str, Any], letter_type: str = "personal") -> str:
"""
Format a letter with basic structure (paragraphs).
Args:
content: The raw letter content (string).
metadata: Dictionary containing metadata (currently not used for formatting in this placeholder).
letter_type: Type of letter (personal, formal, business, cover).
Returns:
Formatted letter content (currently just returns the input content).
This is a placeholder and would be expanded to apply specific
formatting rules (e.g., indentation, spacing) based on letter type
and metadata in a full implementation before generating HTML.
For this module, we primarily rely on the HTML generation functions
to handle the visual formatting.
"""
# This is a basic placeholder. In a real implementation, this function
# might process the raw text content to add indentation, adjust line breaks,
# or handle specific markdown-like syntax before it's passed to the
# HTML generation functions.
# For now, we assume the input `content` uses double newlines for paragraphs.
return content
def get_letter_preview_html(content: str, metadata: Dict[str, Any], letter_type: str = "personal") -> str:
"""
Generate HTML for letter preview based on letter type and metadata.
This function acts as a dispatcher to the specific HTML generation functions.
Args:
content: The letter content string.
metadata: Dictionary containing metadata like sender/recipient info, date, etc.
letter_type: Type of letter ("personal", "formal", "business", "cover").
Defaults to "personal".
Returns:
HTML string for letter preview, styled appropriately for the type.
Includes basic styling for a printable letter appearance.
"""
# Dispatch to the appropriate HTML generation function based on letter type
# Pass the content and metadata to the specific functions
if letter_type == "personal":
return get_personal_letter_html(content, metadata)
elif letter_type == "formal":
return get_formal_letter_html(content, metadata)
elif letter_type == "business":
return get_business_letter_html(content, metadata)
elif letter_type == "cover":
return get_cover_letter_html(content, metadata)
else:
# Fallback for unrecognized types, displaying raw content in a styled box
return f"""
<div style="max-width: 800px; margin: 20px auto; padding: 20px; border: 1px solid #ccc; font-family: sans-serif; line-height: 1.6; background-color: #fff8f8; color: #333; border-radius: 8px;">
<h3 style="color: #e53935; margin-top: 0;">Preview Unavailable for Unknown Letter Type</h3>
<p>The letter type '{letter_type}' is not recognized. Displaying raw content:</p>
<pre style="white-space: pre-wrap; word-wrap: break-word; background-color: #f8f8f8; padding: 15px; border: 1px solid #ddd; border-radius: 4px; overflow-x: auto;">{content}</pre>
</div>
"""
def get_personal_letter_html(content: str, metadata: Dict[str, Any]) -> str:
"""
Generate HTML for personal letter preview with basic styling.
Uses a more informal layout and font style.
Args:
content: The letter content string.
metadata: Dictionary containing personal letter metadata (sender_name, date).
Returns:
HTML string for personal letter preview.
"""
# Extract metadata with default empty strings for robustness
sender_name = metadata.get("sender_name", "")
# recipient_name = metadata.get("recipient_name", "") # Less common in personal body, but could be used in greeting
date = metadata.get("date", "")
# Split content into paragraphs based on double newlines
# Use list comprehension to strip whitespace and filter out empty strings
paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()]
# Format paragraphs as HTML <p> tags with bottom margin
formatted_paragraphs = "".join(f"<p style='margin-bottom: 1em;'>{paragraph}</p>" for paragraph in paragraphs)
# Basic HTML structure with inline styles for a personal letter feel
# Styles aim for a warm, readable appearance
html = f"""
<div style="max-width: 700px; margin: 20px auto; padding: 30px; border: 1px solid #e0e0e0; border-radius: 8px; background-color: #ffffff; font-family: 'Georgia', serif; line-height: 1.7; color: #333; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
<div style="text-align: right; margin-bottom: 30px; font-size: 0.9em; color: #555;">
{date if date else "[Date]"}
</div>
<div style="margin-bottom: 30px;">
{formatted_paragraphs if formatted_paragraphs else "<p style='color: #888;'>Letter content goes here...</p>"}
</div>
<div style="margin-top: 40px;">
<p style="margin-bottom: 0.5em;">Sincerely,</p>
<p style="font-weight: bold; margin-top: 0;">{sender_name if sender_name else "[Sender Name]"}</p>
</div>
</div>
"""
return html
def get_formal_letter_html(content: str, metadata: Dict[str, Any]) -> str:
"""
Generate HTML for formal letter preview with standard formal structure and styling.
Uses a more professional layout and font style (Arial/sans-serif).
Args:
content: The letter content string.
metadata: Dictionary containing formal letter metadata.
Returns:
HTML string for formal letter preview.
"""
# Extract metadata with default empty strings
sender_name = metadata.get("sender_name", "")
sender_title = metadata.get("sender_title", "")
sender_organization = metadata.get("sender_organization", "")
# Replace newlines in address for HTML display
sender_address = metadata.get("sender_address", "").replace("\n", "<br>")
sender_phone = metadata.get("sender_phone", "")
sender_email = metadata.get("sender_email", "")
recipient_name = metadata.get("recipient_name", "")
recipient_title = metadata.get("recipient_title", "")
recipient_organization = metadata.get("recipient_organization", "")
# Replace newlines in address for HTML display
recipient_address = metadata.get("recipient_address", "").replace("\n", "<br>")
date = metadata.get("date", "")
subject = metadata.get("subject", "") # Added subject line
salutation = metadata.get("salutation", "Dear Sir/Madam,") # Added salutation
complimentary_close = metadata.get("complimentary_close", "Sincerely,") # Added close
# Determine alignment based on letter format (simplified)
# Full Block: All aligned left
# Modified Block: Sender address block, date, closing, and signature are right-aligned
letter_format = metadata.get("letter_format", "Full Block")
sender_address_align = "left"
date_align = "left"
closing_align = "left"
if letter_format == "Modified Block":
sender_address_align = "right"
date_align = "right"
closing_align = "right"
# Split content into paragraphs based on double newlines
paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()]
# Format paragraphs as HTML <p> tags with bottom margin
formatted_paragraphs = "".join(f"<p style='margin-bottom: 1em;'>{paragraph}</p>" for paragraph in paragraphs)
# Basic HTML structure with inline styles for a formal letter
html = f"""
<div style="max-width: 800px; margin: 20px auto; padding: 30px; border: 1px solid #d0d0d0; border-radius: 8px; background-color: #ffffff; font-family: 'Arial', sans-serif; line-height: 1.6; color: #333; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
<div style="text-align: {sender_address_align}; margin-bottom: 20px; font-size: 0.9em;">
<p style="margin: 0;">{sender_name if sender_name else "[Sender Name]"}{', ' + sender_title if sender_title else ''}</p>
<p style="margin: 0;">{sender_organization if sender_organization else "[Sender Organization]"}</p>
<p style="margin: 0;">{sender_address if sender_address else "[Sender Address]"}</p>
<p style="margin: 0;">{sender_phone}</p>
<p style="margin: 0;">{sender_email}</p>
</div>
<div style="text-align: {date_align}; margin-bottom: 20px;">
<p style="margin: 0;">{date if date else "[Date]"}</p>
</div>
<div style="margin-bottom: 20px; font-size: 0.9em;">
<p style="margin: 0;">{recipient_name if recipient_name else "[Recipient Name]"}{', ' + recipient_title if recipient_title else ''}</p>
<p style="margin: 0;">{recipient_organization if recipient_organization else "[Recipient Organization]"}</p>
<p style="margin: 0;">{recipient_address if recipient_address else "[Recipient Address]"}</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0; font-weight: bold;">Subject: {subject if subject else "[Subject Line]"}</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0;">{salutation}</p>
</div>
<div style="margin-bottom: 20px;">
{formatted_paragraphs if formatted_paragraphs else "<p style='color: #888;'>Letter content goes here...</p>"}
</div>
<div style="margin-top: 40px; text-align: {closing_align};">
<p style="margin-bottom: 0.5em;">{complimentary_close}</p>
<p style="font-weight: bold; margin: 0;">{sender_name}</p>
<p style="margin: 0; font-size: 0.9em;">{sender_title}</p>
<p style="margin: 0; font-size: 0.9em;">{sender_organization}</p>
</div>
</div>
"""
return html
def get_business_letter_html(content: str, metadata: Dict[str, Any]) -> str:
"""
Generate HTML for business letter preview with standard business structure and styling.
Includes optional letterhead.
Args:
content: The letter content string.
metadata: Dictionary containing business letter metadata.
Returns:
HTML string for business letter preview.
"""
# Extract metadata with default empty strings
sender_company = metadata.get("sender_company", "")
sender_name = metadata.get("sender_name", "")
sender_title = metadata.get("sender_title", "")
sender_address = metadata.get("sender_address", "").replace("\n", "<br>")
sender_phone = metadata.get("sender_phone", "")
sender_email = metadata.get("sender_email", "")
sender_website = metadata.get("sender_website", "")
recipient_company = metadata.get("recipient_company", "")
recipient_name = metadata.get("recipient_name", "")
recipient_title = metadata.get("recipient_title", "")
recipient_address = metadata.get("recipient_address", "").replace("\n", "<br>")
date = metadata.get("date", "")
subject = metadata.get("subject", "") # Added subject line
salutation = metadata.get("salutation", "Dear Sir/Madam,") # Added salutation
complimentary_close = metadata.get("complimentary_close", "Sincerely,") # Added close
# Determine alignment based on letter format (simplified)
letter_format = metadata.get("letter_format", "Full Block")
sender_info_align = "left"
date_align = "left"
closing_align = "left"
if letter_format == "Modified Block":
sender_info_align = "right"
date_align = "right"
closing_align = "right"
# Include letterhead logic
include_letterhead = metadata.get("include_letterhead", True)
# Split content into paragraphs based on double newlines
paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()]
# Format paragraphs as HTML <p> tags with bottom margin
formatted_paragraphs = "".join(f"<p style='margin-bottom: 1em;'>{paragraph}</p>" for paragraph in paragraphs)
# Create letterhead HTML if included and company name is provided
letterhead_html = ""
if include_letterhead and sender_company:
letterhead_html = f"""
<div style="padding-bottom: 15px; margin-bottom: 20px; border-bottom: 1px solid #eee;">
<h2 style="margin: 0; color: #333; font-size: 1.5em;">{sender_company}</h2>
<p style="margin: 5px 0 0 0; font-size: 0.9em; color: #555;">
{sender_address.replace('<br>', ', ') if sender_address else ''}
{' | ' + sender_phone if sender_phone else ''}
{' | ' + sender_email if sender_email else ''}
{' | ' + sender_website if sender_website else ''}
</p>
</div>
"""
# Basic HTML structure with inline styles for a business letter
html = f"""
<div style="max-width: 800px; margin: 20px auto; padding: 30px; border: 1px solid #d0d0d0; border-radius: 8px; background-color: #ffffff; font-family: 'Arial', sans-serif; line-height: 1.6; color: #333; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
{letterhead_html}
<div style="text-align: {date_align}; margin-bottom: 20px;">
<p style="margin: 0;">{date if date else "[Date]"}</p>
</div>
<div style="margin-bottom: 20px; font-size: 0.9em;">
<p style="margin: 0;">{recipient_name if recipient_name else "[Recipient Name]"}{', ' + recipient_title if recipient_title else ''}</p>
<p style="margin: 0;">{recipient_company if recipient_company else "[Recipient Company]"}</p>
<p style="margin: 0;">{recipient_address if recipient_address else "[Recipient Address]"}</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0; font-weight: bold;">Subject: {subject if subject else "[Subject Line]"}</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0;">{salutation}</p>
</div>
<div style="margin-bottom: 20px;">
{formatted_paragraphs if formatted_paragraphs else "<p style='color: #888;'>Letter content goes here...</p>"}
</div>
<div style="margin-top: 40px; text-align: {closing_align};">
<p style="margin-bottom: 0.5em;">{complimentary_close}</p>
<p style="font-weight: bold; margin: 0;">{sender_name if sender_name else "[Sender Name]"}</p>
<p style="margin: 0; font-size: 0.9em;">{sender_title}</p>
<p style="margin: 0; font-size: 0.9em;">{sender_company}</p>
</div>
</div>
"""
return html
def get_cover_letter_html(content: str, metadata: Dict[str, Any]) -> str:
"""
Generate HTML for cover letter preview with standard cover letter structure and styling.
Includes sender contact block and optional online links.
Args:
content: The letter content string.
metadata: Dictionary containing cover letter metadata.
Returns:
HTML string for cover letter preview.
"""
# Extract metadata with default empty strings
sender_name = metadata.get("sender_name", "")
sender_email = metadata.get("sender_email", "")
sender_phone = metadata.get("sender_phone", "")
sender_location = metadata.get("sender_location", "")
sender_linkedin = metadata.get("sender_linkedin", "")
sender_portfolio = metadata.get("sender_portfolio", "")
recipient_name = metadata.get("recipient_name", "")
recipient_title = metadata.get("recipient_title", "") # Added recipient title
recipient_company = metadata.get("recipient_company", "")
recipient_department = metadata.get("recipient_department", "") # Added department
recipient_address = metadata.get("recipient_address", "").replace("\n", "<br>") # Added recipient address
date = metadata.get("date", "")
job_title = metadata.get("job_title", "") # Added job title for subject
salutation = metadata.get("salutation", "Dear Hiring Manager,") # Added salutation
complimentary_close = metadata.get("complimentary_close", "Sincerely,") # Added close
# Split content into paragraphs based on double newlines
paragraphs = [p.strip() for p in content.split("\n\n") if p.strip()]
# Format paragraphs as HTML <p> tags with bottom margin
formatted_paragraphs = "".join(f"<p style='margin-bottom: 1em;'>{paragraph}</p>" for paragraph in paragraphs)
# Construct sender contact line, only including fields that have values
sender_contact_parts = [sender_location, sender_phone, sender_email]
sender_contact_line = " | ".join(filter(None, sender_contact_parts))
# Construct sender online links line, only including fields that have values
sender_online_parts = []
if sender_linkedin:
# Add basic styling for links
sender_online_parts.append(f'<a href="{sender_linkedin}" style="color: #0077b5; text-decoration: none;">LinkedIn</a>')
if sender_portfolio:
# Add basic styling for links
sender_online_parts.append(f'<a href="{sender_portfolio}" style="color: #0077b5; text-decoration: none;">Portfolio</a>')
sender_online_line = " | ".join(filter(None, sender_online_parts))
# Basic HTML structure with inline styles for a cover letter
# Styles aim for a clean, professional look
html = f"""
<div style="max-width: 800px; margin: 20px auto; padding: 30px; border: 1px solid #d0d0d0; border-radius: 8px; background-color: #ffffff; font-family: 'Arial', sans-serif; line-height: 1.6; color: #333; box-shadow: 0 2px 4px rgba(0,0,0,0.1);">
<div style="text-align: left; margin-bottom: 30px; padding-bottom: 15px; border-bottom: 1px solid #eee;">
<h2 style="margin: 0; color: #333; font-size: 1.5em;">{sender_name if sender_name else "[Your Name]"}</h2>
{'<p style="margin: 5px 0 0 0; font-size: 0.9em; color: #555;">' + sender_contact_line + '</p>' if sender_contact_line else ''}
{'<p style="margin: 2px 0 0 0; font-size: 0.9em;">' + sender_online_line + '</p>' if sender_online_line else ''}
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0;">{date if date else "[Date]"}</p>
</div>
<div style="margin-bottom: 20px; font-size: 0.9em;">
<p style="margin: 0;">{recipient_name if recipient_name else "[Recipient Name]"}{', ' + recipient_title if recipient_title else ''}</p>
<p style="margin: 0;">{recipient_department}</p>
<p style="margin: 0;">{recipient_company if recipient_company else "[Recipient Company]"}</p>
<p style="margin: 0;">{recipient_address if recipient_address else "[Recipient Address]"}</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0; font-weight: bold;">Subject: Application for {job_title if job_title else '[Job Title]'} Position</p>
</div>
<div style="margin-bottom: 20px;">
<p style="margin: 0;">{salutation}</p>
</div>
<div style="margin-bottom: 20px;">
{formatted_paragraphs if formatted_paragraphs else "<p style='color: #888;'>Letter content goes here...</p>"}
</div>
<div style="margin-top: 40px;">
<p style="margin-bottom: 0.5em;">{complimentary_close}</p>
<p style="font-weight: bold; margin: 0;">{sender_name}</p>
</div>
</div>
"""
return html
# Example usage (for testing purposes)
if __name__ == '__main__':
sample_personal_content = """
Hi Sarah,
Hope you're doing well!
Just wanted to send a quick note to say how much I enjoyed catching up last week. It was great hearing about your trip to Italy.
Let's try to do it again soon!
Best,
Emily
"""
sample_personal_metadata = {
"sender_name": "Emily Davis",
"recipient_name": "Sarah Johnson",
"date": "November 5, 2023"
}
sample_formal_content = """
I am writing to formally request a copy of my academic transcript.
I require this document for a graduate school application. The deadline for submission is December 15, 2023.
Please let me know if there are any fees associated with this request or if any further information is needed from my end.
Thank you for your time and assistance.
"""
sample_formal_metadata_full_block = {
"sender_name": "John Smith",
"sender_title": "Student",
"sender_organization": "University of Example",
"sender_address": "123 University Ave\nAnytown, CA 91234",
"sender_phone": "(555) 123-4567",
"sender_email": "john.smith@example.com",
"recipient_name": "Registrar's Office",
"recipient_organization": "University of Example",
"recipient_address": "456 Admin Building\nAnytown, CA 91234",
"date": "November 5, 2023",
"subject": "Request for Academic Transcript",
"salutation": "To the Registrar's Office,",
"complimentary_close": "Sincerely,",
"letter_format": "Full Block"
}
sample_formal_metadata_modified_block = sample_formal_metadata_full_block.copy()
sample_formal_metadata_modified_block["letter_format"] = "Modified Block"
sample_business_content = """
This letter confirms the details of Purchase Order #PO-7890.
We are ordering 50 units of Model X widgets at the agreed-upon price of $100 per unit, totaling $5,000.
Please ensure delivery to our warehouse by November 20, 2023. Payment will be made within 30 days of receipt of invoice.
Thank you for your prompt processing of this order.
"""
sample_business_metadata_full_block = {
"sender_company": "Acme Corp",
"sender_name": "Alice Brown",
"sender_title": "Procurement Manager",
"sender_address": "789 Business Rd\nMetropolis, NY 10001",
"sender_phone": "(555) 987-6543",
"sender_email": "alice.brown@acmecorp.com",
"sender_website": "www.acmecorp.com",
"recipient_company": "Supplier Co.",
"recipient_name": "Sales Department",
"recipient_title": "",
"recipient_address": "101 Vendor Lane\nIndustriatown, TX 75001",
"date": "November 5, 2023",
"subject": "Purchase Order Confirmation - PO-7890",
"salutation": "To the Sales Department,",
"complimentary_close": "Sincerely,",
"letter_format": "Full Block",
"include_letterhead": True
}
sample_business_metadata_modified_block = sample_business_metadata_full_block.copy()
sample_business_metadata_modified_block["letter_format"] = "Modified Block"
sample_business_metadata_no_letterhead = sample_business_metadata_full_block.copy()
sample_business_metadata_no_letterhead["include_letterhead"] = False
sample_cover_letter_content = """
I am writing to express my enthusiastic interest in the Marketing Specialist position advertised on LinkedIn.
With three years of experience in digital marketing and a proven track record in content creation and social media management, I am confident in my ability to contribute to your team. My skills in [Specific Skill 1] and [Specific Skill 2] align perfectly with the requirements outlined in the job description.
In my previous role at [Previous Company], I successfully managed social media campaigns that resulted in a 25% increase in engagement. I am particularly drawn to [Company Name]'s innovative approach to [Industry Trend] and believe my creative problem-solving skills would be a valuable asset.
Thank you for considering my application. I have attached my resume for your review and welcome the opportunity to discuss how my background and skills can benefit [Company Name].
"""
sample_cover_letter_metadata = {
"sender_name": "Jane Doe",
"sender_email": "jane.doe@email.com",
"sender_phone": "(123) 456-7890",
"sender_location": "San Francisco, CA",
"sender_linkedin": "https://linkedin.com/in/janedoe",
"sender_portfolio": "https://janedoeportfolio.com",
"recipient_name": "Hiring Manager",
"recipient_title": "", # Example with no recipient title
"recipient_company": "Innovative Solutions Inc.",
"recipient_department": "Marketing Department",
"recipient_address": "456 Tech Way\nSilicon Valley, CA 95001",
"date": "November 5, 2023",
"job_title": "Marketing Specialist",
"salutation": "Dear Hiring Manager,",
"complimentary_close": "Sincerely,"
}
print("--- Personal Letter HTML Preview ---")
print(get_letter_preview_html(sample_personal_content, sample_personal_metadata, letter_type="personal"))
print("\n--- Formal Letter HTML Preview (Full Block) ---")
print(get_letter_preview_html(sample_formal_content, sample_formal_metadata_full_block, letter_type="formal"))
print("\n--- Formal Letter HTML Preview (Modified Block) ---")
print(get_letter_preview_html(sample_formal_content, sample_formal_metadata_modified_block, letter_type="formal"))
print("\n--- Business Letter HTML Preview (Full Block, with Letterhead) ---")
print(get_letter_preview_html(sample_business_content, sample_business_metadata_full_block, letter_type="business"))
print("\n--- Business Letter HTML Preview (Modified Block, with Letterhead) ---")
print(get_letter_preview_html(sample_business_content, sample_business_metadata_modified_block, letter_type="business"))
print("\n--- Business Letter HTML Preview (Full Block, no Letterhead) ---")
print(get_letter_preview_html(sample_business_content, sample_business_metadata_no_letterhead, letter_type="business"))
print("\n--- Cover Letter HTML Preview ---")
print(get_letter_preview_html(sample_cover_letter_content, sample_cover_letter_metadata, letter_type="cover"))
print("\n--- Unknown Type HTML Preview ---")
print(get_letter_preview_html("Some random content.", {}, letter_type="unknown"))

988
ToBeMigrated/ai_writers/ai_letter_writer/utils/letter_templates.py
View File

@@ -1,988 +0,0 @@
"""
Letter Templates Module
This module provides structured templates and guidance for generating
different types and subtypes of letters.
Templates are defined as dictionaries containing a 'structure' (list of sections)
and 'guidance' (a string).
"""
from typing import Dict, Any, List
# Define letter templates using a nested dictionary structure for easier management
TEMPLATES: Dict[str, Dict[str, Dict[str, Any]]] = {
"personal": {
"congratulations": {
"structure": [
"Greeting",
"Express congratulations",
"Acknowledge the achievement",
"Share personal thoughts/memory (optional)",
"Look to the future/well wishes",
"Closing"
],
"guidance": "Be warm, sincere, and specific about the achievement. Express genuine happiness for the recipient. Keep the tone personal and friendly."
},
"thank_you": {
"structure": [
"Greeting",
"Express gratitude clearly",
"Specify what you are thankful for",
"Explain the impact or how you used it (optional)",
"Share a personal thought or memory (optional)",
"Offer reciprocation or look to the future",
"Closing"
],
"guidance": "Be specific about what you're thankful for and how it affected you. Express sincere appreciation. Personalize the message."
},
"sympathy": {
"structure": [
"Greeting",
"Express sympathy for the loss",
"Acknowledge the significance of the person/situation",
"Share a positive memory or quality (optional)",
"Offer specific support (optional)",
"Closing with comforting words"
],
"guidance": "Be gentle, compassionate, and sincere. Avoid clichés. Focus on offering genuine comfort and acknowledging the recipient's feelings."
},
"apology": {
"structure": [
"Greeting",
"Clearly state your apology",
"Acknowledge the specific mistake or action",
"Express understanding of the impact on the other person",
"Explain (briefly, without making excuses) what happened (optional)",
"Offer amends or suggest how to make things right",
"Assure it won't happen again",
"Closing"
],
"guidance": "Be sincere, take full responsibility for your actions, and focus on making things right. Avoid making excuses or blaming others."
},
"invitation": {
"structure": [
"Greeting",
"Clearly state the invitation",
"Provide full event details (What, When, Where)",
"Explain the significance or purpose (optional)",
"Mention who else might be there (optional)",
"Request RSVP (date and contact method)",
"Express anticipation",
"Closing"
],
"guidance": "Be clear and specific about the details (what, when, where, why). Make it easy for the person to respond."
},
"friendship": {
"structure": [
"Greeting",
"Express appreciation for the friendship",
"Share a recent memory or anecdote",
"Acknowledge the value of the relationship",
"Check in on them or share updates",
"Look to the future (getting together, etc.)",
"Closing"
],
"guidance": "Be warm, personal, and specific about what you value in the friendship. Share updates and show genuine interest."
},
"love": {
"structure": [
"Greeting (Terms of endearment)",
"Express depth of feelings",
"Share a cherished memory or moment",
"Describe specific qualities you love and appreciate",
"Reaffirm commitment or future hopes",
"Closing (Terms of endearment)"
],
"guidance": "Be sincere, personal, and specific about your feelings. Use sensory details and emotional language appropriate for your relationship."
},
"encouragement": {
"structure": [
"Greeting",
"Acknowledge the situation or challenge they face",
"Express belief in their abilities/strength",
"Offer specific words of encouragement or support",
"Remind them of past successes (optional)",
"Offer practical help (optional)",
"Look to the future with hope",
"Closing with support"
],
"guidance": "Be positive, supportive, and specific about the person's strengths and abilities. Offer genuine encouragement and belief in them."
},
"farewell": {
"structure": [
"Greeting",
"State the purpose (saying goodbye)",
"Express feelings about their departure (sadness, happiness for them)",
"Share a positive memory or highlight their contribution",
"Express good wishes for their future endeavors",
"Look to staying in touch (optional)",
"Closing"
],
"guidance": "Be warm, reflective, and forward-looking. Focus on positive memories and express genuine good wishes for their next steps."
},
# Default personal letter template if subtype is not found
"default": {
"structure": [
"Greeting",
"Introduction",
"Main content paragraphs",
"Closing thoughts",
"Signature"
],
"guidance": "Be personal, authentic, and appropriate for your relationship with the recipient. The tone is typically informal to semi-formal."
}
},
"formal": {
"application": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (if known)",
"Subject line (Clear and concise)",
"Salutation (Formal)",
"Introduction (State position applied for and where you saw it)",
"Body paragraphs (Highlight relevant skills and experience)",
"Closing paragraph (Reiterate interest, mention enclosed resume, call to action)",
"Complimentary close (Formal)",
"Signature (Typed name)",
"Enclosures (Mention if attaching resume/portfolio)"
],
"guidance": "Be professional, concise, and specific about your qualifications and genuine interest in the position. Tailor it to the specific job description."
},
"complaint": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state it's a complaint)",
"Salutation (Formal)",
"Introduction (State the purpose: complaint about X service/product)",
"Problem description (Provide specific details: date, time, location, product details, names if applicable)",
"Impact statement (Explain how the problem affected you)",
"Requested resolution (Clearly state what you want: refund, replacement, action)",
"Closing paragraph (Reference attached documents, state expectation for response)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, factual, and specific about the issue and your desired resolution. Maintain a respectful but firm tone. Include all relevant details."
},
"request": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state the request)",
"Salutation (Formal)",
"Introduction (State the purpose: making a request)",
"Request details (Clearly explain what you are requesting)",
"Justification (Explain why the request is necessary or beneficial)",
"Provide supporting information (optional)",
"Closing paragraph (Express gratitude for consideration, reiterate call to action)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, specific, and courteous about your request. Explain why it's important or beneficial to the recipient or organization."
},
"recommendation": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Letter of Recommendation for [Name])",
"Salutation (Formal)",
"Introduction (State your name, title, relationship to the recommendee, and for what purpose the letter is written)",
"Body paragraphs (Describe the recommendee's qualifications, skills, and achievements with specific examples)",
"Highlight relevant experiences and contributions",
"Closing recommendation (Summarize endorsement, strongly recommend the person)",
"Complimentary close (Formal)",
"Signature (Typed name and title)"
],
"guidance": "Be specific, positive, and credible. Use concrete examples and anecdotes to support your recommendation. Tailor it to the specific role/opportunity."
},
"resignation": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (Immediate supervisor/HR)",
"Subject line (Letter of Resignation - [Your Name])",
"Salutation (Formal)",
"Statement of resignation (Clearly state you are resigning)",
"Last day of employment (Specify the date)",
"Gratitude and reflection (Optional: Express thanks for the opportunity/experience)",
"Transition plan/Offer of assistance (Optional: Suggest how to ensure a smooth handover)",
"Closing paragraph (Express good wishes for the company's future)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, positive (if possible), and clear about your departure and last day. Maintain a good relationship."
},
"inquiry": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Clearly state the nature of the inquiry)",
"Salutation (Formal)",
"Introduction (State your purpose for writing - making an inquiry)",
"Inquiry details (Provide necessary context or background)",
"Specific questions (List your questions clearly, perhaps numbered)",
"Closing paragraph (Express gratitude for assistance, indicate when you need a response)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be clear, specific, and courteous about your inquiry. Organize your questions logically for easy answering."
},
"authorization": {
"structure": [
"Sender's contact information (The grantor of authority)",
"Date",
"Recipient's contact information (The person/entity receiving the letter)",
"Subject line (Letter of Authorization)",
"Salutation (Formal)",
"Statement of authorization (Clearly state who is authorized)",
"Authorized person's details (Full name, ID if applicable)",
"Scope of authority (Precisely define what they are authorized to do)",
"Limitations (Specify any restrictions or conditions)",
"Duration of authorization (Start and end dates, if applicable)",
"Closing paragraph (State responsibility, express confidence)",
"Complimentary close (Formal)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and precise about who is authorized, what they can do, for how long, and under what conditions. This is a legal document."
},
"appeal": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (Appeals committee/relevant authority)",
"Subject line (Letter of Appeal - [Your Name] - [Subject of Appeal])",
"Salutation (Formal)",
"Introduction (State your name, the decision being appealed, and the date of the decision)",
"Grounds for appeal (Clearly state the reasons why you believe the decision is incorrect)",
"Provide supporting evidence (Reference attached documents: records, photos, etc.)",
"Explain mitigating circumstances (Optional)",
"Requested outcome (Clearly state what resolution you seek)",
"Closing paragraph (Express hope for reconsideration, gratitude for time)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be respectful, factual, and persuasive. Focus on valid grounds for appeal and provide clear, supporting evidence. Maintain a formal tone."
},
"introduction": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Introduction - [Your Name])",
"Salutation (Formal)",
"Introduction (Introduce yourself and the purpose of the letter)",
"Background information (Briefly describe your relevant background or expertise)",
"Reason for reaching out (Explain why you are introducing yourself to this specific person/entity)",
"Potential areas of collaboration or shared interest (Optional)",
"Call to action (Suggest a meeting, call, or further communication)",
"Closing paragraph (Express enthusiasm for potential connection)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, informative, and engaging. Clearly explain who you are, your expertise, and why you're reaching out to them specifically."
},
# Default formal letter template if subtype is not found
"default": {
"structure": [
"Sender's address",
"Date",
"Recipient's address",
"Subject line",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, clear, and concise. Use formal language and structure. The tone is typically formal."
}
},
"business": {
"sales": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Benefit-oriented)",
"Salutation",
"Attention-grabbing opening (Address a pain point or introduce a benefit)",
"Problem statement (Briefly describe the challenge the recipient faces)",
"Solution presentation (Introduce your product/service as the solution)",
"Benefits and features (Explain how your solution helps, focusing on benefits)",
"Social proof (Optional: Testimonials, case studies, data)",
"Call to action (Clearly state what you want them to do next)",
"Closing paragraph (Reiterate benefit, create urgency/incentive)",
"Complimentary close (Professional)",
"Signature (Typed name and title)",
"Enclosures (Optional: Brochure, pricing)"
],
"guidance": "Be persuasive, customer-focused, and clear about the value proposition. Focus on benefits, not just features. Make the call to action obvious."
},
"proposal": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Clear and descriptive)",
"Salutation",
"Introduction (State purpose: submitting a proposal)",
"Problem statement/Needs assessment (Demonstrate understanding of client's needs)",
"Proposed solution (Describe your solution in detail)",
"Implementation plan (Outline steps and timeline)",
"Costs and investment (Clearly state pricing and payment terms)",
"Benefits and ROI (Explain the value the client will receive)",
"Call to action (Suggest next steps: meeting, discussion)",
"Closing paragraph (Express enthusiasm, availability for questions)",
"Complimentary close (Professional)",
"Signature (Typed name and title)",
"Enclosures (Proposal document, appendix)"
],
"guidance": "Be clear, specific, and persuasive about your solution. Focus on the client's needs and the value you provide. Structure it logically."
},
"order": {
"structure": [
"Letterhead (Your company)",
"Date",
"Recipient's address (Supplier)",
"Subject line (Purchase Order - [PO Number])",
"Salutation",
"Introduction (Reference quote/agreement, state purpose: placing an order)",
"Order details (Item list with quantities, descriptions, unit prices, total)",
"Delivery requirements (Shipping address, requested delivery date, shipping method)",
"Payment terms (Reference agreed terms)",
"Closing paragraph (Express expectation for timely delivery)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and detailed about what you're ordering, quantities, delivery requirements, and payment terms. Include a purchase order number."
},
"quotation": {
"structure": [
"Letterhead (Your company)",
"Date",
"Recipient's address (Customer)",
"Subject line (Quotation for [Product/Service])",
"Salutation",
"Introduction (Reference inquiry, state purpose: providing a quotation)",
"Quotation details (List items/services, descriptions, unit prices, quantities, line totals)",
"Pricing breakdown (Mention taxes, discounts, fees separately)",
"Terms and conditions (Payment terms, delivery terms, warranty)",
"Validity period (State how long the quote is valid)",
"Next steps (How they can place an order)",
"Closing paragraph (Express hope to do business, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and transparent about pricing, terms, and what's included or excluded. Make it easy for the customer to understand and accept."
},
"acknowledgment": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Acknowledgment of [Received Item/Request])",
"Salutation",
"Acknowledgment statement (Clearly state what you have received or are acknowledging)",
"Details of what's being acknowledged (Reference number, date, brief description)",
"Confirm understanding (Optional: Briefly restate the request/issue to show understanding)",
"Next steps (Outline what will happen next, e.g., processing order, investigating issue)",
"Timeline (Provide an estimated timeframe if possible)",
"Closing paragraph (Express gratitude, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be prompt, clear, and specific about what you're acknowledging. Set clear expectations for next steps and timelines."
},
"collection": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Invoice [Invoice Number] - Payment Due)",
"Salutation",
"Introduction (Reference invoice number and due date)",
"Account status (Clearly state the outstanding amount)",
"Payment request (Politely request payment)",
"Payment options (Remind them how to pay)",
"Consequences of non-payment (Optional: Briefly mention late fees or further action, depending on letter stage)",
"Call to action (Request payment by a specific date)",
"Closing paragraph (Express hope for prompt payment, offer to discuss)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be firm but professional. Clearly state the amount due, due date, and payment options. The tone may vary depending on how overdue the payment is."
},
"adjustment": {
"structure": [
"Letterhead",
"Date",
"Recipient's address (Customer who made a complaint)",
"Subject line (Response to your inquiry - [Reference Number])",
"Salutation",
"Acknowledgment of complaint (Reference their communication and the issue)",
"Investigation findings (Explain the outcome of your investigation)",
"Adjustment offered (Clearly state the resolution: refund, replacement, credit, etc.)",
"Apology (Optional: Express regret for the inconvenience)",
"Preventive measures (Optional: Explain steps taken to prevent recurrence)",
"Closing paragraph (Express hope for continued business, offer further assistance)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be responsive, empathetic, and solution-oriented. Clearly explain the adjustment and any preventive measures taken."
},
"credit": {
"structure": [
"Letterhead",
"Date",
"Recipient's address (Applicant)",
"Subject line (Credit Application Status - [Applicant Name])",
"Salutation",
"Introduction (Reference their credit application and the purpose of the letter)",
"Credit decision (Clearly state if credit is approved or denied)",
"If approved: Credit terms (Credit limit, payment terms, interest rates)",
"If denied: Reason for decision (Provide specific, compliant reasons)",
"Requirements (If approved: any further steps or documents needed)",
"Closing paragraph (If approved: Express welcome; If denied: Offer alternative options or appeals process)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and transparent about the credit decision, terms, limits, or reasons for denial. Ensure compliance with regulations if denying credit."
},
"follow_up": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line (Following up on [Previous Communication/Meeting])",
"Salutation",
"Reference to previous communication (Mention date, topic, or meeting)",
"Purpose of follow-up (Clearly state why you are writing again)",
"Action items/Next steps (Remind of agreed-upon actions or propose next steps)",
"Provide additional information (Optional)",
"Call to action (If applicable, e.g., request a response, schedule a meeting)",
"Closing paragraph (Reiterate interest, express anticipation)",
"Complimentary close (Professional)",
"Signature (Typed name and title)"
],
"guidance": "Be clear, specific, and action-oriented. Reference previous communication and clearly state the purpose of your follow-up and desired outcome."
},
# Default business letter template if subtype is not found
"default": {
"structure": [
"Letterhead",
"Date",
"Recipient's address",
"Subject line",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, clear, and concise. Focus on the business purpose of your letter. The tone is typically formal to semi-formal."
}
},
"cover": {
"standard": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information (if known)",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation (Formal)",
"Introduction (State the position you are applying for, where you saw the advertisement, and a brief statement of enthusiasm)",
"Body paragraph 1 (Highlight skills and experience directly relevant to the job description - often 1-2 key qualifications)",
"Body paragraph 2 (Provide a specific example or anecdote demonstrating your abilities)",
"Body paragraph 3 (Connect your passion/goals to the company's mission/values - optional but effective)",
"Closing paragraph (Reiterate interest, mention attached resume, express availability for interview)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Be professional, specific about your most relevant qualifications, and clear about your interest in the position. Tailor every cover letter to the specific job and company."
},
"career_change": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the position and acknowledge your career transition)",
"Body paragraph 1 (Highlight transferable skills from previous roles)",
"Body paragraph 2 (Explain your motivation for the career change and how your skills apply)",
"Body paragraph 3 (Demonstrate understanding of the new industry/role)",
"Closing paragraph (Reiterate enthusiasm, mention enclosed resume, call to action)",
"Complimentary close",
"Signature"
],
"guidance": "Focus on transferable skills and explain your career transition. Connect your past experience and new skills directly to the requirements of the target role."
},
"entry_level": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Job Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the position and your enthusiasm for the opportunity as a recent graduate/entrant)",
"Body paragraph 1 (Highlight relevant education, coursework, GPA if strong)",
"Body paragraph 2 (Describe relevant internships, projects, or volunteer experience)",
"Body paragraph 3 (Showcase soft skills: teamwork, communication, eagerness to learn)",
"Closing paragraph (Reiterate interest, mention attached resume, express availability for interview)",
"Complimentary close",
"Signature"
],
"guidance": "Emphasize education, relevant internships/projects, and transferable skills gained through academic or extracurricular activities. Show strong potential and enthusiasm."
},
"executive": {
"structure": [
"Your contact information",
"Date",
"Recipient's contact information (Senior Executive/Board Member)",
"Subject line (Executive Application - [Your Name] - [Position])",
"Salutation (Formal)",
"Introduction (State position applying for, brief summary of executive profile)",
"Body paragraph 1 (Highlight strategic leadership experience and key achievements)",
"Body paragraph 2 (Discuss relevant industry expertise and market insights)",
"Body paragraph 3 (Describe experience in driving growth, managing teams, achieving results)",
"Closing paragraph (Reiterate interest, express desire to discuss contribution to the organization)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Emphasize strategic leadership experience, significant achievements with measurable results, and industry expertise. Use a confident, authoritative, and forward-looking tone."
},
"creative": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Application - [Your Name] - [Creative Role])",
"Salutation",
"Creative introduction (Engaging hook related to the role or your passion)",
"Body paragraph 1 (Highlight relevant creative experience and skills)",
"Body paragraph 2 (Reference specific portfolio pieces or projects that showcase your style/abilities)",
"Body paragraph 3 (Describe your creative process or approach)",
"Closing paragraph (Reiterate enthusiasm, mention attached resume/portfolio link, call to action)",
"Complimentary close",
"Signature"
],
"guidance": "Use a more engaging and expressive style appropriate for a creative role while maintaining professionalism. Highlight specific creative achievements and link to your portfolio."
},
"technical": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Application - [Your Name] - [Technical Role])",
"Salutation (Formal)",
"Introduction (State position, source, and brief technical interest)",
"Body paragraph 1 (Highlight specific technical skills and proficiencies relevant to the job description)",
"Body paragraph 2 (Describe relevant technical projects or challenges you've solved)",
"Body paragraph 3 (Discuss problem-solving abilities and experience with relevant technologies)",
"Closing paragraph (Reiterate interest, mention attached resume, express availability for technical discussion/interview)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Focus on technical skills, relevant projects, and problem-solving abilities. Use appropriate technical terminology accurately."
},
"academic": {
"structure": [
"Your contact information",
"Date",
"Recipient's contact information (Search Committee Chair)",
"Subject line (Application for [Position] - [Your Name])",
"Salutation (Formal)",
"Introduction (State the position, the department, and express your strong interest)",
"Body paragraph 1 (Discuss your research experience, focus on key projects and contributions)",
"Body paragraph 2 (Describe your teaching philosophy and relevant teaching experience)",
"Body paragraph 3 (Mention publications, presentations, grants, and other scholarly contributions)",
"Closing paragraph (Reiterate enthusiasm for joining the faculty, express availability for interview/presentation)",
"Complimentary close (Formal)",
"Signature (Typed name)"
],
"guidance": "Focus on research experience, teaching philosophy, publications, and contributions to the field. Use a scholarly and professional tone suitable for academia."
},
"remote": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Remote Application - [Your Name] - [Job Title])",
"Salutation",
"Introduction (State the remote position, source, and enthusiasm for remote work)",
"Body paragraph 1 (Highlight experience working remotely or independently)",
"Body paragraph 2 (Emphasize self-management, time management, and organizational skills required for remote work)",
"Body paragraph 3 (Describe strong written and verbal communication skills, essential for remote collaboration)",
"Closing paragraph (Reiterate interest in the remote role, mention attached resume, express availability for video interview)",
"Complimentary close",
"Signature"
],
"guidance": "Emphasize self-motivation, excellent communication skills (especially written), time management, and any prior experience working independently or in remote teams."
},
"referral": {
"structure": [
"Your contact information",
"Date",
"Hiring Manager contact information",
"Subject line (Referral Application - [Your Name] - [Job Title] - Referred by [Referrer's Name])",
"Salutation",
"Referral introduction (Immediately state who referred you and for what position)",
"Body paragraph 1 (Briefly explain your connection to the referrer and how you learned about the role)",
"Body paragraph 2 (Highlight key qualifications relevant to the job description)",
"Body paragraph 3 (Express strong interest in the position and the company)",
"Closing paragraph (Reiterate enthusiasm, mention attached resume, express availability for interview)",
"Complimentary close",
"Signature"
],
"guidance": "Mention the referral prominently and early. Explain your connection to the referrer and how it aligns with your interest in the role. Still, ensure you highlight your own qualifications."
},
# Default cover letter template if subtype is not found
"default": {
"structure": [
"Contact information",
"Date",
"Recipient's information",
"Subject line",
"Salutation",
"Introduction",
"Body paragraphs",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be professional, specific about your qualifications, and clear about your interest in the position. Tailor your letter to the specific job and company."
}
},
"recommendation": {
# Recommendation letters are often considered a subtype of Formal,
# but can be a top-level type in some systems. Keeping the structure
# consistent with the original request, but noting this potential overlap.
"standard": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information (e.g., Admissions Committee, Hiring Manager)",
"Subject line (Letter of Recommendation for [Name])",
"Salutation (Formal)",
"Introduction (State your name, title, relationship to the recommendee, how long you've known them, and for what opportunity the letter is written)",
"Body paragraph 1 (Describe their relevant skills and qualities, providing specific examples)",
"Body paragraph 2 (Discuss their achievements or contributions, with context and impact)",
"Body paragraph 3 (Optional: Mention character traits, teamwork, or specific anecdotes)",
"Overall Endorsement (Summarize your strong recommendation and why they are a good fit)",
"Closing paragraph (Offer to provide further information)",
"Complimentary close (Formal)",
"Signature (Typed name and title)"
],
"guidance": "Be specific, positive, and credible. Use concrete examples and anecdotes to support your recommendation. Clearly state your relationship with the person and for what opportunity you are recommending them."
},
# Default recommendation letter template if subtype is not found
"default": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Letter of Recommendation for [Name])",
"Salutation",
"Introduction",
"Body paragraphs describing qualifications and experiences",
"Specific examples and anecdotes",
"Overall endorsement and recommendation",
"Closing and offer for further information",
"Complimentary close",
"Signature"
],
"guidance": "Provide a strong, positive, and specific endorsement based on your professional or academic relationship with the individual."
}
},
"complaint": {
# Complaint letters are often considered a subtype of Formal or Business,
# but can be a top-level type. Keeping the structure consistent.
"product": {
"structure": [
"Your contact information",
"Date",
"Company contact information",
"Subject line (Complaint Regarding [Product Name/Model])",
"Salutation (Formal)",
"Introduction (State purpose: complaining about a product, include product name, model, date/place of purchase)",
"Problem description (Explain the specific defect or issue with the product in detail)",
"History of the problem (Mention if you've tried fixing it, contacted support, etc.)",
"Desired resolution (Clearly state if you want a refund, replacement, repair)",
"Call to action (State what you expect the company to do and by when)",
"Closing paragraph (Reference attached documents like receipt, express expectation for resolution)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Be clear, factual, and specific about the product issue and your desired resolution. Include all relevant details like model number, date of purchase, and copies of receipts. Maintain a firm but professional tone."
},
"service": {
"structure": [
"Your contact information",
"Date",
"Company/Service Provider contact information",
"Subject line (Complaint Regarding [Service Type/Issue])",
"Salutation (Formal)",
"Introduction (State purpose: complaining about a service received, include date/time/location of service)",
"Problem description (Explain the specific issue with the service provided in detail)",
"Impact of the issue (Explain how this problem affected you)",
"Desired resolution (Clearly state what you want: refund, re-performance of service, compensation)",
"Call to action (State what you expect the company to do and by when)",
"Closing paragraph (Reference any relevant documents, express expectation for resolution)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Be clear, factual, and specific about the service issue and your desired resolution. Include details like dates, times, and names of service providers if possible. Maintain a firm but professional tone."
},
"billing": {
"structure": [
"Your contact information",
"Date",
"Company contact information",
"Subject line (Complaint Regarding Billing Error - Account #[Your Account Number])",
"Salutation (Formal)",
"Introduction (State purpose: complaining about a billing error, include account number and invoice number)",
"Problem description (Explain the specific error on the bill: incorrect charge, double billing, etc.)",
"Provide supporting evidence (Reference payments made, attach relevant statements)",
"Desired resolution (Clearly state what you want: correction of bill, refund, credit)",
"Call to action (State what you expect the company to do and by when)",
"Closing paragraph (Reference attached documents, express expectation for resolution)",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Be clear, factual, and specific about the billing error. Provide supporting documentation like invoices or payment records. Clearly state the desired correction."
},
# Default complaint letter template if subtype is not found
"default": {
"structure": [
"Your contact information",
"Date",
"Recipient's contact information",
"Subject line (Complaint Regarding [Issue Summary])",
"Salutation",
"Introduction (State the purpose of the letter - to complain)",
"Detailed description of the problem",
"Explanation of the impact",
"Desired resolution",
"Call to action",
"Closing",
"Signature"
],
"guidance": "Be clear, factual, and specific about the issue and your desired resolution. Maintain a respectful but firm tone and provide relevant details."
}
},
"thank_you": {
# Thank You letters are often considered a subtype of Personal or Business,
# but can be a top-level type. Keeping the structure consistent.
"personal": {
"structure": [
"Greeting",
"Express gratitude clearly and sincerely",
"Specify what you are thankful for (gift, favor, support)",
"Explain the impact it had on you or how you used it",
"Share a personal thought or memory related to it (optional)",
"Look to the future or express continued appreciation",
"Closing"
],
"guidance": "Be warm, sincere, and specific about what you are thankful for. Personalize the message and explain the impact of their action or gift."
},
"professional": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Thank You - [Your Name])",
"Salutation (Formal/Semi-formal)",
"Express gratitude clearly (e.g., Thank you for the interview, thank you for your help)",
"Specify what you are thankful for (Meeting date/topic, specific assistance)",
"Reiterate interest or connection (e.g., Reiterate interest in the job, mention something discussed)",
"Express appreciation for their time or effort",
"Closing paragraph (Optional: look to future interaction)",
"Complimentary close (Formal/Semi-formal)",
"Signature"
],
"guidance": "Be prompt, professional, and specific. Reiterate your interest or key points discussed. Send within 24 hours for interviews."
},
"after_interview": {
"structure": [
"Your contact information",
"Date",
"Interviewer's contact information",
"Subject line (Thank You - [Your Name] - [Job Title])",
"Salutation (Formal)",
"Express sincere thanks for the interview opportunity",
"Mention the specific position and date of the interview",
"Reiterate your strong interest in the role and the company",
"Reference a specific point discussed during the interview to show engagement",
"Briefly highlight how your skills/experience align with a need discussed",
"Express enthusiasm for next steps",
"Complimentary close (Formal)",
"Signature"
],
"guidance": "Send within 24 hours of the interview. Be specific, professional, and reiterate your key strengths and interest. Proofread carefully."
},
# Default thank you letter template if subtype is not found
"default": {
"structure": [
"Greeting",
"Express thanks",
"Specify reason for thanks",
"Closing"
],
"guidance": "Be sincere and specific about what you are thankful for."
}
},
"invitation": {
# Invitation letters are often considered a subtype of Personal or Formal,
# but can be a top-level type. Keeping the structure consistent.
"event": { # e.g., party, gathering, wedding
"structure": [
"Greeting",
"State the purpose: extending an invitation",
"Event details (Type of event, Host)",
"Date and Time",
"Location (Full address)",
"Purpose/Theme (Optional)",
"Special instructions (Dress code, what to bring, etc. - optional)",
"RSVP information (Date, Contact method)",
"Express anticipation",
"Closing"
],
"guidance": "Be clear about all the event details (What, When, Where). Make it easy for guests to RSVP. Tone can be formal or informal depending on the event."
},
"interview": {
"structure": [
"Company Letterhead",
"Date",
"Candidate's contact information",
"Subject line (Interview Invitation - [Job Title] - [Your Name])",
"Salutation (Formal)",
"State the purpose: inviting them for an interview",
"Specify the position applied for",
"Propose date(s) and time(s) for the interview",
"Provide location details (Address, or link for virtual)",
"Mention who they will meet with (Names and titles)",
"Explain the interview format/duration (Optional)",
"Instructions (What to bring, who to contact with questions)",
"Call to action (Request confirmation or scheduling)",
"Closing paragraph (Express anticipation)",
"Complimentary close (Formal)",
"Signature (Interviewer/HR Contact Name and Title)"
],
"guidance": "Be professional, clear, and provide all necessary details for the candidate. Make the scheduling process straightforward."
},
"meeting": {
"structure": [
"Sender's contact information",
"Date",
"Recipient's contact information",
"Subject line (Invitation to Meeting - [Meeting Topic])",
"Salutation",
"State the purpose: inviting them to a meeting",
"Meeting details (Date, Time, Location/Virtual link)",
"Purpose/Agenda (Clearly state what the meeting is about)",
"Expected duration (Optional)",
"Preparation required (Optional: Documents to review)",
"RSVP information (Optional)",
"Closing paragraph",
"Complimentary close",
"Signature"
],
"guidance": "Be clear about the purpose, date, time, and location. Provide an agenda so attendees can prepare. The tone can be formal or informal depending on the context."
},
# Default invitation letter template if subtype is not found
"default": {
"structure": [
"Greeting",
"Invitation statement",
"Event/Meeting details (What, When, Where)",
"Purpose (Optional)",
"RSVP information",
"Closing"
],
"guidance": "Be clear and specific about the details of the event or meeting."
}
},
# Overall default template if letter type is not recognized
"default": {
"structure": [
"Introduction",
"Body paragraphs",
"Conclusion"
],
"guidance": "Be clear, concise, and appropriate for your audience and purpose. This is a generic structure."
}
}
def get_template_by_type(letter_type: str, subtype: str = "default") -> Dict[str, Any]:
"""
Get a template for a specific letter type and subtype using a dictionary lookup.
Args:
letter_type: Type of letter (e.g., "personal", "formal", "business", "cover").
subtype: Subtype of letter (e.g., "congratulations", "application", "sales").
Defaults to "default" if no subtype is specified.
Returns:
Template dictionary with 'structure' (List[str]) and 'guidance' (str).
Returns the default template if the letter type or subtype is not found.
"""
# Get templates for the specific letter type, or the overall default templates
type_templates = TEMPLATES.get(letter_type, TEMPLATES["default"])
# Get the template for the specific subtype, or the default for that letter type
template = type_templates.get(subtype, type_templates.get("default", TEMPLATES["default"])) # Fallback to overall default
# Ensure the returned template always has 'structure' and 'guidance' keys
# This handles cases where an incomplete template might have been defined (error tolerance)
if "structure" not in template or not isinstance(template["structure"], list):
template["structure"] = ["Introduction", "Body", "Conclusion"]
template["guidance"] = "Generic template: structure or guidance missing."
if "guidance" not in template or not isinstance(template["guidance"], str):
template["guidance"] = "Generic guidance: structure or guidance missing."
return template
# Example usage (for testing purposes)
if __name__ == '__main__':
# Test cases
print("--- Testing Letter Templates ---")
personal_congrats = get_template_by_type("personal", "congratulations")
print("\nPersonal Congratulations Template:")
print(f"Structure: {personal_congrats['structure']}")
print(f"Guidance: {personal_congrats['guidance']}")
formal_complaint = get_template_by_type("formal", "complaint")
print("\nFormal Complaint Template:")
print(f"Structure: {formal_complaint['structure']}")
print(f"Guidance: {formal_complaint['guidance']}")
business_sales = get_template_by_type("business", "sales")
print("\nBusiness Sales Template:")
print(f"Structure: {business_sales['structure']}")
print(f"Guidance: {business_sales['guidance']}")
cover_entry_level = get_template_by_type("cover", "entry_level")
print("\nCover Entry Level Template:")
print(f"Structure: {cover_entry_level['structure']}")
print(f"Guidance: {cover_entry_level['guidance']}")
unknown_type = get_template_by_type("unknown_type", "some_subtype")
print("\nUnknown Type Template (Should be Default):")
print(f"Structure: {unknown_type['structure']}")
print(f"Guidance: {unknown_type['guidance']}")
personal_unknown_subtype = get_template_by_type("personal", "unknown_subtype")
print("\nPersonal Unknown Subtype Template (Should be Personal Default):")
print(f"Structure: {personal_unknown_subtype['structure']}")
print(f"Guidance: {personal_unknown_subtype['guidance']}")

557
ToBeMigrated/ai_writers/ai_outline_writer/README.md
View File

@@ -1,557 +0,0 @@
# Blog Outline Generator
A powerful AI-powered tool for generating comprehensive blog outlines with advanced editing capabilities, content generation, and image integration.
## 🛠 Technical Architecture
### Core Components
- **Backend**: Python-based implementation using Streamlit for UI
- **AI Integration**:
- Text Generation: Integration with multiple LLM providers (Gemini, OpenAI, Anthropic)
- Image Generation: Support for multiple image generation APIs (Gemini-AI, Dalle3, Stability-AI)
- **Data Structures**:
```python
class OutlineConfig:
content_type: ContentType
content_depth: ContentDepth
outline_style: OutlineStyle
target_word_count: int
num_main_sections: int
num_subsections_per_section: int
include_images: bool
image_style: str
image_engine: str
```
### Key Technologies
- **Streamlit**: Web application framework
- **Asyncio**: Asynchronous operations for AI calls
- **Loguru**: Advanced logging system
- **BeautifulSoup**: Web content parsing
- **Pydantic**: Data validation
- **Markdown**: Content formatting
## 🌟 Features with Examples
### 1. Content Generation
- **AI-Powered Content Creation**:
```python
# Example prompt for content generation
prompt = f"""
Generate content for a {content_type} article about {topic}.
Target audience: {target_audience}
Word count: {target_word_count}
Style: {outline_style}
"""
content = await llm_text_gen(prompt)
```
- **Multiple Content Types**:
```python
# Example configuration for different content types
config = OutlineConfig(
content_type=ContentType.TUTORIAL,
content_depth=ContentDepth.INTERMEDIATE,
target_word_count=2000
)
```
### 2. Outline Structure
- **Flexible Section Management**:
```python
# Example section generation
async def generate_sections(self, topic: str) -> List[str]:
sections = []
for i in range(self.config.num_main_sections):
section = await self._generate_section(topic, i)
sections.append(section)
return sections
```
- **Optional Components**:
```python
# Example FAQ generation
async def generate_faqs(self, topic: str) -> List[str]:
prompt = f"""
Generate 5 common questions about {topic}
Content type: {self.config.content_type}
Target audience: {self.config.target_audience}
"""
return await llm_text_gen(prompt)
```
### 3. Advanced Editing Capabilities
- **Section Content Editor**:
```python
# Example content editing interface
def edit_section_content(self, section: str, content: str) -> str:
edited_content = st.text_area(
"Edit Content",
value=content,
height=300,
key=f"content_edit_{section}"
)
return edited_content
```
- **Subsection Management**:
```python
# Example subsection reordering
def reorder_subsections(self, section: str, subsections: List[str]) -> List[str]:
for i, subsection in enumerate(subsections):
if st.button("↑", key=f"move_up_{section}_{i}"):
subsections[i], subsections[i-1] = subsections[i-1], subsections[i]
return subsections
```
### 4. Image Generation
- **AI Image Generation**:
```python
# Example image generation
async def generate_image(self, prompt: str, style: str) -> str:
image_prompt = f"""
Create a {style} image for: {prompt}
Style: {self.config.image_style}
"""
return await generate_image(image_prompt)
```
### 5. Content Optimization
- **SEO Features**:
```python
# Example SEO optimization
def optimize_content(self, content: str, keywords: List[str]) -> str:
for keyword in keywords:
content = self._naturally_insert_keyword(content, keyword)
return content
```
## 📊 Technical Implementation Details
### 1. Content Generation Pipeline
```python
async def generate_content(self, topic: str) -> Dict:
# 1. Generate outline structure
outline = await self.generate_outline(topic)
# 2. Generate content for each section
for section in outline:
content = await self.generate_section_content(section)
outline[section]['content'] = content
# 3. Generate images if enabled
if self.config.include_images:
for section in outline:
image = await self.generate_section_image(section)
outline[section]['image'] = image
return outline
```
### 2. AI Integration
```python
class AIIntegration:
def __init__(self, provider: str):
self.provider = provider
self.model = self._initialize_model()
async def generate_text(self, prompt: str) -> str:
if self.provider == "gemini":
return await gemini_text_response(prompt)
elif self.provider == "openai":
return await openai_chatgpt(prompt)
```
### 3. Image Processing
```python
class ImageProcessor:
def __init__(self, engine: str):
self.engine = engine
async def generate_image(self, prompt: str) -> str:
if self.engine == "Gemini-AI":
return await generate_gemini_image(prompt)
elif self.engine == "Dalle3":
return await generate_dalle3_images(prompt)
```
## 🔧 Configuration Examples
### 1. Basic Configuration
```python
config = OutlineConfig(
content_type=ContentType.GUIDE,
content_depth=ContentDepth.INTERMEDIATE,
target_word_count=2000,
num_main_sections=5,
num_subsections_per_section=3
)
```
### 2. Advanced Configuration
```python
config = OutlineConfig(
content_type=ContentType.TUTORIAL,
content_depth=ContentDepth.ADVANCED,
outline_style=OutlineStyle.MODERN,
target_word_count=3000,
include_images=True,
image_style="realistic",
image_engine="Gemini-AI",
target_audience="developers",
language="English",
keywords=["python", "tutorial", "advanced"]
)
```
## 📝 Usage Examples
### 1. Basic Usage
```python
# Initialize generator
generator = BlogOutlineGenerator()
# Generate outline
outline = await generator.generate_outline("Python Programming Basics")
# Export to markdown
markdown = generator.to_markdown()
```
### 2. Advanced Usage
```python
# Custom configuration
config = OutlineConfig(
content_type=ContentType.TUTORIAL,
content_depth=ContentDepth.ADVANCED,
include_images=True
)
# Initialize with config
generator = BlogOutlineGenerator(config)
# Generate with custom settings
outline = await generator.generate_outline(
"Advanced Python Decorators",
keywords=["python", "decorators", "advanced"]
)
# Export to multiple formats
markdown = generator.to_markdown()
json_output = generator.to_json()
html_output = generator.to_html()
```
## 🔍 Technical Considerations
### 1. Performance Optimization
- Asynchronous operations for AI calls
- Caching of generated content
- Batch processing for images
- Memory management for large documents
### 2. Error Handling
```python
try:
content = await llm_text_gen(prompt)
except Exception as e:
logger.error(f"Content generation failed: {e}")
return None
```
### 3. Data Validation
```python
from pydantic import BaseModel, validator
class SectionContent(BaseModel):
title: str
content: str
image_path: Optional[str]
@validator('content')
def validate_content_length(cls, v):
if len(v.split()) < 100:
raise ValueError("Content too short")
return v
```
## 🌟 Features
### 1. Content Generation
- **AI-Powered Content Creation**: Generate high-quality content for each section using advanced language models
- **Multiple Content Types**: Support for various content formats including:
- How-to guides
- Tutorials
- Listicles
- Comparisons
- Case studies
- Opinion pieces
- News articles
- Reviews
- General guides
- **Customizable Content Depth**:
- Basic: Simple, easy-to-understand content
- Intermediate: Balanced depth with practical examples
- Advanced: Detailed technical content
- Expert: In-depth analysis and advanced concepts
### 2. Outline Structure
- **Flexible Section Management**:
- Customizable number of main sections
- Configurable subsections per section
- Dynamic section reordering
- Easy addition/removal of sections
- **Optional Components**:
- Introduction section
- Conclusion section
- FAQ section
- Additional resources section
### 3. Advanced Editing Capabilities
- **Section Content Editor**:
- Rich text editing interface
- Real-time word count tracking
- Formatting options (Bold, Italic, Lists, Code Blocks, Links)
- AI-powered content enhancement
- **Subsection Management**:
- Drag-and-drop reordering
- Individual subsection editing
- Add/remove subsection functionality
- Bulk editing capabilities
- **Metadata Editing**:
- Section-specific settings
- Content depth adjustment
- Target word count configuration
- Image settings customization
### 4. Image Generation
- **AI Image Generation**:
- Multiple image styles (realistic, illustration, minimalist, photographic, artistic)
- Support for multiple image engines (Gemini-AI, Dalle3, Stability-AI)
- Custom image prompts
- Image regeneration capability
- **Image Integration**:
- Automatic image placement
- Image preview and editing
- Image prompt viewing and editing
- Image style customization
### 5. Content Optimization
- **SEO Features**:
- Keyword integration
- Content structure optimization
- Meta description generation
- SEO-friendly formatting
- **Audience Targeting**:
- Customizable target audience
- Language selection
- Content tone adjustment
- Reading level optimization
### 6. Export Options
- **Multiple Formats**:
- Markdown export
- JSON export
- HTML export
- Custom formatting options
- **Download Capabilities**:
- One-click download
- Format-specific styling
- Custom file naming
- Batch export options
### 7. User Interface
- **Intuitive Design**:
- Clean, modern interface
- Responsive layout
- Easy navigation
- Clear visual hierarchy
- **Interactive Features**:
- Real-time preview
- Drag-and-drop functionality
- Quick edit options
- Contextual help
### 8. Statistics and Analytics
- **Content Metrics**:
- Word count tracking
- Section statistics
- Subsection counts
- Content depth analysis
- **Progress Tracking**:
- Generation progress
- Edit history
- Version comparison
- Performance metrics
## 🚀 Getting Started
### Installation
```bash
pip install -r requirements.txt
```
### Usage
1. Launch the application:
```bash
streamlit run lib/ai_writers/ai_outline_writer/outline_ui.py
```
2. Configure your outline:
- Enter your blog topic
- Select content type and depth
- Choose outline style
- Set target word count
- Configure sections and subsections
3. Generate and edit:
- Click "Generate Outline"
- Review and edit sections
- Customize content and images
- Export in your preferred format
## 🔧 Configuration Options
### Basic Settings
- **Blog Topic**: Main subject of your content
- **Content Type**: Type of content to generate
- **Content Depth**: Level of detail and complexity
- **Outline Style**: Structure and formatting style
### Advanced Settings
- **Target Word Count**: Desired length of the content
- **Number of Sections**: Customize main sections
- **Subsections**: Configure subsections per section
- **Image Settings**: Customize image generation
- **Target Audience**: Define your audience
- **Language**: Select content language
- **Keywords**: Add SEO keywords
- **Excluded Topics**: Specify topics to avoid
## 📊 Output Formats
### 1. Preview Mode
- Interactive preview of the entire outline
- Real-time editing capabilities
- Image preview and management
- Content statistics
### 2. Markdown Export
- Clean markdown formatting
- Proper heading hierarchy
- Image embedding
- Code block formatting
### 3. JSON Export
- Structured data format
- Complete outline information
- Content and image metadata
- Configuration details
### 4. HTML Export
- Styled HTML output
- Responsive design
- Image integration
- Custom CSS support
## 💡 Best Practices
### Content Generation
1. Start with a clear topic and target audience
2. Choose appropriate content type and depth
3. Use relevant keywords for SEO
4. Review and edit generated content
5. Add personal insights and examples
### Outline Structure
1. Maintain logical flow between sections
2. Balance section lengths
3. Include relevant subsections
4. Add appropriate transitions
5. Ensure comprehensive coverage
### Image Usage
1. Choose appropriate image styles
2. Generate relevant images
3. Optimize image placement
4. Review image prompts
5. Consider image licensing
## 🔄 Workflow
1. **Initial Setup**
- Configure basic settings
- Set content parameters
- Define target audience
2. **Generation**
- Generate initial outline
- Review structure
- Generate content
- Create images
3. **Editing**
- Review and edit content
- Adjust structure
- Customize images
- Optimize for SEO
4. **Export**
- Choose export format
- Review final output
- Download content
- Save configuration
## 📝 Tips and Tricks
### Content Generation
- Use specific keywords for better results
- Provide clear context for the AI
- Review and refine generated content
- Add personal expertise
### Structure Optimization
- Maintain consistent section lengths
- Use clear subsection hierarchies
- Include relevant examples
- Add practical applications
### Image Enhancement
- Use descriptive image prompts
- Experiment with different styles
- Consider image placement
- Review image relevance
## 🤝 Contributing
We welcome contributions! Please follow these steps:
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Submit a pull request
## 📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
## 📞 Support
For support, please:
1. Check the documentation
2. Review existing issues
3. Create a new issue if needed
4. Contact the maintainers
## 🔮 Future Enhancements
Planned features:
- Multi-language support
- Advanced AI models
- More export formats
- Enhanced editing tools
- Collaboration features
- Version control integration
- Analytics dashboard
- Custom templates
- API integration
- Mobile optimization

317
ToBeMigrated/ai_writers/ai_outline_writer/get_blog_outline.py
View File

@@ -1,317 +0,0 @@
"""
Enhanced Blog Outline Generator
This module provides a sophisticated outline generation system that creates detailed,
well-structured outlines for blog posts based on user preferences and content requirements.
"""
import sys
from typing import Dict, List, Optional
from enum import Enum
from dataclasses import dataclass
from loguru import logger
import json
from lib.gpt_providers.text_generation.main_text_generation import llm_text_gen
from lib.gpt_providers.text_to_image_generation.main_generate_image_from_prompt import generate_image
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}")
class ContentType(Enum):
"""Types of content that can be generated."""
HOW_TO = "how-to"
TUTORIAL = "tutorial"
LISTICLE = "listicle"
COMPARISON = "comparison"
CASE_STUDY = "case-study"
OPINION = "opinion"
NEWS = "news"
REVIEW = "review"
GUIDE = "guide"
class ContentDepth(Enum):
"""Depth levels for content coverage."""
BASIC = "basic"
INTERMEDIATE = "intermediate"
ADVANCED = "advanced"
EXPERT = "expert"
class OutlineStyle(Enum):
"""Styles for outline structure."""
TRADITIONAL = "traditional"
MODERN = "modern"
CONVERSATIONAL = "conversational"
ACADEMIC = "academic"
SEO_OPTIMIZED = "seo-optimized"
@dataclass
class OutlineConfig:
"""Configuration for outline generation."""
content_type: ContentType = ContentType.GUIDE
content_depth: ContentDepth = ContentDepth.INTERMEDIATE
outline_style: OutlineStyle = OutlineStyle.MODERN
target_word_count: int = 2000
num_main_sections: int = 5
num_subsections_per_section: int = 3
include_introduction: bool = True
include_conclusion: bool = True
include_faqs: bool = True
include_resources: bool = True
target_audience: str = "general"
language: str = "English"
keywords: List[str] = None
exclude_topics: List[str] = None
include_images: bool = True
image_style: str = "realistic"
image_engine: str = "Gemini-AI"
@dataclass
class SectionContent:
"""Content for a section including text and image."""
title: str
content: str
image_prompt: Optional[str] = None
image_path: Optional[str] = None
class BlogOutlineGenerator:
"""Enhanced blog outline generator with comprehensive controls."""
def __init__(self, config: Optional[OutlineConfig] = None):
"""Initialize the outline generator with optional configuration."""
self.config = config or OutlineConfig()
self.outline = {}
self.section_contents = {}
def generate_outline(self, topic: str) -> Dict[str, List[str]]:
"""Generate a blog outline based on the topic and configuration."""
try:
# Create a focused prompt for outline generation
prompt = f"""Generate a blog outline for topic: {topic}
Content Type: {self.config.content_type.value}
Target Audience: {self.config.target_audience}
Content Depth: {self.config.content_depth.value}
Style: {self.config.outline_style.value}
Word Count Target: {self.config.target_word_count}
Main Sections: {self.config.num_main_sections}
Subsections per Section: {self.config.num_subsections_per_section}
Requirements:
- Create exactly {self.config.num_main_sections} main sections
- Each section should have exactly {self.config.num_subsections_per_section} subsections
- Focus on {self.config.content_type.value} content style
- Target {self.config.target_audience} audience
- Maintain {self.config.content_depth.value} depth
- Follow {self.config.outline_style.value} style
- Optimize for {self.config.target_word_count} words total
IMPORTANT: You must return a valid JSON object with main sections as keys and lists of subsections as values.
Example format: {{"Section 1": ["Subsection 1.1", "Subsection 1.2"], "Section 2": ["Subsection 2.1", "Subsection 2.2"]}}
Do not include any additional text or explanations, only the JSON object."""
# Get outline from LLM
outline_json = llm_text_gen(prompt)
# Clean the response to ensure it's valid JSON
outline_json = outline_json.strip()
if not outline_json.startswith('{'):
outline_json = outline_json[outline_json.find('{'):]
if not outline_json.endswith('}'):
outline_json = outline_json[:outline_json.rfind('}')+1]
# Parse the outline
try:
outline = json.loads(outline_json)
except json.JSONDecodeError as e:
logger.error(f"JSON parsing error: {str(e)}")
logger.error(f"Raw response: {outline_json}")
# Fallback to a basic outline structure
outline = {
f"Section {i+1}": [f"Subsection {i+1}.{j+1}" for j in range(self.config.num_subsections_per_section)]
for i in range(self.config.num_main_sections)
}
# Add introduction and conclusion if configured
if self.config.include_introduction:
outline = {"Introduction": ["Overview", "Importance", "What to Expect"]} | outline
if self.config.include_conclusion:
outline["Conclusion"] = ["Summary", "Key Takeaways", "Next Steps"]
# Add FAQs if configured
if self.config.include_faqs:
# Generate topic-specific FAQs
faq_prompt = f"""Generate 3 specific and relevant FAQ questions for a blog post about: {topic}
Content Type: {self.config.content_type.value}
Target Audience: {self.config.target_audience}
Content Depth: {self.config.content_depth.value}
Requirements:
- Questions should be specific to the topic
- Cover common concerns and important aspects
- Be relevant to the target audience
- Include both basic and advanced questions
Format: Return only a JSON array of 3 questions.
Example format: ["Question 1?", "Question 2?", "Question 3?"]"""
try:
faq_json = llm_text_gen(faq_prompt)
faq_json = faq_json.strip()
if not faq_json.startswith('['):
faq_json = faq_json[faq_json.find('['):]
if not faq_json.endswith(']'):
faq_json = faq_json[:faq_json.rfind(']')+1]
faqs = json.loads(faq_json)
outline["Frequently Asked Questions"] = faqs
except Exception as e:
logger.error(f"Error generating FAQs: {str(e)}")
outline["Frequently Asked Questions"] = [
f"Common Question about {topic} 1",
f"Common Question about {topic} 2",
f"Common Question about {topic} 3"
]
# Add resources if configured
if self.config.include_resources:
outline["Additional Resources"] = [
"Further Reading",
"Tools and References",
"Related Topics"
]
return outline
except Exception as e:
logger.error(f"Error generating outline: {str(e)}")
return {}
def generate_section_content(self, section: str, subsections: List[str]) -> Optional[SectionContent]:
"""Generate content for a section."""
try:
# Create a focused prompt for content generation
prompt = f"""Generate content for section: {section}
Subsections: {', '.join(subsections)}
Content Type: {self.config.content_type.value}
Target Audience: {self.config.target_audience}
Content Depth: {self.config.content_depth.value}
Style: {self.config.outline_style.value}
Word Count Target: {self.config.target_word_count // self.config.num_main_sections}
Requirements:
- Write content for each subsection
- Maintain {self.config.content_depth.value} depth
- Target {self.config.target_audience} audience
- Follow {self.config.outline_style.value} style
- Optimize for {self.config.target_word_count // self.config.num_main_sections} words
- Include relevant examples and data points
- Use clear, engaging language
Format: Return only a JSON object with 'content' and 'image_prompt' fields.
Example format: {{"content": "Section content here...", "image_prompt": "Image description here..."}}"""
# Get content from LLM
content_json = llm_text_gen(prompt)
content_data = json.loads(content_json)
# Generate image if configured
image_path = None
if self.config.include_images:
image_path = self.generate_section_image(section)
return SectionContent(
title=section,
content=content_data["content"],
image_prompt=content_data.get("image_prompt"),
image_path=image_path
)
except Exception as e:
logger.error(f"Error generating content for section {section}: {str(e)}")
return None
def generate_section_image(self, section: str) -> Optional[str]:
"""Generate an image for a section."""
try:
# Create a focused prompt for image generation
prompt = f"""Generate an image prompt for section: {section}
Style: {self.config.image_style}
Engine: {self.config.image_engine}
Content Type: {self.config.content_type.value}
Target Audience: {self.config.target_audience}
Requirements:
- Create a {self.config.image_style} style image
- Optimize for {self.config.image_engine} engine
- Match {self.config.content_type.value} content type
- Appeal to {self.config.target_audience} audience
- Be visually engaging and relevant
Format: Return only a JSON object with an 'image_prompt' field.
Example format: {{"image_prompt": "Detailed image description here..."}}"""
# Get image prompt from LLM
prompt_json = llm_text_gen(prompt)
prompt_data = json.loads(prompt_json)
# Generate image using the specified engine
if self.config.image_engine == "Gemini-AI":
image_path = generate_gemini_image(prompt_data["image_prompt"])
elif self.config.image_engine == "Dalle3":
image_path = generate_dalle_image(prompt_data["image_prompt"])
else: # Stability-AI
image_path = generate_stability_image(prompt_data["image_prompt"])
return image_path
except Exception as e:
logger.error(f"Error generating image for section {section}: {str(e)}")
return None
def to_markdown(self) -> str:
"""Convert outline to markdown format with content and images."""
markdown = f"# {self.outline.get('Introduction', [''])[0]}\n\n"
for section, subsections in self.outline.items():
if section not in ["Introduction", "Conclusion", "FAQs", "Additional Resources"]:
markdown += f"## {section}\n\n"
# Add section content if available
if section in self.section_contents:
content = self.section_contents[section]
markdown += f"{content.content}\n\n"
# Add image if available
if content.image_path:
markdown += f"![{section}]({content.image_path})\n\n"
# Add subsections
for subsection in subsections:
markdown += f"- {subsection}\n"
markdown += "\n"
if "Conclusion" in self.outline:
markdown += "## Conclusion\n\n"
for subsection in self.outline["Conclusion"]:
markdown += f"- {subsection}\n"
markdown += "\n"
if "FAQs" in self.outline:
markdown += "## Frequently Asked Questions\n\n"
for faq in self.outline["FAQs"]:
markdown += f"- {faq}\n"
markdown += "\n"
if "Additional Resources" in self.outline:
markdown += "## Additional Resources\n\n"
for resource in self.outline["Additional Resources"]:
markdown += f"- {resource}\n"
return markdown

739
ToBeMigrated/ai_writers/ai_outline_writer/outline_ui.py
View File

@@ -1,739 +0,0 @@
"""
Streamlit UI for Enhanced Blog Outline Generator
This module provides a user-friendly interface for generating comprehensive blog outlines
with AI-powered content and image generation capabilities.
"""
import streamlit as st
import asyncio
from pathlib import Path
from typing import Optional, Dict, List
import json
import time
from datetime import datetime
from .get_blog_outline import (
BlogOutlineGenerator,
OutlineConfig,
ContentType,
ContentDepth,
OutlineStyle
)
# Custom CSS for better styling
st.markdown("""
<style>
.main {
background-color: #f5f5f5;
}
.stButton>button {
background-color: #4CAF50;
color: white;
padding: 10px 24px;
border-radius: 4px;
border: none;
font-weight: bold;
width: 100%;
}
.stButton>button:hover {
background-color: #45a049;
}
/* Add specific styling for the generate outline button */
.generate-outline-button {
width: 100%;
margin: 20px 0;
}
.generate-outline-button > button {
width: 100%;
height: 50px;
font-size: 1.2rem;
}
.section-card {
background-color: white;
padding: 20px;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
margin-bottom: 20px;
width: 100%;
}
.content-preview {
background-color: #f8f9fa;
padding: 15px;
border-radius: 4px;
margin: 10px 0;
width: 100%;
}
.image-container {
display: flex;
justify-content: center;
margin: 20px 0;
width: 100%;
}
.stats-card {
background-color: #e8f5e9;
padding: 15px;
border-radius: 8px;
margin: 10px 0;
width: 100%;
}
.edit-section {
background-color: #e3f2fd;
padding: 15px;
border-radius: 4px;
margin: 10px 0;
width: 100%;
}
.subsection-list {
margin-left: 20px;
width: 100%;
}
/* Main container width */
.main .block-container {
max-width: 100%;
padding: 2rem;
}
/* Full width for the outline display */
.outline-container {
width: 100%;
max-width: 100%;
margin: 0 auto;
padding: 20px;
}
/* Section styling */
.section-header {
font-size: 1.5rem;
font-weight: bold;
color: #2c3e50;
margin-bottom: 1rem;
padding-bottom: 0.5rem;
border-bottom: 2px solid #e0e0e0;
}
.subsection-item {
font-size: 1.1rem;
color: #34495e;
margin: 0.5rem 0;
padding-left: 1rem;
}
/* Content area styling */
.content-area {
background-color: white;
padding: 2rem;
border-radius: 8px;
box-shadow: 0 2px 4px rgba(0,0,0,0.1);
margin: 1rem 0;
}
/* Make sure all Streamlit elements use full width */
.stMarkdown, .stText, .stTextArea, .stSelectbox, .stSlider {
width: 100% !important;
}
/* Full width for code blocks */
.stCodeBlock {
width: 100% !important;
}
/* Full width for the main content */
.main .block-container {
padding-left: 2rem;
padding-right: 2rem;
max-width: 100%;
}
/* Adjust the main content area */
.main .block-container > div {
max-width: 100%;
}
/* Make sure the outline content uses full width */
.outline-content {
width: 100%;
max-width: 100%;
margin: 0;
padding: 0;
}
/* Adjust the preview section */
.preview-section {
width: 100%;
max-width: 100%;
margin: 0;
padding: 1rem;
}
</style>
""", unsafe_allow_html=True)
def edit_section_content(section: str, content: str) -> str:
"""Edit section content with advanced options."""
st.markdown('<div class="edit-section">', unsafe_allow_html=True)
# Content editing
edited_content = st.text_area(
"Edit Content",
value=content,
height=300,
key=f"content_edit_{section}"
)
# Word count and formatting
col1, col2 = st.columns(2)
with col1:
word_count = len(edited_content.split())
st.info(f"Word Count: {word_count}")
with col2:
formatting = st.multiselect(
"Formatting Options",
["Bold", "Italic", "Lists", "Code Blocks", "Links"],
key=f"format_{section}"
)
# AI enhancement options
with st.expander("AI Enhancement Options"):
enhance_options = st.multiselect(
"Select Enhancements",
["Improve Clarity", "Add Examples", "Expand Details", "Add Statistics", "Improve SEO"],
key=f"enhance_{section}"
)
if st.button("Apply Enhancements", key=f"apply_enhance_{section}"):
with st.spinner("Applying enhancements..."):
# TODO: Implement AI enhancement logic
st.success("Enhancements applied!")
st.markdown('</div>', unsafe_allow_html=True)
return edited_content
def edit_subsections(section: str, subsections: List[str]) -> List[str]:
"""Edit subsections with reordering and editing capabilities."""
st.markdown('<div class="edit-section">', unsafe_allow_html=True)
# Reorder subsections
st.markdown("### Reorder Subsections")
for i, subsection in enumerate(subsections):
col1, col2 = st.columns([4, 1])
with col1:
subsections[i] = st.text_input(
f"Subsection {i+1}",
value=subsection,
key=f"subsection_{section}_{i}"
)
with col2:
if st.button("↑", key=f"move_up_{section}_{i}") and i > 0:
subsections[i], subsections[i-1] = subsections[i-1], subsections[i]
st.experimental_rerun()
if st.button("↓", key=f"move_down_{section}_{i}") and i < len(subsections)-1:
subsections[i], subsections[i+1] = subsections[i+1], subsections[i]
st.experimental_rerun()
# Add/remove subsections
col1, col2 = st.columns(2)
with col1:
if st.button("Add Subsection", key=f"add_sub_{section}"):
subsections.append("New Subsection")
st.experimental_rerun()
with col2:
if st.button("Remove Last Subsection", key=f"remove_sub_{section}"):
if subsections:
subsections.pop()
st.experimental_rerun()
st.markdown('</div>', unsafe_allow_html=True)
return subsections
def edit_section_metadata(section: str, generator: BlogOutlineGenerator):
"""Edit section metadata and settings."""
st.markdown('<div class="edit-section">', unsafe_allow_html=True)
# Section settings
st.markdown("### Section Settings")
# Image settings
if generator.config.include_images:
col1, col2 = st.columns(2)
with col1:
new_image_style = st.selectbox(
"Image Style",
["realistic", "illustration", "minimalist", "photographic", "artistic"],
key=f"img_style_{section}"
)
with col2:
new_image_engine = st.selectbox(
"Image Engine",
["Gemini-AI", "Dalle3", "Stability-AI"],
key=f"img_engine_{section}"
)
if st.button("Regenerate Image", key=f"regen_img_{section}"):
with st.spinner("Regenerating image..."):
# TODO: Implement image regeneration logic
st.success("Image regenerated!")
# Content settings
st.markdown("### Content Settings")
col1, col2 = st.columns(2)
with col1:
target_word_count = st.number_input(
"Target Word Count",
min_value=100,
max_value=2000,
value=500,
step=100,
key=f"word_count_{section}"
)
with col2:
content_depth = st.selectbox(
"Content Depth",
[depth.value for depth in ContentDepth],
key=f"depth_{section}"
)
st.markdown('</div>', unsafe_allow_html=True)
def display_section(section: str, subsections: List[str], content: Optional[Dict] = None, generator: Optional[BlogOutlineGenerator] = None):
"""Display a section with its content and subsections."""
st.markdown(f"""
<div class="section-card">
<div class="section-header">{section}</div>
""", unsafe_allow_html=True)
# Section editing controls
col1, col2 = st.columns([4, 1])
with col1:
st.markdown(f"### {section}")
with col2:
edit_mode = st.checkbox("Edit Mode", key=f"edit_mode_{section}")
if content:
# Display content with word count
word_count = len(content.content.split())
st.markdown(f"""
<div class="content-preview">
<p><strong>Content Preview</strong> ({word_count} words)</p>
{content.content[:500]}...
</div>
""", unsafe_allow_html=True)
# Image generation and display - Always show if images are enabled
if generator and generator.config.include_images:
st.markdown("### Image Generation")
col1, col2, col3 = st.columns([2, 2, 1])
with col1:
image_style = st.selectbox(
"Image Style",
["realistic", "illustration", "minimalist", "photographic", "artistic"],
index=["realistic", "illustration", "minimalist", "photographic", "artistic"].index(generator.config.image_style),
key=f"img_style_{section}"
)
with col2:
image_engine = st.selectbox(
"Image Engine",
["Gemini-AI", "Dalle3", "Stability-AI"],
index=["Gemini-AI", "Dalle3", "Stability-AI"].index(generator.config.image_engine),
key=f"img_engine_{section}"
)
with col3:
if st.button("Generate Image", key=f"gen_img_{section}"):
with st.spinner(f"Generating image for {section}..."):
# Update config with selected options
generator.config.image_style = image_style
generator.config.image_engine = image_engine
image_path = generator.generate_section_image(section)
if image_path:
st.success("Image generated successfully!")
st.experimental_rerun()
else:
st.error("Failed to generate image")
# Display existing image if available
if content.image_path:
st.markdown('<div class="image-container">', unsafe_allow_html=True)
st.image(content.image_path, caption=section, use_column_width=True)
st.markdown('</div>', unsafe_allow_html=True)
# Display image prompt in expander
if content.image_prompt:
with st.expander("View Image Prompt"):
st.code(content.image_prompt, language="text")
# Edit mode controls
if edit_mode:
st.markdown("### Edit Content")
# Edit content
edited_content = edit_section_content(section, content.content)
if edited_content != content.content:
content.content = edited_content
st.experimental_rerun()
st.markdown("### Edit Subsections")
# Edit subsections
edited_subsections = edit_subsections(section, subsections)
if edited_subsections != subsections:
subsections[:] = edited_subsections
st.experimental_rerun()
st.markdown("### Edit Metadata")
# Edit metadata
if generator:
edit_section_metadata(section, generator)
else:
# Display subsections in view mode
st.markdown("### Subsections")
st.markdown('<div class="subsection-list">', unsafe_allow_html=True)
for subsection in subsections:
st.markdown(f'<div class="subsection-item">• {subsection}</div>', unsafe_allow_html=True)
st.markdown('</div>', unsafe_allow_html=True)
st.markdown("</div>", unsafe_allow_html=True)
def display_stats(generator, outline):
"""Display statistics about the generated outline."""
total_sections = len(outline)
total_subsections = sum(len(subsections) for subsections in outline.values())
total_content = sum(len(content.content.split()) for content in generator.section_contents.values())
col1, col2, col3 = st.columns(3)
with col1:
st.markdown(f"""
<div class="stats-card">
<h3>📊 Statistics</h3>
<p>Total Sections: {total_sections}</p>
<p>Total Subsections: {total_subsections}</p>
<p>Estimated Word Count: {total_content}</p>
</div>
""", unsafe_allow_html=True)
with col2:
st.markdown(f"""
<div class="stats-card">
<h3>🎯 Target</h3>
<p>Target Word Count: {generator.config.target_word_count}</p>
<p>Content Depth: {generator.config.content_depth.value}</p>
<p>Style: {generator.config.outline_style.value}</p>
</div>
""", unsafe_allow_html=True)
with col3:
st.markdown(f"""
<div class="stats-card">
<h3>📝 Content Type</h3>
<p>Type: {generator.config.content_type.value}</p>
<p>Audience: {generator.config.target_audience}</p>
<p>Language: {generator.config.language}</p>
</div>
""", unsafe_allow_html=True)
def main():
# Header with description
st.title("Blog Outline Generator")
st.markdown("""
Generate comprehensive blog outlines with AI-powered content and images.
Customize your outline with various options and get detailed content for each section.
""")
# Main content area with full width
st.markdown('<div class="outline-container">', unsafe_allow_html=True)
# Move topic input to main area and make it more prominent
st.markdown("### Enter Your Blog Topic")
topic = st.text_input("", placeholder="Enter your blog topic here for creating outline...", key="blog_topic")
st.markdown("---") # Add a separator
st.markdown("### Configuration Options")
# Create tabs for different configuration sections
tab1, tab2, tab3, tab4 = st.tabs([
"📝 Content Type & Target",
"📊 Content Structure",
"🎨 Style & Sections",
"🖼️ Image & Optimization"
])
with tab1:
st.markdown("#### Content Type & Target")
col1, col2, col3 = st.columns(3)
with col1:
content_type = st.selectbox(
"Content Type",
[type.value for type in ContentType],
index=[type.value for type in ContentType].index(ContentType.GUIDE.value),
help="Select the type of content you want to generate"
)
with col2:
target_audience = st.selectbox(
"Target Audience",
["General", "Technical", "Professional", "Academic", "Business", "Students", "Developers"],
index=0,
help="Select your target audience"
)
with col3:
language = st.selectbox(
"Language",
["English", "Spanish", "French", "German", "Italian", "Portuguese", "Chinese", "Japanese", "Korean"],
index=0,
help="Select the language for your content"
)
with tab2:
st.markdown("#### Content Structure")
col1, col2 = st.columns(2)
with col1:
num_main_sections = st.slider(
"Number of Main Sections",
min_value=3,
max_value=10,
value=5,
step=1,
help="Choose how many main sections your outline should have"
)
num_subsections = st.slider(
"Subsections per Section",
min_value=2,
max_value=5,
value=3,
step=1,
help="Choose how many subsections each main section should have"
)
with col2:
target_word_count = st.slider(
"Target Word Count",
min_value=500,
max_value=5000,
value=2000,
step=100,
help="Set your target word count for the entire blog post"
)
# Display content statistics
st.markdown("##### Content Statistics")
st.markdown(f"""
- Estimated Sections: {num_main_sections}
- Total Subsections: {num_main_sections * num_subsections}
- Target Word Count: {target_word_count}
- Average Words per Section: {target_word_count // num_main_sections}
""")
with tab3:
st.markdown("#### Style & Sections")
col1, col2 = st.columns(2)
with col1:
content_depth = st.selectbox(
"Content Depth",
[depth.value for depth in ContentDepth],
index=[depth.value for depth in ContentDepth].index(ContentDepth.INTERMEDIATE.value),
help="Select the depth of content coverage"
)
outline_style = st.selectbox(
"Outline Style",
[style.value for style in OutlineStyle],
index=[style.value for style in OutlineStyle].index(OutlineStyle.MODERN.value),
help="Select the style of your outline"
)
with col2:
st.markdown("##### Additional Sections")
include_intro = st.checkbox("Include Introduction", value=True, help="Add an introduction section")
include_conclusion = st.checkbox("Include Conclusion", value=True, help="Add a conclusion section")
include_faqs = st.checkbox("Include FAQs", value=True, help="Add a FAQ section")
include_resources = st.checkbox("Include Resources", value=True, help="Add a resources section")
with tab4:
st.markdown("#### Image & Optimization")
col1, col2 = st.columns(2)
with col1:
st.markdown("##### Image Settings")
include_images = st.checkbox("Enable Image Generation", value=True, help="Enable AI image generation for sections")
if include_images:
image_style = st.selectbox(
"Image Style",
["realistic", "illustration", "minimalist", "photographic", "artistic"],
index=0,
help="Select the style for generated images"
)
image_engine = st.selectbox(
"Image Engine",
["Gemini-AI", "Dalle3", "Stability-AI"],
index=0,
help="Select the AI engine for image generation"
)
with col2:
st.markdown("##### Content Optimization")
keywords = st.text_area(
"Keywords (comma-separated)",
help="Enter keywords for SEO optimization, separated by commas"
)
exclude_topics = st.text_area(
"Topics to Exclude (comma-separated)",
help="Enter topics you want to exclude from the content"
)
st.markdown("---") # Add a separator before the generate button
# Create configuration
config = OutlineConfig(
content_type=ContentType(content_type),
content_depth=ContentDepth(content_depth),
outline_style=OutlineStyle(outline_style),
target_word_count=target_word_count,
num_main_sections=num_main_sections,
num_subsections_per_section=num_subsections,
include_introduction=include_intro,
include_conclusion=include_conclusion,
include_faqs=include_faqs,
include_resources=include_resources,
include_images=include_images,
image_style=image_style if include_images else "realistic",
image_engine=image_engine if include_images else "Gemini-AI",
target_audience=target_audience,
language=language,
keywords=[k.strip() for k in keywords.split(',')] if keywords else None,
exclude_topics=[t.strip() for t in exclude_topics.split(',')] if exclude_topics else None
)
# Initialize generator
generator = BlogOutlineGenerator(config)
# Store the generated outline in session state
if 'outline' not in st.session_state:
st.session_state.outline = None
if 'section_contents' not in st.session_state:
st.session_state.section_contents = {}
# Generate outline button with full width
st.markdown('<div class="generate-outline-button">', unsafe_allow_html=True)
if not topic:
st.warning("Please enter a blog topic to generate the outline.")
if st.button("Generate Outline", type="primary", use_container_width=True, disabled=not topic):
with st.spinner("Generating outline and content..."):
try:
# Add progress bar
progress_bar = st.progress(0)
for i in range(100):
time.sleep(0.01)
progress_bar.progress(i + 1)
outline = generator.generate_outline(topic)
st.session_state.outline = outline
st.session_state.section_contents = generator.section_contents
# Display results
st.success("Outline generated successfully!")
# Add copy button and display outline in full width
st.markdown('<div class="outline-content">', unsafe_allow_html=True)
outline_text = json.dumps(outline, indent=2)
st.code(outline_text, language="json")
st.button("Copy Outline", key="copy_outline",
help="Copy the outline to clipboard",
on_click=lambda: st.write(f'<script>navigator.clipboard.writeText(`{outline_text}`)</script>',
unsafe_allow_html=True))
st.markdown('</div>', unsafe_allow_html=True)
# Display statistics
display_stats(generator, outline)
# Output format selection
output_format = st.radio(
"Output Format",
["Preview", "Markdown", "JSON", "HTML"]
)
if output_format == "Preview":
# Display outline with content and images
st.markdown('<div class="preview-section">', unsafe_allow_html=True)
for section, subsections in outline.items():
content = generator.section_contents.get(section)
display_section(section, subsections, content, generator)
st.markdown('</div>', unsafe_allow_html=True)
elif output_format == "Markdown":
markdown_output = generator.to_markdown()
st.markdown('<div class="outline-content">', unsafe_allow_html=True)
st.code(markdown_output, language="markdown")
st.download_button(
"Download Markdown",
markdown_output,
file_name="blog_outline.md",
mime="text/markdown"
)
st.markdown('</div>', unsafe_allow_html=True)
elif output_format == "JSON":
json_output = json.dumps({
"outline": outline,
"contents": {
section: {
"title": content.title,
"content": content.content,
"image_prompt": content.image_prompt,
"image_path": content.image_path
}
for section, content in generator.section_contents.items()
}
}, indent=2)
st.markdown('<div class="outline-content">', unsafe_allow_html=True)
st.code(json_output, language="json")
st.download_button(
"Download JSON",
json_output,
file_name="blog_outline.json",
mime="application/json"
)
st.markdown('</div>', unsafe_allow_html=True)
elif output_format == "HTML":
html_output = f"""
<!DOCTYPE html>
<html>
<head>
<title>{topic} - Blog Outline</title>
<style>
body {{ font-family: Arial, sans-serif; max-width: 100%; margin: 0 auto; padding: 20px; }}
.section {{ margin-bottom: 30px; }}
.content {{ background: #f8f9fa; padding: 15px; border-radius: 4px; }}
img {{ max-width: 100%; height: auto; }}
</style>
</head>
<body>
<h1>{topic}</h1>
{generator.to_markdown().replace('#', '##')}
</body>
</html>
"""
st.markdown('<div class="outline-content">', unsafe_allow_html=True)
st.code(html_output, language="html")
st.download_button(
"Download HTML",
html_output,
file_name="blog_outline.html",
mime="text/html"
)
st.markdown('</div>', unsafe_allow_html=True)
except Exception as e:
st.error(f"Error generating outline: {str(e)}")
st.markdown('</div>', unsafe_allow_html=True)
# Display the outline if it exists in session state
if st.session_state.outline:
st.markdown('<div class="preview-section">', unsafe_allow_html=True)
for section, subsections in st.session_state.outline.items():
content = st.session_state.section_contents.get(section)
display_section(section, subsections, content, generator)
st.markdown('</div>', unsafe_allow_html=True)
st.markdown('</div>', unsafe_allow_html=True) # Close the outline container
if __name__ == "__main__":
main()

163
ToBeMigrated/ai_writers/blog_rewriter_updater/README.md
View File

@@ -1,163 +0,0 @@
# AI Blog Rewriter & Updater
A powerful AI-powered tool for rewriting and updating existing blog content with improved quality, factual accuracy, and SEO optimization.
## Features
### 1. Content Import
- **URL Import**: Automatically extract content from any blog URL
- **Manual Input**: Paste content directly with title, meta description, and author information
- **Smart Content Extraction**: Preserves structure, headings, images, and metadata
### 2. Content Analysis
- **Metrics Analysis**:
- Word count
- Sentence count
- Paragraph count
- Average words per sentence
- Average sentences per paragraph
- **Structure Analysis**:
- Heading hierarchy
- Content organization
- Image analysis
- **Age Analysis**:
- Content age calculation
- Publication date detection
### 3. Web Research
- **Topic Extraction**: Automatically identifies key topics for fact-checking
- **Multi-Source Research**: Gathers information from various sources
- **Research Depth Control**: Choose between low, medium, and high research depth
- **Source Organization**: Categorizes research by topic with source details
### 4. Rewriting Modes
- **Standard Rewrite**: Improve clarity and flow while maintaining core message
- **SEO Optimization**: Enhance content for search engines with targeted keywords
- **Simplification**: Make complex content more accessible
- **Expansion**: Add more details and examples
- **Fact Check**: Update outdated information
- **Tone Shift**: Change writing style while preserving content
- **Modernization**: Update with current information and trends
### 5. Customization Options
- **Tone Selection**:
- Professional
- Conversational
- Academic
- Enthusiastic
- Authoritative
- Friendly
- Technical
- Inspirational
- **Length Control**:
- Maintain original length
- Create shorter version
- Create longer version
- Custom word count
- **SEO Features**:
- Focus keyword optimization
- Meta description generation
- Title optimization
- **Special Instructions**: Add custom requirements for the rewrite
### 6. Image Generation
- **AI Image Suggestions**: Get recommendations for relevant images
- **Custom Image Generation**: Create images based on content
- **Style Options**:
- Realistic
- Artistic
- Cartoon
- 3D Render
- **Image Placement**: Suggested optimal placement within content
### 7. Export Options
- **Preview Mode**: View formatted content
- **Markdown Export**: Get clean markdown version
- **Image Integration**: Include generated images with captions
- **Meta Information**: Export with optimized title and meta description
## Usage
1. **Import Content**
- Choose between URL import or manual content entry
- Provide necessary metadata (title, author, etc.)
2. **Analysis & Research**
- Review content analysis metrics
- Examine research findings
- Identify areas for improvement
3. **Configure Rewrite Settings**
- Select rewrite mode
- Choose target tone
- Set content length
- Add focus keywords
- Provide special instructions
4. **Review & Export**
- Preview rewritten content
- Generate suggested images
- Export in desired format
## Technical Details
### Dependencies
- Streamlit for UI
- BeautifulSoup for content extraction
- GPT providers for text generation
- Image generation capabilities
- Web research APIs (Exa, Tavily)
### Key Components
- `BlogRewriter` class: Core functionality
- Content extraction and analysis
- Research integration
- AI-powered rewriting
- Image generation
- Export capabilities
### Error Handling
- Robust error handling for URL extraction
- Fallback mechanisms for content parsing
- Graceful degradation for API failures
- User-friendly error messages
## Best Practices
1. **Content Import**
- Use clean, well-structured URLs
- Provide complete metadata for manual entry
- Ensure content is properly formatted
2. **Research Settings**
- Choose appropriate research depth
- Review research findings carefully
- Verify source credibility
3. **Rewrite Configuration**
- Select appropriate tone for audience
- Use relevant focus keywords
- Provide clear special instructions
4. **Image Generation**
- Use descriptive prompts
- Choose appropriate style
- Consider image placement
## Limitations
- Maximum content length for processing
- API rate limits for research
- Image generation constraints
- Language support limitations
## Future Enhancements
- Multi-language support
- Advanced SEO analysis
- Content structure templates
- Collaborative editing
- Integration with CMS platforms
- Custom AI model selection
- Advanced image editing
- Content versioning

11
ToBeMigrated/ai_writers/blog_rewriter_updater/ai_blog_rewriter.py
View File

@@ -1,11 +0,0 @@
"""
AI Blog Rewriter Module
This module provides the main entry point for the blog rewriter functionality,
importing and using the utility and UI modules.
"""
from .blog_rewriter_ui import write_blog_rewriter
if __name__ == "__main__":
write_blog_rewriter()

624
ToBeMigrated/ai_writers/blog_rewriter_updater/blog_rewriter_ui.py
View File

@@ -1,624 +0,0 @@
"""
Blog Rewriter UI Module
This module contains the Streamlit interface for the blog rewriter,
providing a user-friendly way to interact with the rewriting functionality.
"""
import streamlit as st
import json
from datetime import datetime
from .blog_rewriter_utils import BlogRewriter, REWRITE_MODES, TONE_OPTIONS, MAX_META_DESCRIPTION_LENGTH
def write_blog_rewriter():
"""Main function to display the blog rewriter UI."""
st.title("AI Blog Rewriter & Updater")
# Create a container for the header section
with st.container():
st.markdown("""
<div style="background-color: #f8f9fa; padding: 20px; border-radius: 10px; margin-bottom: 20px;">
<h3 style="margin-top: 0;">Revitalize Your Content</h3>
<p>Update, fact-check, and enhance your existing blog posts with AI assistance.
Our tool analyzes your content, researches the latest information, and rewrites your blog
to be more engaging, accurate, and SEO-friendly.</p>
</div>
""", unsafe_allow_html=True)
# Initialize the BlogRewriter class
if "blog_rewriter" not in st.session_state:
st.session_state.blog_rewriter = BlogRewriter()
# Initialize session state variables
if "original_content" not in st.session_state:
st.session_state.original_content = {}
if "content_analysis" not in st.session_state:
st.session_state.content_analysis = {}
if "research_results" not in st.session_state:
st.session_state.research_results = {}
if "rewritten_content" not in st.session_state:
st.session_state.rewritten_content = {}
if "generated_images" not in st.session_state:
st.session_state.generated_images = {}
if "current_step" not in st.session_state:
st.session_state.current_step = 1
# Create tabs for the workflow
tab1, tab2, tab3, tab4 = st.tabs([
"1️⃣ Import Content",
"2️⃣ Analyze & Research",
"3️⃣ Rewrite Settings",
"4️⃣ Results & Export"
])
# Tab 1: Import Content
with tab1:
st.header("Import Your Blog Content")
import_method = st.radio(
"Choose import method:",
["Import from URL", "Paste content manually"],
horizontal=True
)
if import_method == "Import from URL":
url = st.text_input(
"Enter blog URL:",
placeholder="https://example.com/blog-post",
help="Enter the full URL of the blog post you want to rewrite"
)
if st.button("Import Content", type="primary"):
if not url:
st.error("Please enter a valid URL")
else:
with st.spinner("Extracting content from URL..."):
# Extract content from URL
st.session_state.original_content = st.session_state.blog_rewriter.extract_content_from_url(url)
if "error" in st.session_state.original_content:
st.error(f"Error extracting content: {st.session_state.original_content['error']}")
else:
st.success("Content extracted successfully!")
st.session_state.current_step = 2
st.rerun()
else:
col1, col2 = st.columns([3, 1])
with col1:
title = st.text_input(
"Blog Title:",
placeholder="Enter the title of your blog post"
)
with col2:
author = st.text_input(
"Author (optional):",
placeholder="Author name"
)
meta_description = st.text_area(
"Meta Description (optional):",
placeholder="Enter the meta description of your blog post",
max_chars=MAX_META_DESCRIPTION_LENGTH,
height=80
)
content = st.text_area(
"Blog Content:",
placeholder="Paste your blog content here...",
height=300
)
if st.button("Import Content", type="primary"):
if not title or not content:
st.error("Please enter both title and content")
else:
# Store the manually entered content
st.session_state.original_content = {
"title": title,
"meta_description": meta_description,
"content": content,
"author": author,
"headings": [],
"images": [],
"publish_date": None,
"url": None
}
st.success("Content imported successfully!")
st.session_state.current_step = 2
st.rerun()
# Display the imported content if available
if st.session_state.original_content and "title" in st.session_state.original_content:
with st.expander("View Imported Content", expanded=False):
st.subheader(st.session_state.original_content["title"])
if st.session_state.original_content.get("meta_description"):
st.markdown(f"**Meta Description:** {st.session_state.original_content['meta_description']}")
if st.session_state.original_content.get("author"):
st.markdown(f"**Author:** {st.session_state.original_content['author']}")
if st.session_state.original_content.get("publish_date"):
st.markdown(f"**Published:** {st.session_state.original_content['publish_date']}")
st.markdown("**Content Preview:**")
content_preview = st.session_state.original_content["content"]
if len(content_preview) > 1000:
content_preview = content_preview[:1000] + "..."
st.text_area("", content_preview, height=200, disabled=True)
# Display images if available
if st.session_state.original_content.get("images"):
st.markdown(f"**Images:** {len(st.session_state.original_content['images'])} images found")
# Tab 2: Analyze & Research
with tab2:
st.header("Analyze & Research")
if not st.session_state.original_content or "title" not in st.session_state.original_content:
st.info("Please import your blog content first")
else:
col1, col2 = st.columns(2)
with col1:
if st.button("Analyze Content", type="primary"):
with st.spinner("Analyzing content..."):
# Analyze the content
st.session_state.content_analysis = st.session_state.blog_rewriter.analyze_content(
st.session_state.original_content
)
st.success("Content analysis complete!")
with col2:
research_depth = st.selectbox(
"Research Depth:",
["low", "medium", "high"],
index=1,
format_func=lambda x: {"low": "Basic", "medium": "Standard", "high": "Comprehensive"}[x],
help="Choose the depth of research to update your content"
)
if st.button("Conduct Research", type="primary"):
with st.spinner("Researching latest information..."):
# Conduct research
st.session_state.research_results = st.session_state.blog_rewriter.conduct_research(
st.session_state.original_content["title"],
st.session_state.original_content["content"],
research_depth
)
st.success("Research complete!")
# Display content analysis if available
if st.session_state.content_analysis:
st.subheader("Content Analysis")
metrics = st.session_state.content_analysis.get("metrics", {})
# Create metrics display
col1, col2, col3, col4 = st.columns(4)
with col1:
st.metric("Word Count", metrics.get("word_count", 0))
with col2:
st.metric("Paragraphs", metrics.get("paragraph_count", 0))
with col3:
st.metric("Sentences", metrics.get("sentence_count", 0))
with col4:
content_age = st.session_state.content_analysis.get("content_age", {})
if "months" in content_age:
st.metric("Content Age", f"{content_age['months']} months")
elif "error" in content_age:
st.metric("Content Age", "Unknown")
# Heading structure
heading_structure = st.session_state.content_analysis.get("heading_structure", {})
if heading_structure:
st.markdown("**Heading Structure:**")
for level, count in sorted(heading_structure.items()):
st.markdown(f"H{level}: {count} headings")
# Image analysis
images = st.session_state.content_analysis.get("images", {})
if images:
st.markdown(f"**Images:** {images.get('count', 0)} images found, {images.get('with_alt_text', 0)} with alt text")
# Display research results if available
if st.session_state.research_results:
st.subheader("Research Results")
topics = st.session_state.research_results.get("topics", [])
if topics:
for topic in topics:
with st.expander(f"Topic: {topic['topic']}", expanded=False):
for i, source in enumerate(topic.get("sources", [])):
st.markdown(f"**Source {i+1}:** {source.get('title', 'Untitled')}")
st.markdown(f"**URL:** {source.get('url', 'No URL')}")
st.markdown(f"**Content Preview:** {source.get('content', 'No content')[:200]}...")
st.markdown("---")
else:
st.info("No research results available")
# Enable proceeding to the next step if both analysis and research are done
if st.session_state.content_analysis and st.session_state.research_results:
if st.button("Proceed to Rewrite Settings", type="primary"):
st.session_state.current_step = 3
st.rerun()
# Tab 3: Rewrite Settings
with tab3:
st.header("Rewrite Settings")
if not st.session_state.original_content or "title" not in st.session_state.original_content:
st.info("Please import your blog content first")
elif not st.session_state.content_analysis or not st.session_state.research_results:
st.info("Please complete content analysis and research first")
else:
# Create a form for rewrite settings
with st.form("rewrite_settings_form"):
st.subheader("Content Transformation")
col1, col2 = st.columns(2)
with col1:
rewrite_mode = st.selectbox(
"Rewrite Mode:",
list(REWRITE_MODES.keys()),
format_func=lambda x: x.replace("_", " ").title(),
help="Choose how you want to transform your content"
)
st.info(REWRITE_MODES[rewrite_mode])
with col2:
tone = st.selectbox(
"Target Tone:",
TONE_OPTIONS,
index=0,
help="Choose the tone for your rewritten content"
)
st.subheader("Content Length")
original_word_count = st.session_state.content_analysis.get("metrics", {}).get("word_count", 0)
length_option = st.radio(
"Target Length:",
["same", "shorter", "longer", "custom"],
format_func=lambda x: {
"same": f"Same as original ({original_word_count} words)",
"shorter": f"Shorter (about {int(original_word_count * 0.7)} words)",
"longer": f"Longer (about {int(original_word_count * 1.3)} words)",
"custom": "Custom word count"
}[x],
horizontal=True
)
if length_option == "custom":
target_word_count = st.number_input(
"Custom Word Count:",
min_value=100,
max_value=10000,
value=original_word_count,
step=100
)
else:
target_word_count = {
"same": original_word_count,
"shorter": int(original_word_count * 0.7),
"longer": int(original_word_count * 1.3)
}[length_option]
st.subheader("SEO Optimization")
keywords = st.text_input(
"Focus Keywords (comma-separated):",
placeholder="e.g., digital marketing, SEO, content strategy",
help="Enter keywords to optimize your content for"
)
st.subheader("Additional Instructions")
special_instructions = st.text_area(
"Special Instructions (optional):",
placeholder="Add any specific instructions for rewriting your content...",
help="Provide any additional instructions for the AI"
)
# Submit button
submitted = st.form_submit_button("Rewrite Blog", type="primary")
if submitted:
# Process the form data
user_preferences = {
"rewrite_mode": rewrite_mode,
"tone": tone,
"target_word_count": target_word_count,
"keywords": [k.strip() for k in keywords.split(",")] if keywords else [],
"special_instructions": special_instructions
}
with st.spinner("Rewriting your blog..."):
# Rewrite the blog
st.session_state.rewritten_content = st.session_state.blog_rewriter.rewrite_blog(
st.session_state.original_content,
user_preferences,
st.session_state.research_results,
st.session_state.content_analysis
)
if "error" in st.session_state.rewritten_content:
st.error(f"Error rewriting blog: {st.session_state.rewritten_content['error']}")
else:
st.success("Blog rewritten successfully!")
st.session_state.current_step = 4
st.rerun()
# Tab 4: Results & Export
with tab4:
st.header("Results & Export")
if not st.session_state.rewritten_content or "title" not in st.session_state.rewritten_content:
st.info("Please complete the rewriting process first")
else:
# Display the rewritten content
st.subheader("Rewritten Blog")
# Title and meta description
st.markdown(f"## {st.session_state.rewritten_content['title']}")
if st.session_state.rewritten_content.get("meta_description"):
with st.expander("Meta Description", expanded=True):
st.text_area(
"",
st.session_state.rewritten_content["meta_description"],
height=80,
disabled=True
)
# Create tabs for different views
content_tab1, content_tab2 = st.tabs(["Preview", "Markdown"])
with content_tab1:
st.markdown(st.session_state.rewritten_content["content"])
with content_tab2:
st.text_area(
"",
st.session_state.rewritten_content["content"],
height=400
)
# Image generation section
st.subheader("Generate Images")
suggested_images = st.session_state.rewritten_content.get("suggested_images", [])
if suggested_images:
st.markdown("**Suggested Images:**")
for i, img in enumerate(suggested_images):
with st.expander(f"Image {i+1}: {img.get('description', 'No description')}", expanded=False):
st.markdown(f"**Description:** {img.get('description', 'No description')}")
st.markdown(f"**Caption:** {img.get('caption', 'No caption')}")
st.markdown(f"**Placement:** {img.get('placement', 'No placement specified')}")
# Generate image button
col1, col2 = st.columns([3, 1])
with col1:
image_prompt = st.text_area(
"Image Prompt:",
value=img.get('description', ''),
key=f"image_prompt_{i}"
)
with col2:
style = st.selectbox(
"Style:",
["realistic", "artistic", "cartoon", "3d_render"],
key=f"style_{i}"
)
if st.button("Generate Image", key=f"gen_img_{i}"):
with st.spinner("Generating image..."):
image_path = st.session_state.blog_rewriter.generate_image(image_prompt, style)
if image_path:
# Store the generated image
if "generated_images" not in st.session_state:
st.session_state.generated_images = {}
st.session_state.generated_images[f"image_{i}"] = {
"path": image_path,
"caption": img.get('caption', ''),
"placement": img.get('placement', '')
}
st.success("Image generated successfully!")
st.rerun()
# Display the generated image if available
if f"image_{i}" in st.session_state.generated_images:
st.image(
st.session_state.generated_images[f"image_{i}"]["path"],
caption=st.session_state.generated_images[f"image_{i}"]["caption"],
use_column_width=True
)
else:
st.info("No image suggestions available")
# Custom image generation
with st.expander("Generate Custom Image", expanded=True):
col1, col2 = st.columns([3, 1])
with col1:
custom_image_prompt = st.text_area(
"Image Prompt:",
placeholder="Describe the image you want to generate..."
)
with col2:
custom_style = st.selectbox(
"Style:",
["realistic", "artistic", "cartoon", "3d_render"]
)
if st.button("Generate Custom Image"):
if not custom_image_prompt:
st.error("Please enter an image prompt")
else:
with st.spinner("Generating image..."):
image_path = st.session_state.blog_rewriter.generate_image(custom_image_prompt, custom_style)
if image_path:
# Store the generated image
if "generated_images" not in st.session_state:
st.session_state.generated_images = {}
st.session_state.generated_images["custom_image"] = {
"path": image_path,
"caption": "Custom generated image",
"placement": "Custom placement"
}
st.success("Image generated successfully!")
st.rerun()
# Display the generated custom image if available
if "custom_image" in st.session_state.generated_images:
st.image(
st.session_state.generated_images["custom_image"]["path"],
caption=st.session_state.generated_images["custom_image"]["caption"],
use_column_width=True
)
# Export options
st.subheader("Export Options")
col1, col2, col3 = st.columns(3)
with col1:
st.download_button(
"Download as Markdown",
data=st.session_state.rewritten_content["content"],
file_name=f"{st.session_state.rewritten_content['title'].replace(' ', '_')}.md",
mime="text/markdown"
)
with col2:
# Create HTML version
html_content = f"""
<!DOCTYPE html>
<html>
<head>
<title>{st.session_state.rewritten_content['title']}</title>
<meta name="description" content="{st.session_state.rewritten_content.get('meta_description', '')}">
<style>
body {{ font-family: Arial, sans-serif; line-height: 1.6; max-width: 800px; margin: 0 auto; padding: 20px; }}
h1, h2, h3, h4, h5, h6 {{ color: #333; }}
img {{ max-width: 100%; height: auto; }}
pre {{ background-color: #f5f5f5; padding: 15px; border-radius: 5px; overflow-x: auto; }}
blockquote {{ border-left: 5px solid #eee; padding-left: 15px; margin-left: 0; }}
</style>
</head>
<body>
<h1>{st.session_state.rewritten_content['title']}</h1>
{st.session_state.rewritten_content['content']}
</body>
</html>
"""
st.download_button(
"Download as HTML",
data=html_content,
file_name=f"{st.session_state.rewritten_content['title'].replace(' ', '_')}.html",
mime="text/html"
)
with col3:
# Create JSON version with all content and metadata
json_content = {
"title": st.session_state.rewritten_content["title"],
"meta_description": st.session_state.rewritten_content.get("meta_description", ""),
"content": st.session_state.rewritten_content["content"],
"suggested_images": st.session_state.rewritten_content.get("suggested_images", []),
"generated_images": [
{
"caption": img_data["caption"],
"placement": img_data["placement"],
"path": img_data["path"]
}
for img_key, img_data in st.session_state.generated_images.items()
] if hasattr(st.session_state, "generated_images") else [],
"original_title": st.session_state.original_content.get("title", ""),
"original_url": st.session_state.original_content.get("url", ""),
"rewrite_date": datetime.now().isoformat()
}
st.download_button(
"Download as JSON",
data=json.dumps(json_content, indent=2),
file_name=f"{st.session_state.rewritten_content['title'].replace(' ', '_')}.json",
mime="application/json"
)
# Copy to clipboard buttons
st.subheader("Quick Copy")
col1, col2, col3 = st.columns(3)
with col1:
if st.button("Copy Title", key="copy_title"):
st.code(st.session_state.rewritten_content["title"])
st.success("Title copied to clipboard!")
with col2:
if st.button("Copy Meta Description", key="copy_meta"):
st.code(st.session_state.rewritten_content.get("meta_description", ""))
st.success("Meta description copied to clipboard!")
with col3:
if st.button("Copy Full Content", key="copy_content"):
st.success("Content copied to clipboard!")
# Comparison with original
with st.expander("Compare with Original", expanded=False):
comp_col1, comp_col2 = st.columns(2)
with comp_col1:
st.subheader("Original")
st.markdown(f"**Title:** {st.session_state.original_content.get('title', '')}")
if st.session_state.original_content.get("meta_description"):
st.markdown(f"**Meta Description:** {st.session_state.original_content['meta_description']}")
st.text_area(
"Original Content",
st.session_state.original_content.get("content", ""),
height=300,
disabled=True
)
with comp_col2:
st.subheader("Rewritten")
st.markdown(f"**Title:** {st.session_state.rewritten_content['title']}")
if st.session_state.rewritten_content.get("meta_description"):
st.markdown(f"**Meta Description:** {st.session_state.rewritten_content['meta_description']}")
st.text_area(
"Rewritten Content",
st.session_state.rewritten_content["content"],
height=300,
disabled=True
)
# Start over button
if st.button("Start Over", type="primary"):
# Reset session state
for key in ["original_content", "content_analysis", "research_results",
"rewritten_content", "generated_images", "current_step"]:
if key in st.session_state:
del st.session_state[key]
st.rerun()
if __name__ == "__main__":
write_blog_rewriter()

595
ToBeMigrated/ai_writers/blog_rewriter_updater/blog_rewriter_utils.py
View File

@@ -1,595 +0,0 @@
"""
Blog Rewriter Utilities Module
This module contains the core functionality for rewriting and updating blog content,
including content extraction, analysis, research, and rewriting capabilities.
"""
import requests
from bs4 import BeautifulSoup
import re
import time
import logging
from typing import Dict, List, Tuple, Optional, Any
import json
import os
from datetime import datetime
# Import required modules from the project
from ...gpt_providers.text_generation.main_text_generation import llm_text_gen
from ...gpt_providers.text_to_image_generation.main_generate_image_from_prompt import generate_image
from ...ai_web_researcher.metaphor_basic_neural_web_search import metaphor_search_articles
from ...ai_web_researcher.tavily_ai_search import do_tavily_ai_search
# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
# Define constants
MAX_TITLE_LENGTH = 70
MAX_META_DESCRIPTION_LENGTH = 160
REWRITE_MODES = {
"standard": "Standard rewrite with improved clarity and flow",
"seo_optimization": "Optimize for search engines with targeted keywords",
"simplification": "Simplify complex content for broader audience",
"expansion": "Expand with additional details and examples",
"fact_check": "Focus on fact-checking and updating information",
"tone_shift": "Change the tone while preserving content",
"modernization": "Update outdated content with current information"
}
# Define tone options
TONE_OPTIONS = [
"Professional", "Conversational", "Academic", "Enthusiastic",
"Authoritative", "Friendly", "Technical", "Inspirational"
]
class BlogRewriter:
"""Class to handle blog rewriting functionality."""
def __init__(self):
"""Initialize the BlogRewriter class."""
self.original_content = {}
self.rewritten_content = {}
self.research_results = {}
self.content_analysis = {}
self.image_suggestions = []
def extract_content_from_url(self, url: str) -> Dict[str, Any]:
"""
Extract content from a given URL.
Args:
url: The URL to extract content from
Returns:
Dictionary containing extracted content
"""
logger.info(f"Extracting content from URL: {url}")
try:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.5',
'Connection': 'keep-alive',
'Upgrade-Insecure-Requests': '1',
'Cache-Control': 'max-age=0'
}
response = requests.get(url, headers=headers, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
# Extract title
title = soup.title.string if soup.title else ""
# Extract meta description
meta_desc = ""
meta_tag = soup.find("meta", attrs={"name": "description"})
if meta_tag and "content" in meta_tag.attrs:
meta_desc = meta_tag["content"]
# Extract main content - try multiple strategies
content = ""
# Strategy 1: Look for article tag
article_tag = soup.find("article")
if article_tag:
content = article_tag.get_text(separator="\n\n")
# Strategy 2: Look for main content areas
if not content:
main_content = soup.find(["main", "div", "section"], class_=re.compile(r"content|article|post|entry|main|body"))
if main_content:
for elem in main_content.find_all(["nav", "aside", "footer", "comments", "script", "style", "header"]):
elem.decompose()
content = main_content.get_text(separator="\n\n")
# Strategy 3: Look for specific content classes
if not content:
content_classes = ["post-content", "entry-content", "article-content", "blog-content", "content-area"]
for class_name in content_classes:
content_div = soup.find("div", class_=class_name)
if content_div:
for elem in content_div.find_all(["nav", "aside", "footer", "comments", "script", "style", "header"]):
elem.decompose()
content = content_div.get_text(separator="\n\n")
break
# Strategy 4: Look for content within body
if not content:
body = soup.find("body")
if body:
# Remove unwanted elements
for elem in body.find_all(["nav", "aside", "footer", "comments", "script", "style", "header"]):
elem.decompose()
content = body.get_text(separator="\n\n")
# Clean up the content
content = re.sub(r'\n{3,}', '\n\n', content)
content = re.sub(r'\s{2,}', ' ', content)
content = content.strip()
# Extract headings with their hierarchy
headings = []
for h in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6"]):
headings.append({
"level": int(h.name[1]),
"text": h.get_text().strip()
})
# Extract images with more metadata
images = []
for img in soup.find_all("img"):
if img.get("src") and not img.get("src").startswith("data:"):
image_url = img.get("src")
if not image_url.startswith(("http://", "https://")):
base_url = "/".join(url.split("/")[:3])
image_url = f"{base_url}/{image_url.lstrip('/')}"
images.append({
"url": image_url,
"alt_text": img.get("alt", ""),
"title": img.get("title", ""),
"class": img.get("class", []),
"width": img.get("width"),
"height": img.get("height")
})
# Extract publish date with multiple strategies
publish_date = None
# Try meta tags first
date_meta = soup.find("meta", attrs={"property": "article:published_time"})
if date_meta and "content" in date_meta.attrs:
publish_date = date_meta["content"]
else:
# Try other meta tags
for prop in ["datePublished", "dateCreated", "dateModified"]:
date_meta = soup.find("meta", attrs={"property": prop})
if date_meta and "content" in date_meta.attrs:
publish_date = date_meta["content"]
break
# Try HTML elements if meta tags failed
if not publish_date:
date_elem = soup.find(["time", "span", "div"], class_=re.compile(r"date|time|publish|posted|created"))
if date_elem and date_elem.get_text():
publish_date = date_elem.get_text().strip()
# Extract author with multiple strategies
author = None
# Try meta tags first
author_meta = soup.find("meta", attrs={"name": "author"})
if author_meta and "content" in author_meta.attrs:
author = author_meta["content"]
else:
# Try other meta tags
for prop in ["article:author", "author"]:
author_meta = soup.find("meta", attrs={"property": prop})
if author_meta and "content" in author_meta.attrs:
author = author_meta["content"]
break
# Try HTML elements if meta tags failed
if not author:
author_elem = soup.find(["a", "span", "div"], class_=re.compile(r"author|byline|writer|posted-by"))
if author_elem and author_elem.get_text():
author = author_elem.get_text().strip()
# Log content extraction results
logger.info(f"Extracted content length: {len(content)} characters")
logger.info(f"Found {len(headings)} headings")
logger.info(f"Found {len(images)} images")
logger.info(f"Publish date: {publish_date}")
logger.info(f"Author: {author}")
return {
"title": title,
"meta_description": meta_desc,
"content": content,
"headings": headings,
"images": images,
"publish_date": publish_date,
"author": author,
"url": url
}
except Exception as e:
logger.error(f"Error extracting content from URL: {e}")
return {
"title": "",
"meta_description": "",
"content": "",
"headings": [],
"images": [],
"publish_date": None,
"author": None,
"url": url,
"error": str(e)
}
def analyze_content(self, content: Dict[str, Any]) -> Dict[str, Any]:
"""
Analyze the extracted content to provide insights.
Args:
content: Dictionary containing extracted content
Returns:
Dictionary containing content analysis
"""
logger.info("Analyzing content")
analysis = {}
# Basic metrics
text_content = content.get("content", "")
word_count = len(text_content.split())
sentence_count = len(re.split(r'[.!?]+', text_content))
paragraph_count = len(re.split(r'\n\n+', text_content))
analysis["metrics"] = {
"word_count": word_count,
"sentence_count": sentence_count,
"paragraph_count": paragraph_count,
"avg_words_per_sentence": round(word_count / max(sentence_count, 1), 1),
"avg_sentences_per_paragraph": round(sentence_count / max(paragraph_count, 1), 1)
}
# Heading structure analysis
headings = content.get("headings", [])
heading_structure = {}
for h in headings:
level = h["level"]
if level not in heading_structure:
heading_structure[level] = 0
heading_structure[level] += 1
analysis["heading_structure"] = heading_structure
# Content age analysis
publish_date = content.get("publish_date")
if publish_date:
try:
if "T" in publish_date:
pub_date = datetime.fromisoformat(publish_date.replace("Z", "+00:00"))
else:
date_formats = [
"%Y-%m-%d", "%d-%m-%Y", "%B %d, %Y", "%b %d, %Y",
"%d %B %Y", "%d %b %Y", "%Y/%m/%d", "%d/%m/%Y"
]
for fmt in date_formats:
try:
pub_date = datetime.strptime(publish_date, fmt)
break
except ValueError:
continue
now = datetime.now()
age_days = (now - pub_date).days
analysis["content_age"] = {
"days": age_days,
"months": round(age_days / 30, 1),
"years": round(age_days / 365, 1)
}
except Exception as e:
logger.warning(f"Could not parse publish date: {e}")
analysis["content_age"] = {"error": "Could not determine content age"}
else:
analysis["content_age"] = {"error": "No publish date found"}
# Image analysis
images = content.get("images", [])
analysis["images"] = {
"count": len(images),
"with_alt_text": sum(1 for img in images if img.get("alt_text"))
}
return analysis
def conduct_research(self, title: str, content: str, research_depth: str = "medium") -> Dict[str, Any]:
"""
Conduct web research to find updated information related to the blog content.
Args:
title: Blog title
content: Blog content
research_depth: Depth of research (low, medium, high)
Returns:
Dictionary containing research results
"""
logger.info(f"Conducting research with depth: {research_depth}")
# Extract key topics from the content
prompt = f"""
Extract 3-5 key topics or claims from this blog content that might need fact-checking or updating.
For each topic, provide a concise search query that would help find the most recent information.
Blog title: {title}
First 1000 characters of content:
{content[:1000]}...
Format your response as a JSON array of objects with 'topic' and 'query' fields.
"""
try:
topics_json = llm_text_gen(prompt)
topics_json = re.search(r'\[.*\]', topics_json, re.DOTALL)
if topics_json:
topics = json.loads(topics_json.group(0))
else:
topics = [
{"topic": title, "query": title + " latest information"},
{"topic": "Updates on " + title, "query": title + " recent developments"}
]
except Exception as e:
logger.error(f"Error extracting topics: {e}")
topics = [
{"topic": title, "query": title + " latest information"},
{"topic": "Updates on " + title, "query": title + " recent developments"}
]
# Determine number of results based on research depth
num_results = {"low": 2, "medium": 3, "high": 5}.get(research_depth, 3)
research_results = {"topics": []}
# Conduct research for each topic
for topic in topics[:3]: # Limit to 3 topics
topic_results = {"topic": topic["topic"], "sources": []}
# Try Exa search first
try:
exa_results = metaphor_search_articles(topic["query"], num_results=num_results)
if exa_results:
topic_results["sources"].extend(exa_results)
except Exception as e:
logger.warning(f"Exa search failed: {e}")
# If Exa didn't return enough results, try Tavily
if len(topic_results["sources"]) < num_results:
try:
tavily_results = do_tavily_ai_search(topic["query"], num_results=num_results)
if tavily_results:
existing_urls = [s["url"] for s in topic_results["sources"]]
for result in tavily_results:
if result["url"] not in existing_urls:
topic_results["sources"].append(result)
existing_urls.append(result["url"])
except Exception as e:
logger.warning(f"Tavily search failed: {e}")
research_results["topics"].append(topic_results)
return research_results
def generate_rewrite_prompt(self, original_content: Dict[str, Any],
user_preferences: Dict[str, Any],
research_results: Dict[str, Any],
content_analysis: Dict[str, Any]) -> str:
"""
Generate a prompt for the LLM to rewrite the blog.
Args:
original_content: Original blog content
user_preferences: User preferences for rewriting
research_results: Research results for updating content
content_analysis: Analysis of the original content
Returns:
Prompt string for the LLM
"""
logger.info("Generating rewrite prompt")
# Extract key information
title = original_content.get("title", "")
content = original_content.get("content", "")
# Truncate content if it's too long
max_content_length = 6000
if len(content) > max_content_length:
content_preview = content[:max_content_length] + "...\n[Content truncated due to length]"
else:
content_preview = content
# Format research results
research_summary = ""
for topic in research_results.get("topics", []):
research_summary += f"\n## {topic['topic']}\n"
for i, source in enumerate(topic.get("sources", [])[:3]):
research_summary += f"Source {i+1}: {source.get('title', 'Untitled')}\n"
research_summary += f"URL: {source.get('url', 'No URL')}\n"
research_summary += f"Content: {source.get('content', 'No content')[:300]}...\n\n"
# Build the prompt
prompt = f"""
# Blog Rewriting Task
## Original Blog Information
Title: {title}
Word Count: {content_analysis.get('metrics', {}).get('word_count', 'Unknown')}
Estimated Age: {content_analysis.get('content_age', {}).get('months', 'Unknown')} months
## Rewriting Instructions
Mode: {user_preferences.get('rewrite_mode', 'standard')}
Target Tone: {user_preferences.get('tone', 'Professional')}
Target Word Count: {user_preferences.get('target_word_count', 'Same as original')}
Focus Keywords: {', '.join(user_preferences.get('keywords', []))}
## Special Instructions
{user_preferences.get('special_instructions', 'No special instructions')}
## Recent Research Findings
{research_summary if research_summary else "No research results available."}
## Original Content
{content_preview}
## Your Task
Please rewrite this blog post according to the instructions above. The rewritten blog should:
1. Maintain the core message and value of the original content
2. Update any outdated information based on the research findings
3. Adopt the requested tone and style
4. Incorporate the focus keywords naturally
5. Improve readability and engagement
6. Maintain a logical structure with appropriate headings
7. Include a compelling introduction and conclusion
## Output Format
Please provide your response in the following JSON format:
```json
{{
"title": "Rewritten title",
"meta_description": "SEO-optimized meta description (max 160 characters)",
"content": "Full rewritten content with proper markdown formatting",
"suggested_images": [
{{
"description": "Brief description of a suggested image",
"caption": "Suggested caption for the image",
"placement": "Where this image should be placed (e.g., 'After introduction', 'Before conclusion')"
}}
]
}}
```
Ensure the JSON is properly formatted and valid.
"""
return prompt
def rewrite_blog(self, original_content: Dict[str, Any],
user_preferences: Dict[str, Any],
research_results: Dict[str, Any],
content_analysis: Dict[str, Any]) -> Dict[str, Any]:
"""
Rewrite the blog based on original content, user preferences, and research.
Args:
original_content: Original blog content
user_preferences: User preferences for rewriting
research_results: Research results for updating content
content_analysis: Analysis of the original content
Returns:
Dictionary containing rewritten content
"""
logger.info("Rewriting blog content")
# Generate the prompt
prompt = self.generate_rewrite_prompt(
original_content, user_preferences, research_results, content_analysis
)
# Call the LLM to rewrite the content
try:
response = llm_text_gen(prompt)
# Clean the response of any invalid control characters
response = ''.join(char for char in response if ord(char) >= 32 or char in '\n\r\t')
# Extract JSON from the response
json_match = re.search(r'```json\s*(.*?)\s*```', response, re.DOTALL)
if json_match:
json_str = json_match.group(1)
else:
# If no JSON block found, try to find JSON-like content
json_match = re.search(r'\{.*\}', response, re.DOTALL)
if json_match:
json_str = json_match.group(0)
else:
json_str = response
# Clean up the JSON string
json_str = re.sub(r'```(json)?', '', json_str).strip()
# Remove any remaining invalid control characters
json_str = ''.join(char for char in json_str if ord(char) >= 32 or char in '\n\r\t')
# Parse the JSON with error handling
try:
rewritten_content = json.loads(json_str)
except json.JSONDecodeError as e:
logger.error(f"JSON parsing error: {e}")
# Try to fix common JSON issues
json_str = json_str.replace('\\n', '\\\\n') # Fix escaped newlines
json_str = json_str.replace('\\"', '"') # Fix escaped quotes
json_str = json_str.replace('\\t', '\\\\t') # Fix escaped tabs
rewritten_content = json.loads(json_str)
# Validate the response structure
required_fields = ["title", "meta_description", "content"]
for field in required_fields:
if field not in rewritten_content:
rewritten_content[field] = original_content.get(field, "")
logger.warning(f"Missing required field '{field}' in rewritten content")
# Ensure suggested_images exists
if "suggested_images" not in rewritten_content:
rewritten_content["suggested_images"] = []
# Clean up the content field
if "content" in rewritten_content:
# Remove any remaining invalid control characters
rewritten_content["content"] = ''.join(
char for char in rewritten_content["content"]
if ord(char) >= 32 or char in '\n\r\t'
)
# Normalize whitespace
rewritten_content["content"] = re.sub(r'\s+', ' ', rewritten_content["content"])
rewritten_content["content"] = re.sub(r'\n{3,}', '\n\n', rewritten_content["content"])
return rewritten_content
except Exception as e:
logger.error(f"Error rewriting blog: {e}")
return {
"title": original_content.get("title", ""),
"meta_description": original_content.get("meta_description", ""),
"content": original_content.get("content", ""),
"suggested_images": [],
"error": str(e)
}
def generate_image(self, image_prompt: str, style: str = "realistic") -> str:
"""
Generate an image based on the prompt.
Args:
image_prompt: Prompt for image generation
style: Style of the image
Returns:
Path to the generated image
"""
logger.info(f"Generating image with prompt: {image_prompt}")
try:
image_path = generate_image(image_prompt, style=style)
return image_path
except Exception as e:
logger.error(f"Error generating image: {e}")
return ""

47
ToBeMigrated/blog_metadata/README.md
View File

@@ -1,47 +0,0 @@
# AI Blog Metadata Generator
The AI Blog Metadata Generator module is designed to assist in creating SEO-optimized metadata for blog articles. Utilizing artificial intelligence, this module generates high-quality metadata to enhance the visibility and engagement of blog posts.
## Prerequisites
To use this module, ensure that the following prerequisites are met:
- Python 3.6 or higher
- Streamlit
- Loguru
- Asyncio
- A GPT provider (e.g., OpenAI, Gemini)
## Installation
Install the required packages using the Python package installer, pip:
```bash
pip install -r requirements.txt
```
## Usage
Follow these steps to utilize the AI Blog Metadata Generator module:
### Generate Blog Title
The module provides a function to create a blog title that is both SEO-optimized and engaging. This function ensures the title adheres to SEO best practices and avoids negative keywords.
### Generate Meta Description
This functionality creates a compelling meta description for the blog content. The description is kept between 150-160 characters to ensure it meets SEO standards.
### Generate Blog Tags
The module suggests relevant and specific tags for the blog content. This helps in categorizing and improving the discoverability of the blog post.
### Generate Blog Categories
The module identifies the main topics and suggests the most relevant categories for the blog content. This function ensures that the blog is categorized appropriately for the target audience and taxonomy.
## Helper Functions
The module includes helper functions to run the asyncio event loop within Streamlit, ensuring smooth and efficient operation of asynchronous tasks such as generating metadata.
By leveraging this module, users can enhance their blog posts with well-crafted metadata, improving their visibility and engagement in search engines.

435
ToBeMigrated/blog_metadata/get_blog_metadata.py
View File

@@ -1,435 +0,0 @@
import os
import time
import datetime
import sys
import streamlit as st
from loguru import logger
import random
import asyncio
import re
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}"
)
from ..gpt_providers.text_generation.main_text_generation import llm_text_gen
async def blog_metadata(blog_article):
"""
Generate comprehensive SEO metadata for a blog article.
Args:
blog_article (str): The content of the blog article
Returns:
tuple: (blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug)
"""
logger.info("Generating comprehensive blog metadata")
progress_bar = st.progress(0)
total_steps = 6 # Increased steps for new metadata types
status_container = st.empty()
try:
# Step 1: Generate blog title
status_container.info("Generating SEO-optimized blog title...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_title = generate_blog_title(blog_article)
progress_bar.progress(1 / total_steps)
# Step 2: Generate blog meta description
status_container.info("Creating compelling meta description...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_meta_desc = generate_blog_description(blog_article)
progress_bar.progress(2 / total_steps)
# Step 3: Generate blog tags
status_container.info("Extracting relevant blog tags...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_tags = get_blog_tags(blog_article)
progress_bar.progress(3 / total_steps)
# Step 4: Generate blog categories
status_container.info("Identifying primary blog categories...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_categories = get_blog_categories(blog_article)
progress_bar.progress(4 / total_steps)
# Step 5: Generate social media hashtags
status_container.info("Creating social media hashtags...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_hashtags = generate_blog_hashtags(blog_article)
progress_bar.progress(5 / total_steps)
# Step 6: Generate SEO URL slug
status_container.info("Generating SEO-friendly URL slug...")
await asyncio.sleep(random.uniform(0.5, 1.5))
blog_slug = generate_blog_slug(blog_title)
progress_bar.progress(6 / total_steps)
# Present the result in a table format
status_container.success("✅ Blog SEO Metadata generation complete")
#st.table({
# "Metadata": ["Blog Title", "Meta Description", "Tags", "Categories", "Social Hashtags", "URL Slug"],
# "Value": [blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug]
#})
return blog_title, blog_meta_desc, blog_tags, blog_categories, blog_hashtags, blog_slug
except Exception as e:
status_container.error(f"Error generating metadata: {str(e)}")
logger.error(f"Failed to generate metadata: {str(e)}")
# Return default values to ensure the blog generation process can continue
return f"Blog Article", "An informative blog post", "content, blog", "General, Information", "#content #blog", "blog-article"
def generate_blog_title(blog_article):
"""
Generate an SEO-optimized and engaging title for a blog article.
Args:
blog_article (str): The content of the blog article
Returns:
str: An SEO-optimized title
"""
logger.info("Generating SEO-optimized blog title")
# Extract the first 3000 characters for title generation
snippet = blog_article[:3000] if len(blog_article) > 3000 else blog_article
prompt = f"""As an expert SEO copywriter, create the perfect blog title based on this content.
REQUIREMENTS:
1. Make it compelling, specific, and actionable
2. Include primary keywords naturally near the beginning
3. Keep it between 50-60 characters (10-12 words maximum)
4. Make it promise clear value to the reader
5. Use power words that evoke emotion where appropriate
AVOID:
- Clickbait tactics or false promises
- Generic titles that could apply to any article
- Using words like "unveiling", "unleash", "power of", "ultimate guide", or "complete"
- ALL CAPS or excessive punctuation!!!!
EXAMPLES OF GREAT TITLES:
- "7 Proven Strategies to Improve Your Email Marketing ROI"
- "Why Remote Work Improves Productivity: New Research Findings"
- "How to Build a Personal Budget That Actually Works"
CONTENT TO ANALYZE:
"{snippet}"
Reply with ONLY the title and no other text or explanation.
"""
try:
title = llm_text_gen(prompt)
# Clean up any quotes or extra spaces
title = title.strip('"\'').strip()
logger.info(f"Generated title: {title}")
return title
except Exception as err:
logger.error(f"Failed to generate blog title: {err}")
return "Blog Article" # Fallback title
def generate_blog_description(blog_content):
"""
Generate an SEO-optimized meta description for the blog.
Args:
blog_content (str): The content of the blog article
Returns:
str: An SEO-optimized meta description
"""
logger.info("Generating SEO-optimized meta description")
# Extract the first 2000 characters for description generation
snippet = blog_content[:2000] if len(blog_content) > 2000 else blog_content
prompt = f"""As an SEO expert, write the perfect meta description for this blog content.
REQUIREMENTS:
1. Exactly 150-160 characters (this is critical for SEO)
2. Include primary keywords naturally
3. Compelling value proposition that makes readers want to click
4. Clear indication of what the reader will learn/gain
5. End with an implicit call-to-action when possible
EXAMPLES OF EXCELLENT META DESCRIPTIONS:
- "Learn how to increase email open rates by 43% with these 5 proven strategies from industry experts. Implement today for immediate results."
- "Discover why 67% of professionals struggle with work-life balance and explore research-backed techniques to reclaim your time and energy."
CONTENT TO SUMMARIZE:
"{snippet}"
Reply with ONLY the meta description and no other text. Keep it between 150-160 characters exactly.
"""
try:
description = llm_text_gen(prompt)
# Clean up any quotes or extra spaces
description = description.strip('"\'').strip()
logger.info(f"Generated meta description: {description}")
return description
except Exception as err:
logger.error(f"Failed to generate blog description: {err}")
return "An informative blog post about this topic." # Fallback description
def get_blog_tags(blog_article):
"""
Generate relevant SEO tags for a blog article.
Args:
blog_article (str): The content of the blog article
Returns:
str: Comma-separated list of relevant tags
"""
logger.info("Generating SEO-optimized blog tags")
# Extract the first 3000 characters for tag generation
snippet = blog_article[:3000] if len(blog_article) > 3000 else blog_article
prompt = f"""As an SEO specialist, extract the 4-6 most relevant tags for this blog post.
REQUIREMENTS:
1. Choose specific, targeted keywords that accurately represent the content
2. Include a mix of broad and specific tags
3. Focus on terms users would actually search for
4. Include at least one long-tail keyword phrase
5. Ensure all tags are directly addressed in the content
CONTENT TO ANALYZE:
"{snippet}"
Reply with ONLY the tags as a comma-separated list (e.g., "keyword1, keyword2, keyword3, keyword phrase"). Provide 4-6 tags total.
"""
try:
tags = llm_text_gen(prompt)
# Clean up any quotes or extra commas
tags = tags.strip('"\'').strip()
if tags.endswith(','):
tags = tags[:-1]
logger.info(f"Generated tags: {tags}")
return tags
except Exception as err:
logger.error(f"Failed to generate blog tags: {err}")
return "content, blog" # Fallback tags
def get_blog_categories(blog_article):
"""
Identify the most appropriate blog categories for the article.
Args:
blog_article (str): The content of the blog article
Returns:
str: Comma-separated list of relevant categories
"""
logger.info("Generating blog categories")
# Extract the first 2000 characters for category generation
snippet = blog_article[:2000] if len(blog_article) > 2000 else blog_article
prompt = f"""As a content strategist, identify the 2-3 most appropriate high-level categories for this blog.
REQUIREMENTS:
1. Choose broad, established categories used in content organization
2. Select categories that best represent the main themes of the article
3. Consider the target audience and their interests
4. Focus on categories that would help with site navigation
5. Aim for a primary category and 1-2 supporting categories
EXAMPLES OF GOOD CATEGORIES:
- Marketing, Social Media, Strategy
- Finance, Personal Budgeting, Money Management
- Productivity, Remote Work, Business
CONTENT TO ANALYZE:
"{snippet}"
Reply with ONLY the categories as a comma-separated list (e.g., "Category1, Category2, Category3"). Provide 2-3 categories total.
"""
try:
categories = llm_text_gen(prompt)
# Clean up any quotes or extra commas
categories = categories.strip('"\'').strip()
if categories.endswith(','):
categories = categories[:-1]
logger.info(f"Generated categories: {categories}")
return categories
except Exception as err:
logger.error(f"Failed to generate blog categories: {err}")
return "General, Information" # Fallback categories
def generate_blog_hashtags(blog_article):
"""
Generate social media hashtags for promoting the blog article.
Args:
blog_article (str): The content of the blog article
Returns:
str: Space-separated list of hashtags starting with #
"""
logger.info("Generating social media hashtags")
# Extract the first 2000 characters for hashtag generation
snippet = blog_article[:2000] if len(blog_article) > 2000 else blog_article
prompt = f"""As a social media strategist, create 5-7 effective hashtags for this blog content.
REQUIREMENTS:
1. Mix of popular and niche hashtags for better visibility
2. Include industry-specific and trending hashtags where relevant
3. Avoid overly generic hashtags (like #content or #blog)
4. Format each hashtag with # symbol and camelCase or separate words
5. Include at least one branded or campaign-style hashtag
EXAMPLES OF EFFECTIVE HASHTAG SETS:
- #EmailMarketing #ROITips #DigitalStrategy #MarketingTips #GrowthHacking #EmailROI
- #RemoteWork #ProductivityTips #FutureOfWork #WorkFromHome #RemoteProductivity #HRInsights
CONTENT TO ANALYZE:
"{snippet}"
Reply with ONLY the hashtags, each starting with # and separated by spaces. Provide 5-7 hashtags total.
"""
try:
hashtags = llm_text_gen(prompt)
# Clean up any quotes or extra spaces
hashtags = hashtags.strip('"\'').strip()
# Ensure all hashtags start with #
if not hashtags.startswith('#'):
hashtags = ' '.join([f"#{tag.strip('#')}" for tag in hashtags.split()])
logger.info(f"Generated hashtags: {hashtags}")
return hashtags
except Exception as err:
logger.error(f"Failed to generate blog hashtags: {err}")
return "#content #blog" # Fallback hashtags
def generate_blog_slug(blog_title):
"""
Generate an SEO-friendly URL slug from the blog title.
Args:
blog_title (str): The title of the blog article
Returns:
str: An SEO-friendly URL slug
"""
logger.info("Generating SEO-friendly URL slug")
try:
# Use a prompt to generate a customized slug
prompt = f"""As an SEO specialist, create an SEO-friendly URL slug for this blog title: "{blog_title}"
REQUIREMENTS:
1. Keep it under 60 characters
2. Use only lowercase letters, numbers, and hyphens
3. Include primary keywords near the beginning
4. Remove all unnecessary words (a, the, and, or, but, etc.)
5. Ensure it's human-readable and descriptive
EXAMPLES:
- Title: "10 Effective Ways to Improve Your Email Marketing ROI This Quarter"
Slug: "improve-email-marketing-roi"
- Title: "Why Most Remote Workers Are More Productive According to New Research"
Slug: "remote-workers-productivity-research"
Reply with ONLY the slug and no other text or explanation.
"""
slug = llm_text_gen(prompt)
# Clean up and normalize the slug
slug = slug.strip('"\'').strip()
# If the LLM didn't create a proper slug, do it programmatically
if not re.match(r'^[a-z0-9-]+$', slug):
# Fallback to simple programmatic slug creation
slug = blog_title.lower()
# Remove special characters
slug = re.sub(r'[^a-z0-9\s-]', '', slug)
# Replace spaces with hyphens
slug = re.sub(r'\s+', '-', slug)
# Remove redundant hyphens
slug = re.sub(r'-+', '-', slug)
# Limit length to 60 characters
slug = slug[:60].strip('-')
logger.info(f"Generated slug: {slug}")
return slug
except Exception as err:
logger.error(f"Failed to generate blog slug: {err}")
# Create a simple slug programmatically as fallback
slug = blog_title.lower()
slug = re.sub(r'[^a-z0-9\s-]', '', slug)
slug = re.sub(r'\s+', '-', slug)
slug = re.sub(r'-+', '-', slug)
slug = slug[:60].strip('-')
return slug
# Helper function to run the asyncio event loop within Streamlit
def run_async(coro):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
result = loop.run_until_complete(coro)
loop.close()
return result
def get_blog_metadata_longform(longform_content):
""" Function for caching long-form content """
# Open the file in write mode ("w") to overwrite existing content.
filepath = os.path.join(os.getenv("CONTENT_SAVE_DIR"), "lognform_metadata_file")
with open(filepath, "w") as file:
# Write the text to the file
file.write(longform_content)
print(f"String saved successfully to: {filepath}")
#genai.configure(api_key=os.environ['GEMINI_API_KEY'])
#file_path = genai.upload_file(path=filepath)
# Wait for the file to finish processing
#while file_path.state.name == 'PROCESSING':
# print('Waiting for video to be processed.')
# time.sleep(2)
# file_path = genai.get_file(video_file.name)
#print(f'Video processing complete: {file_path.uri}')
# Create a cache with a 5 minute TTL
#cache = caching.CachedContent.create(
# model='models/gemini-1.5-flash-001',
# display_name='Alwrity Longform content', # used to identify the cache
# system_instruction=(
# 'You are an expert file analyzer , and your job is to answer '
# 'the user\'s query based on the file you have access to.'
# ),
# contents=[file_path],
# ttl=datetime.timedelta(minutes=15),
#)
# Construct a GenerativeModel which uses the created cache.
#model = genai.GenerativeModel.from_cached_content(cached_content=cache)
# Query the model
#response = model.generate_content([(
# 'SUmmarize the given file '
# 'in 10 lines '
# 'list main points')])
#print(response.usage_metadata)
#return(response.text)
return("TBD: Not implemented")

222
ToBeMigrated/chatbot_custom/README.md
View File

@@ -1,222 +0,0 @@
# Enhanced ALwrity Chatbot
An intelligent conversational AI assistant that transforms content creation, SEO analysis, and workflow automation through advanced AI-powered interactions.
## 🚀 Major Enhancements
### **Before vs After Transformation**
| **Before** | **After** |
|------------|-----------|
| Basic RAG chatbot | Intelligent workflow-driven assistant |
| Simple Q&A interface | Context-aware conversational AI |
| Manual tool selection | Smart intent analysis & tool routing |
| Static responses | Dynamic, personalized interactions |
| Limited functionality | Comprehensive content creation hub |
## 🎯 Key Improvements
### 1. **Smart Intent Analysis & Tool Routing**
*Impact: High | Complexity: High*
- **Enhanced Intent Detection**: Advanced NLP analysis of user queries
- **Confidence Scoring**: Reliability metrics for intent predictions
- **Context-Aware Routing**: Intelligent tool selection based on conversation history
- **Multi-Intent Handling**: Process complex requests with multiple objectives
### 2. **Workflow Automation Engine**
*Impact: High | Complexity: High*
- **Pre-built Workflows**: Ready-to-use processes for common tasks
- **Custom Workflow Creation**: Build personalized automation sequences
- **Progress Tracking**: Visual workflow progress with step-by-step guidance
- **Smart Step Guidance**: Context-aware assistance at each workflow stage
### 3. **Real-Time Analysis Integration**
*Impact: High | Complexity: High*
- **Instant URL Analysis**: Real-time SEO and content analysis
- **Live SEO Scoring**: Dynamic website performance metrics
- **Content Gap Detection**: Automated competitive analysis
- **Technical SEO Alerts**: Proactive issue identification
### 4. **Enhanced AI Prompts & Context System**
*Impact: High | Complexity: High*
- **Advanced System Prompts**: Specialized prompts for different content types
- **Comprehensive Context Building**: Multi-layered conversation understanding
- **Dynamic Response Structures**: Adaptive formatting based on user needs
- **Smart Follow-up Generation**: Intelligent conversation continuation
### 5. **Modular UI Components** ⭐ *NEW*
*Impact: High | Complexity: Medium*
- **Intelligent Sidebar Manager**: Organized dashboard with smart features
- **Component-Based Architecture**: Reusable UI elements for maintainability
- **Responsive Design**: Optimized interface for different screen sizes
- **State Management**: Persistent UI preferences and interactions
### 6. **Intelligent Sidebar Hub**
*Impact: Medium | Complexity: Medium*
- **Smart Dashboard**: Real-time metrics and usage analytics
- **Quick Tools Access**: One-click access to frequently used features
- **Organized Categories**: Intuitive grouping of tools and workflows
- **User Preferences**: Customizable interface and content settings
### 7. **Content Workspace Management**
*Impact: Medium | Complexity: Medium*
- **Draft System**: Save and manage work-in-progress content
- **Workspace Export**: Multiple format export options (JSON, TXT, etc.)
- **Content Ideas Generator**: AI-powered content suggestions
- **Session Management**: Persistent conversation and workspace state
## 📁 Project Structure
```
lib/chatbot_custom/
├── enhanced_alwrity_chatbot.py # Main enhanced chatbot (1,783 lines)
├── enhanced_alwrity_chatbot_modular.py # Modular version with UI components
├── ui/ # UI Components Module
│ ├── __init__.py # UI package initialization
│ └── sidebar.py # Sidebar Manager component
├── README.md # This comprehensive documentation
├── SETUP.md # Setup and configuration guide
└── ENHANCEMENT_SUMMARY.md # Detailed enhancement summary
```
## 🔧 Installation
The enhanced chatbot uses existing ALwrity dependencies. Install all requirements from the project root:
```bash
pip install -r requirements.txt
```
> **Note**: All required dependencies are already included in the main project `requirements.txt`. No additional packages needed.
## ⚙️ Environment Variables
Create a `.env` file in the project root with your API keys:
```env
OPENAI_API_KEY=your_openai_api_key
GOOGLE_API_KEY=your_google_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
SERPER_API_KEY=your_serper_api_key
```
## 🚀 Running the Chatbot
### Standard Version
```bash
streamlit run lib/chatbot_custom/enhanced_alwrity_chatbot.py
```
### Modular Version (Recommended)
```bash
streamlit run lib/chatbot_custom/enhanced_alwrity_chatbot_modular.py
```
## 💻 Usage Examples
### Smart Tool Routing
```python
# User input: "I need to analyze my competitor's website"
# System automatically:
# 1. Detects intent: competitor analysis
# 2. Routes to: website analyzer + competitor tools
# 3. Provides: comprehensive competitive analysis
```
### Real-Time Analysis Integration
```python
# User input: "Check the SEO of https://example.com"
# System provides:
# - Technical SEO analysis
# - Content gap analysis
# - On-page optimization suggestions
# - Competitor comparison
```
### Workflow Automation
```python
# Blog Creation Workflow:
# Step 1: Topic research and keyword analysis
# Step 2: Content outline generation
# Step 3: SEO optimization suggestions
# Step 4: Content creation with AI assistance
# Step 5: Final review and export options
```
## 🔄 Workflow Examples
### **Blog Creation Workflow**
1. **Research Phase**: Keyword analysis and competitor research
2. **Planning Phase**: Content outline and structure creation
3. **Creation Phase**: AI-assisted content generation
4. **Optimization Phase**: SEO enhancement and refinement
5. **Publishing Phase**: Final review and export options
### **Competitor Analysis Workflow**
1. **Discovery Phase**: Identify key competitors and URLs
2. **Analysis Phase**: Technical SEO and content analysis
3. **Comparison Phase**: Gap analysis and opportunities
4. **Strategy Phase**: Actionable recommendations
5. **Reporting Phase**: Comprehensive analysis export
## 🎨 User Experience Improvements
- **Intuitive Interface**: Clean, modern design with logical information hierarchy
- **Smart Suggestions**: Context-aware tool and workflow recommendations
- **Visual Progress Tracking**: Clear workflow progress indicators
- **Personalized Experience**: Adaptive interface based on user preferences
- **Efficient Navigation**: Quick access to frequently used features
- **Comprehensive Help**: Contextual guidance and documentation
## 📊 Performance Metrics
- **🎯 100% ALwrity Tool Integration**: Seamless access to all ALwrity features
- **⚡ 3x Workflow Efficiency**: Automated processes reduce manual steps
- **🧠 5x Smarter Responses**: Context-aware AI with advanced prompting
- **📈 Real-time Analysis**: Instant SEO and content insights
- **🎨 Enhanced UI/UX**: Modern, intuitive interface design
## 🔮 Future Enhancements
- **Multi-language Support**: Content creation in multiple languages
- **Advanced Analytics Dashboard**: Comprehensive usage and performance metrics
- **Team Collaboration Features**: Shared workspaces and collaborative editing
- **API Integration**: External tool connections and data synchronization
- **Mobile Optimization**: Enhanced mobile experience and responsive design
- **Voice Interface**: Speech-to-text and voice commands
- **Plugin System**: Extensible architecture for custom integrations
## 🤝 Contributing
We welcome contributions to enhance the ALwrity chatbot further!
### Steps to Contribute:
1. **Fork the Repository**: Create your own copy of the project
2. **Create Feature Branch**: `git checkout -b feature/AmazingFeature`
3. **Commit Changes**: `git commit -m 'Add AmazingFeature'`
4. **Push to Branch**: `git push origin feature/AmazingFeature`
5. **Open Pull Request**: Submit your changes for review
### Development Guidelines:
- Follow existing code style and conventions
- Add comprehensive documentation for new features
- Include unit tests for new functionality
- Ensure compatibility with existing ALwrity tools
## 📚 Documentation
- **[Setup Guide](SETUP.md)**: Detailed installation and configuration instructions
- **[Enhancement Summary](ENHANCEMENT_SUMMARY.md)**: Comprehensive overview of improvements
- **[ALwrity Documentation](../../README.md)**: Main project documentation
## 🆘 Support
- **GitHub Issues**: [Report bugs or request features](https://github.com/AJaySi/AI-Writer/issues)
- **Documentation**: Comprehensive guides and API references
- **Community**: Join discussions and get help from other users
---
**🎉 Experience the power of intelligent content creation with Enhanced ALwrity!**
*Transform your content workflow with AI-driven automation, real-time analysis, and intelligent assistance.*

21
ToBeMigrated/chatbot_custom/core/__init__.py
View File

@@ -1,21 +0,0 @@
"""
Core modules for the Enhanced ALwrity Chatbot.
This package contains the core functionality split into manageable modules:
- workflow_engine: Handles multi-tool workflows and automation
- tool_router: Intelligent tool routing based on user intent
- intent_analyzer: Advanced user intent analysis
- context_manager: Conversation context and state management
"""
from .workflow_engine import WorkflowEngine
from .tool_router import SmartToolRouter
from .intent_analyzer import IntentAnalyzer
from .context_manager import ContextManager
__all__ = [
'WorkflowEngine',
'SmartToolRouter',
'IntentAnalyzer',
'ContextManager'
]

413
ToBeMigrated/chatbot_custom/core/context_manager.py
View File

@@ -1,413 +0,0 @@
"""
Context Manager for Enhanced ALwrity Chatbot.
Manages conversation context, state, and user preferences with persistence.
"""
import json
import os
from datetime import datetime, timedelta
from typing import Dict, List, Any, Optional
from dataclasses import dataclass, asdict
@dataclass
class ConversationTurn:
"""Represents a single conversation turn."""
timestamp: str
user_input: str
intent: str
tools_used: List[str]
response_summary: str
satisfaction_score: Optional[float] = None
@dataclass
class UserPreferences:
"""User preferences and settings."""
content_preferences: List[str]
preferred_tone: str
preferred_length: str
industry_focus: List[str]
language: str
timezone: str
notification_settings: Dict[str, bool]
@dataclass
class WorkflowState:
"""Represents the state of an active workflow."""
workflow_id: str
workflow_name: str
current_step: int
total_steps: int
step_data: Dict[str, Any]
started_at: str
last_updated: str
is_paused: bool = False
class ContextManager:
"""Advanced conversation context and state management."""
def __init__(self, user_id: str = "default", context_file: str = None):
self.user_id = user_id
self.context_file = context_file or f"user_context_{user_id}.json"
self.context_dir = "lib/chatbot_custom/user_contexts"
# Ensure context directory exists
os.makedirs(self.context_dir, exist_ok=True)
self.context_path = os.path.join(self.context_dir, self.context_file)
# Initialize context data
self.conversation_history: List[ConversationTurn] = []
self.user_preferences: UserPreferences = UserPreferences(
content_preferences=[],
preferred_tone="professional",
preferred_length="medium",
industry_focus=[],
language="en",
timezone="UTC",
notification_settings={}
)
self.active_workflows: List[WorkflowState] = []
self.tool_usage_history: List[Dict[str, Any]] = []
self.session_data: Dict[str, Any] = {}
self.analytics_data: Dict[str, Any] = {
"total_interactions": 0,
"tools_used_count": {},
"workflows_completed": 0,
"average_session_length": 0,
"last_active": None
}
# Load existing context
self.load_context()
def add_conversation_turn(self, user_input: str, intent: str,
tools_used: List[str], response_summary: str,
satisfaction_score: Optional[float] = None):
"""Add a new conversation turn to history."""
turn = ConversationTurn(
timestamp=datetime.now().isoformat(),
user_input=user_input,
intent=intent,
tools_used=tools_used,
response_summary=response_summary,
satisfaction_score=satisfaction_score
)
self.conversation_history.append(turn)
# Keep only last 50 turns to manage memory
if len(self.conversation_history) > 50:
self.conversation_history = self.conversation_history[-50:]
# Update analytics
self.analytics_data["total_interactions"] += 1
self.analytics_data["last_active"] = datetime.now().isoformat()
# Update tool usage statistics
for tool in tools_used:
if tool in self.analytics_data["tools_used_count"]:
self.analytics_data["tools_used_count"][tool] += 1
else:
self.analytics_data["tools_used_count"][tool] = 1
self.save_context()
def update_user_preferences(self, preferences: Dict[str, Any]):
"""Update user preferences."""
for key, value in preferences.items():
if hasattr(self.user_preferences, key):
setattr(self.user_preferences, key, value)
self.save_context()
def get_recent_context(self, turns: int = 5) -> List[ConversationTurn]:
"""Get recent conversation turns for context."""
return self.conversation_history[-turns:] if self.conversation_history else []
def get_recent_topics(self, hours: int = 24) -> List[str]:
"""Get topics discussed in recent hours."""
cutoff_time = datetime.now() - timedelta(hours=hours)
recent_topics = []
for turn in self.conversation_history:
turn_time = datetime.fromisoformat(turn.timestamp)
if turn_time > cutoff_time:
# Extract topics from intent and tools used
recent_topics.append(turn.intent)
recent_topics.extend(turn.tools_used)
# Return unique topics
return list(set(recent_topics))
def get_tool_usage_history(self, limit: int = 10) -> List[str]:
"""Get recent tool usage history."""
recent_tools = []
for turn in self.conversation_history[-limit:]:
recent_tools.extend(turn.tools_used)
return recent_tools
def start_workflow(self, workflow_id: str, workflow_name: str, total_steps: int):
"""Start a new workflow."""
workflow_state = WorkflowState(
workflow_id=workflow_id,
workflow_name=workflow_name,
current_step=0,
total_steps=total_steps,
step_data={},
started_at=datetime.now().isoformat(),
last_updated=datetime.now().isoformat()
)
self.active_workflows.append(workflow_state)
self.save_context()
return workflow_state
def update_workflow_step(self, workflow_id: str, step_data: Dict[str, Any]):
"""Update workflow step data."""
for workflow in self.active_workflows:
if workflow.workflow_id == workflow_id:
workflow.current_step += 1
workflow.step_data.update(step_data)
workflow.last_updated = datetime.now().isoformat()
# Check if workflow is completed
if workflow.current_step >= workflow.total_steps:
self.complete_workflow(workflow_id)
self.save_context()
return workflow
return None
def complete_workflow(self, workflow_id: str):
"""Mark workflow as completed and remove from active workflows."""
self.active_workflows = [w for w in self.active_workflows if w.workflow_id != workflow_id]
self.analytics_data["workflows_completed"] += 1
self.save_context()
def pause_workflow(self, workflow_id: str):
"""Pause an active workflow."""
for workflow in self.active_workflows:
if workflow.workflow_id == workflow_id:
workflow.is_paused = True
workflow.last_updated = datetime.now().isoformat()
self.save_context()
return True
return False
def resume_workflow(self, workflow_id: str):
"""Resume a paused workflow."""
for workflow in self.active_workflows:
if workflow.workflow_id == workflow_id:
workflow.is_paused = False
workflow.last_updated = datetime.now().isoformat()
self.save_context()
return True
return False
def get_active_workflows(self) -> List[WorkflowState]:
"""Get all active workflows."""
return [w for w in self.active_workflows if not w.is_paused]
def get_paused_workflows(self) -> List[WorkflowState]:
"""Get all paused workflows."""
return [w for w in self.active_workflows if w.is_paused]
def set_session_data(self, key: str, value: Any):
"""Set session-specific data."""
self.session_data[key] = value
def get_session_data(self, key: str, default: Any = None) -> Any:
"""Get session-specific data."""
return self.session_data.get(key, default)
def clear_session_data(self):
"""Clear all session data."""
self.session_data.clear()
def get_context_for_intent_analysis(self) -> Dict[str, Any]:
"""Get context data for intent analysis."""
return {
"recent_topics": self.get_recent_topics(),
"user_preferences": asdict(self.user_preferences),
"active_workflows": [w.workflow_name for w in self.get_active_workflows()],
"tool_usage_history": self.get_tool_usage_history(),
"session_data": self.session_data
}
def get_user_analytics(self) -> Dict[str, Any]:
"""Get user analytics and usage statistics."""
# Calculate average session length
if self.conversation_history:
session_starts = []
current_session_start = None
for turn in self.conversation_history:
turn_time = datetime.fromisoformat(turn.timestamp)
if not current_session_start:
current_session_start = turn_time
elif (turn_time - current_session_start).total_seconds() > 3600: # 1 hour gap = new session
session_starts.append(current_session_start)
current_session_start = turn_time
if current_session_start:
session_starts.append(current_session_start)
# Most used tools
most_used_tools = sorted(
self.analytics_data["tools_used_count"].items(),
key=lambda x: x[1],
reverse=True
)[:5]
# Recent activity pattern
recent_activity = {}
for turn in self.conversation_history[-20:]: # Last 20 turns
date = turn.timestamp.split('T')[0] # Get date part
if date in recent_activity:
recent_activity[date] += 1
else:
recent_activity[date] = 1
return {
**self.analytics_data,
"most_used_tools": most_used_tools,
"recent_activity_pattern": recent_activity,
"active_workflows_count": len(self.get_active_workflows()),
"paused_workflows_count": len(self.get_paused_workflows()),
"conversation_turns": len(self.conversation_history)
}
def export_conversation_history(self, format: str = "json") -> str:
"""Export conversation history in specified format."""
if format.lower() == "json":
return json.dumps([asdict(turn) for turn in self.conversation_history], indent=2)
elif format.lower() == "txt":
text_export = []
for turn in self.conversation_history:
text_export.append(f"[{turn.timestamp}] User: {turn.user_input}")
text_export.append(f"Intent: {turn.intent}, Tools: {', '.join(turn.tools_used)}")
text_export.append(f"Response: {turn.response_summary}")
text_export.append("-" * 50)
return "\n".join(text_export)
else:
raise ValueError("Unsupported export format. Use 'json' or 'txt'.")
def cleanup_old_data(self, days: int = 30):
"""Clean up old conversation data beyond specified days."""
cutoff_date = datetime.now() - timedelta(days=days)
self.conversation_history = [
turn for turn in self.conversation_history
if datetime.fromisoformat(turn.timestamp) > cutoff_date
]
self.save_context()
def save_context(self):
"""Save context data to file."""
try:
context_data = {
"user_id": self.user_id,
"conversation_history": [asdict(turn) for turn in self.conversation_history],
"user_preferences": asdict(self.user_preferences),
"active_workflows": [asdict(workflow) for workflow in self.active_workflows],
"analytics_data": self.analytics_data,
"last_saved": datetime.now().isoformat()
}
with open(self.context_path, 'w', encoding='utf-8') as f:
json.dump(context_data, f, indent=2, ensure_ascii=False)
except Exception as e:
print(f"Error saving context: {e}")
def load_context(self):
"""Load context data from file."""
try:
if os.path.exists(self.context_path):
with open(self.context_path, 'r', encoding='utf-8') as f:
context_data = json.load(f)
# Load conversation history
self.conversation_history = [
ConversationTurn(**turn_data)
for turn_data in context_data.get("conversation_history", [])
]
# Load user preferences
prefs_data = context_data.get("user_preferences", {})
if prefs_data:
self.user_preferences = UserPreferences(**prefs_data)
# Load active workflows
self.active_workflows = [
WorkflowState(**workflow_data)
for workflow_data in context_data.get("active_workflows", [])
]
# Load analytics data
self.analytics_data.update(context_data.get("analytics_data", {}))
except Exception as e:
print(f"Error loading context: {e}")
# Continue with default values if loading fails
def reset_context(self):
"""Reset all context data (use with caution)."""
self.conversation_history.clear()
self.active_workflows.clear()
self.session_data.clear()
self.analytics_data = {
"total_interactions": 0,
"tools_used_count": {},
"workflows_completed": 0,
"average_session_length": 0,
"last_active": None
}
# Reset user preferences to defaults
self.user_preferences = UserPreferences(
content_preferences=[],
preferred_tone="professional",
preferred_length="medium",
industry_focus=[],
language="en",
timezone="UTC",
notification_settings={}
)
self.save_context()
def get_context_summary(self) -> str:
"""Get a human-readable summary of the current context."""
summary_parts = []
# Basic stats
summary_parts.append(f"Total interactions: {self.analytics_data['total_interactions']}")
summary_parts.append(f"Conversation turns: {len(self.conversation_history)}")
# Active workflows
active_workflows = self.get_active_workflows()
if active_workflows:
workflow_names = [w.workflow_name for w in active_workflows]
summary_parts.append(f"Active workflows: {', '.join(workflow_names)}")
# Recent topics
recent_topics = self.get_recent_topics(hours=6) # Last 6 hours
if recent_topics:
summary_parts.append(f"Recent topics: {', '.join(recent_topics[:5])}")
# User preferences
if self.user_preferences.content_preferences:
summary_parts.append(f"Content preferences: {', '.join(self.user_preferences.content_preferences)}")
summary_parts.append(f"Preferred tone: {self.user_preferences.preferred_tone}")
return "\n".join(summary_parts)

413
ToBeMigrated/chatbot_custom/core/intent_analyzer.py
View File

@@ -1,413 +0,0 @@
"""
Intent Analyzer for Enhanced ALwrity Chatbot.
Advanced user intent analysis with context awareness and multi-intent detection.
"""
from typing import Dict, List, Any
class IntentAnalyzer:
"""Advanced user intent analysis with context awareness."""
def __init__(self):
self.intent_keywords = {
"write": {
"keywords": ["write", "create", "generate", "compose", "draft", "author", "produce", "craft"],
"sub_intents": ["blog", "article", "story", "social", "product", "email", "copy", "script"]
},
"analyze": {
"keywords": ["analyze", "review", "check", "examine", "evaluate", "audit", "assess", "study"],
"sub_intents": ["seo", "competitor", "website", "content", "performance", "traffic", "keywords"]
},
"seo": {
"keywords": ["seo", "optimize", "rank", "keyword", "search", "meta", "visibility", "serp"],
"sub_intents": ["on_page", "technical", "content_gap", "backlinks", "local", "mobile"]
},
"social": {
"keywords": ["social", "facebook", "twitter", "linkedin", "instagram", "youtube", "tiktok"],
"sub_intents": ["post", "campaign", "engagement", "hashtags", "stories", "ads"]
},
"research": {
"keywords": ["research", "competitor", "market", "trend", "keyword", "analysis", "study"],
"sub_intents": ["competitor", "keyword", "market", "content_gap", "audience", "trends"]
},
"plan": {
"keywords": ["plan", "strategy", "calendar", "schedule", "roadmap", "organize", "structure"],
"sub_intents": ["content_calendar", "strategy", "campaign", "workflow", "editorial"]
},
"workflow": {
"keywords": ["workflow", "automate", "process", "step", "guide", "complete", "pipeline"],
"sub_intents": ["blog_creation", "seo_audit", "social_campaign", "content_strategy"]
},
"optimize": {
"keywords": ["optimize", "improve", "enhance", "boost", "increase", "maximize", "refine"],
"sub_intents": ["seo", "content", "performance", "conversion", "speed", "engagement"]
},
"learn": {
"keywords": ["learn", "how", "tutorial", "guide", "help", "explain", "teach", "show"],
"sub_intents": ["seo", "content", "social", "tools", "strategy", "best_practices"]
},
"fix": {
"keywords": ["fix", "solve", "repair", "troubleshoot", "debug", "resolve", "correct"],
"sub_intents": ["seo_issues", "technical", "content", "performance", "errors"]
}
}
self.content_type_keywords = {
"blog": ["blog", "article", "post", "content"],
"social": ["social", "post", "tweet", "update", "story"],
"email": ["email", "newsletter", "campaign", "sequence"],
"video": ["video", "youtube", "script", "transcript"],
"ad": ["ad", "advertisement", "promotion", "campaign"],
"product": ["product", "description", "listing", "catalog"],
"news": ["news", "press", "announcement", "release"],
"story": ["story", "narrative", "fiction", "creative"],
"technical": ["technical", "documentation", "manual", "guide"],
"academic": ["academic", "research", "paper", "thesis"]
}
self.urgency_keywords = {
"high": ["urgent", "asap", "immediately", "emergency", "critical", "now"],
"medium": ["soon", "quickly", "fast", "priority", "important"],
"low": ["eventually", "when possible", "later", "sometime"]
}
self.complexity_indicators = {
"high": ["comprehensive", "detailed", "complete", "full", "extensive", "thorough"],
"medium": ["moderate", "standard", "regular", "normal", "typical"],
"low": ["simple", "basic", "quick", "brief", "short", "minimal"]
}
def analyze_user_intent(self, prompt: str, context: Dict[str, Any] = None) -> Dict[str, Any]:
"""Enhanced user intent analysis with context awareness."""
prompt_lower = prompt.lower()
# Detect primary and secondary intents
detected_intents = self._detect_intents(prompt_lower)
# Detect sub-intents
sub_intents = self._detect_sub_intents(prompt_lower, detected_intents)
# Determine content types
content_types = self._detect_content_types(prompt_lower)
# Assess urgency
urgency = self._assess_urgency(prompt_lower)
# Determine complexity
complexity = self._assess_complexity(prompt_lower)
# Calculate confidence scores
confidence_scores = self._calculate_confidence_scores(prompt_lower, detected_intents)
# Context-aware enhancements
if context:
detected_intents, confidence_scores = self._enhance_with_context(
detected_intents, confidence_scores, context, prompt_lower
)
# Determine primary intent
primary_intent = self._determine_primary_intent(detected_intents, confidence_scores)
# Generate suggestions
suggested_workflows = self._suggest_workflows(detected_intents, content_types)
suggested_tools = self._suggest_tools(detected_intents, sub_intents, content_types)
return {
"primary_intent": primary_intent,
"all_intents": detected_intents,
"sub_intents": sub_intents,
"content_types": content_types,
"confidence_scores": confidence_scores,
"urgency": urgency,
"complexity": complexity,
"suggested_workflows": suggested_workflows,
"suggested_tools": suggested_tools,
"intent_strength": self._calculate_intent_strength(confidence_scores),
"multi_intent": len(detected_intents) > 1,
"context_enhanced": context is not None
}
def _detect_intents(self, prompt_lower: str) -> List[str]:
"""Detect all intents in the user prompt."""
detected_intents = []
for intent, data in self.intent_keywords.items():
matches = sum(1 for keyword in data["keywords"] if keyword in prompt_lower)
if matches > 0:
detected_intents.append(intent)
return detected_intents
def _detect_sub_intents(self, prompt_lower: str, detected_intents: List[str]) -> List[str]:
"""Detect sub-intents based on primary intents."""
sub_intents = []
for intent in detected_intents:
if intent in self.intent_keywords:
for sub_intent in self.intent_keywords[intent]["sub_intents"]:
if sub_intent in prompt_lower:
sub_intents.append(sub_intent)
return list(set(sub_intents)) # Remove duplicates
def _detect_content_types(self, prompt_lower: str) -> List[str]:
"""Detect content types mentioned in the prompt."""
content_types = []
for content_type, keywords in self.content_type_keywords.items():
if any(keyword in prompt_lower for keyword in keywords):
content_types.append(content_type)
return content_types
def _assess_urgency(self, prompt_lower: str) -> Dict[str, Any]:
"""Assess the urgency level of the request."""
urgency_level = "normal"
urgency_score = 0.5
for level, keywords in self.urgency_keywords.items():
matches = sum(1 for keyword in keywords if keyword in prompt_lower)
if matches > 0:
if level == "high":
urgency_level = "high"
urgency_score = 0.9
break
elif level == "medium" and urgency_level == "normal":
urgency_level = "medium"
urgency_score = 0.7
elif level == "low" and urgency_level == "normal":
urgency_level = "low"
urgency_score = 0.3
return {
"level": urgency_level,
"score": urgency_score,
"is_urgent": urgency_level in ["high", "medium"]
}
def _assess_complexity(self, prompt_lower: str) -> Dict[str, Any]:
"""Assess the complexity level of the request."""
complexity_level = "medium"
complexity_score = 0.5
for level, keywords in self.complexity_indicators.items():
matches = sum(1 for keyword in keywords if keyword in prompt_lower)
if matches > 0:
complexity_level = level
complexity_score = {"high": 0.9, "medium": 0.5, "low": 0.3}[level]
break
# Additional complexity indicators
word_count = len(prompt_lower.split())
if word_count > 50:
complexity_score = min(complexity_score + 0.2, 1.0)
elif word_count < 10:
complexity_score = max(complexity_score - 0.2, 0.1)
return {
"level": complexity_level,
"score": complexity_score,
"word_count": word_count
}
def _calculate_confidence_scores(self, prompt_lower: str, detected_intents: List[str]) -> Dict[str, float]:
"""Calculate confidence scores for detected intents."""
confidence_scores = {}
for intent in detected_intents:
if intent in self.intent_keywords:
keywords = self.intent_keywords[intent]["keywords"]
matches = sum(1 for keyword in keywords if keyword in prompt_lower)
confidence = matches / len(keywords)
# Boost confidence for exact matches
if intent in prompt_lower:
confidence += 0.3
# Boost confidence for multiple keyword matches
if matches > 2:
confidence += 0.2
confidence_scores[intent] = min(confidence, 1.0)
return confidence_scores
def _enhance_with_context(self, detected_intents: List[str], confidence_scores: Dict[str, float],
context: Dict[str, Any], prompt_lower: str) -> tuple:
"""Enhance intent detection with conversation context."""
enhanced_intents = detected_intents.copy()
enhanced_scores = confidence_scores.copy()
# Recent conversation topics
recent_topics = context.get("recent_topics", [])
for topic in recent_topics:
if topic.lower() in prompt_lower:
# Boost related intents
for intent in self.intent_keywords:
if topic.lower() in self.intent_keywords[intent]["keywords"]:
if intent in enhanced_scores:
enhanced_scores[intent] += 0.1
else:
enhanced_intents.append(intent)
enhanced_scores[intent] = 0.4
# User preferences
user_prefs = context.get("user_preferences", {})
if user_prefs.get("content_preferences"):
for pref in user_prefs["content_preferences"]:
if pref in prompt_lower:
# Boost content creation intents
if "write" in enhanced_scores:
enhanced_scores["write"] += 0.15
# Active workflows
active_workflows = context.get("active_workflows", [])
if active_workflows:
# Boost workflow-related intents
if "workflow" in enhanced_scores:
enhanced_scores["workflow"] += 0.2
else:
enhanced_intents.append("workflow")
enhanced_scores["workflow"] = 0.6
# Tool usage history
tool_history = context.get("tool_usage_history", [])
if tool_history:
last_tools = tool_history[-3:] # Last 3 tools
for tool in last_tools:
# Map tools to intents and boost related intents
tool_intent_mapping = {
"ai_blog_writer": "write",
"content_gap_analysis": "analyze",
"technical_seo": "seo",
"linkedin_writer": "social"
}
if tool in tool_intent_mapping:
intent = tool_intent_mapping[tool]
if intent in enhanced_scores:
enhanced_scores[intent] += 0.1
return enhanced_intents, enhanced_scores
def _determine_primary_intent(self, detected_intents: List[str], confidence_scores: Dict[str, float]) -> str:
"""Determine the primary intent from detected intents."""
if not detected_intents:
return "general"
if len(detected_intents) == 1:
return detected_intents[0]
# Return intent with highest confidence
primary_intent = max(detected_intents, key=lambda x: confidence_scores.get(x, 0))
return primary_intent
def _suggest_workflows(self, detected_intents: List[str], content_types: List[str]) -> List[str]:
"""Suggest relevant workflows based on intents and content types."""
suggested_workflows = []
# Intent-based workflow suggestions
workflow_mapping = {
"write": ["blog_creation_workflow", "content_strategy_workflow"],
"analyze": ["competitor_analysis_workflow", "seo_audit_workflow"],
"seo": ["seo_audit_workflow", "content_gap_workflow"],
"social": ["social_media_workflow", "content_repurposing_workflow"],
"plan": ["content_strategy_workflow", "editorial_calendar_workflow"]
}
for intent in detected_intents:
if intent in workflow_mapping:
suggested_workflows.extend(workflow_mapping[intent])
# Content type specific workflows
if "blog" in content_types:
suggested_workflows.append("blog_creation_workflow")
if "social" in content_types:
suggested_workflows.append("social_media_workflow")
return list(set(suggested_workflows)) # Remove duplicates
def _suggest_tools(self, detected_intents: List[str], sub_intents: List[str],
content_types: List[str]) -> List[str]:
"""Suggest relevant tools based on intents, sub-intents, and content types."""
suggested_tools = []
# Intent-based tool suggestions
tool_mapping = {
"write": ["ai_blog_writer", "story_writer", "email_writer"],
"analyze": ["content_gap_analysis", "website_analyzer", "competitor_analyzer"],
"seo": ["technical_seo", "on_page_seo", "keyword_research"],
"social": ["linkedin_writer", "facebook_writer", "social_campaign"],
"research": ["competitor_analysis", "keyword_research", "market_research"],
"optimize": ["seo_optimizer", "content_optimizer", "performance_optimizer"]
}
for intent in detected_intents:
if intent in tool_mapping:
suggested_tools.extend(tool_mapping[intent])
# Sub-intent specific tools
sub_intent_tools = {
"blog": ["ai_blog_writer", "seo_optimizer"],
"competitor": ["competitor_analysis", "content_gap_analysis"],
"technical": ["technical_seo", "performance_analyzer"],
"social": ["linkedin_writer", "facebook_writer"]
}
for sub_intent in sub_intents:
if sub_intent in sub_intent_tools:
suggested_tools.extend(sub_intent_tools[sub_intent])
# Content type specific tools
content_tools = {
"blog": ["ai_blog_writer", "seo_optimizer"],
"social": ["linkedin_writer", "facebook_writer"],
"email": ["email_writer", "campaign_creator"],
"video": ["youtube_writer", "script_generator"]
}
for content_type in content_types:
if content_type in content_tools:
suggested_tools.extend(content_tools[content_type])
return list(set(suggested_tools)) # Remove duplicates
def _calculate_intent_strength(self, confidence_scores: Dict[str, float]) -> str:
"""Calculate overall intent strength."""
if not confidence_scores:
return "weak"
max_confidence = max(confidence_scores.values())
avg_confidence = sum(confidence_scores.values()) / len(confidence_scores)
if max_confidence >= 0.8 and avg_confidence >= 0.6:
return "strong"
elif max_confidence >= 0.6 or avg_confidence >= 0.4:
return "moderate"
else:
return "weak"
def get_intent_explanation(self, intent_analysis: Dict[str, Any]) -> str:
"""Generate a human-readable explanation of the intent analysis."""
primary = intent_analysis["primary_intent"]
confidence = intent_analysis["confidence_scores"].get(primary, 0)
urgency = intent_analysis["urgency"]["level"]
complexity = intent_analysis["complexity"]["level"]
explanation = f"Primary intent: {primary} (confidence: {confidence:.2f})\n"
if intent_analysis["multi_intent"]:
other_intents = [i for i in intent_analysis["all_intents"] if i != primary]
explanation += f"Additional intents: {', '.join(other_intents)}\n"
if intent_analysis["content_types"]:
explanation += f"Content types: {', '.join(intent_analysis['content_types'])}\n"
explanation += f"Urgency: {urgency}, Complexity: {complexity}\n"
if intent_analysis["suggested_tools"]:
explanation += f"Recommended tools: {', '.join(intent_analysis['suggested_tools'][:3])}"
return explanation

285
ToBeMigrated/chatbot_custom/core/tool_router.py
View File

@@ -1,285 +0,0 @@
"""
Smart Tool Router for Enhanced ALwrity Chatbot.
Intelligent tool routing based on user intent and context.
"""
from typing import Dict, List, Any
class SmartToolRouter:
"""Intelligent tool routing based on user intent and context."""
def __init__(self):
self.tool_categories = {
"content_creation": [
"ai_blog_writer", "story_writer", "essay_writer",
"product_description", "email_writer", "news_writer"
],
"seo_tools": [
"content_gap_analysis", "technical_seo", "on_page_seo",
"competitor_analysis", "keyword_research", "meta_generator"
],
"social_media": [
"linkedin_writer", "facebook_writer", "youtube_writer",
"instagram_writer", "twitter_writer", "social_campaign"
],
"analysis": [
"website_analyzer", "content_analyzer", "competitor_analyzer",
"performance_analyzer", "seo_analyzer"
],
"planning": [
"content_calendar", "content_repurposing", "strategy_planner",
"campaign_planner", "editorial_calendar"
],
"optimization": [
"seo_optimizer", "content_optimizer", "performance_optimizer",
"conversion_optimizer", "speed_optimizer"
]
}
self.intent_tool_mapping = {
"write": ["ai_blog_writer", "story_writer", "essay_writer", "email_writer"],
"analyze": ["content_gap_analysis", "technical_seo", "website_analyzer", "competitor_analyzer"],
"seo": ["on_page_seo", "technical_seo", "content_gap_analysis", "seo_optimizer"],
"social": ["linkedin_writer", "facebook_writer", "youtube_writer", "social_campaign"],
"plan": ["content_calendar", "content_repurposing", "strategy_planner", "campaign_planner"],
"research": ["competitor_analysis", "content_gap_analysis", "keyword_research", "market_research"],
"optimize": ["seo_optimizer", "content_optimizer", "performance_optimizer"],
"create": ["ai_blog_writer", "content_creator", "social_content_creation"],
"audit": ["technical_seo", "seo_analyzer", "website_analyzer", "performance_analyzer"]
}
# Tool confidence weights based on effectiveness
self.tool_weights = {
"ai_blog_writer": 0.9,
"content_gap_analysis": 0.85,
"technical_seo": 0.8,
"linkedin_writer": 0.85,
"competitor_analysis": 0.8,
"seo_optimizer": 0.75,
"content_calendar": 0.7
}
def route_to_tools(self, user_intent: str, context: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Route user intent to relevant tools with confidence scoring."""
suggested_tools = []
user_intent_lower = user_intent.lower()
# Primary intent matching
for intent, tools in self.intent_tool_mapping.items():
if intent in user_intent_lower:
for tool in tools:
confidence = self._calculate_confidence(intent, user_intent, context)
suggested_tools.append({
"tool": tool,
"category": self._get_tool_category(tool),
"confidence": confidence,
"intent_match": intent,
"reason": f"Matches '{intent}' intent"
})
# Context-based suggestions
context_tools = self._get_context_based_suggestions(context, user_intent)
suggested_tools.extend(context_tools)
# Remove duplicates and sort by confidence
unique_tools = {}
for tool in suggested_tools:
tool_name = tool["tool"]
if tool_name not in unique_tools or tool["confidence"] > unique_tools[tool_name]["confidence"]:
unique_tools[tool_name] = tool
# Sort by confidence and return top suggestions
sorted_tools = sorted(unique_tools.values(), key=lambda x: x["confidence"], reverse=True)
return sorted_tools[:8] # Return top 8 suggestions
def _get_tool_category(self, tool: str) -> str:
"""Get category for a tool."""
for category, tools in self.tool_categories.items():
if tool in tools:
return category
return "general"
def _calculate_confidence(self, intent: str, user_text: str, context: Dict[str, Any]) -> float:
"""Calculate confidence score for tool suggestion."""
base_score = 0.5
user_text_lower = user_text.lower()
# Intent match bonus
if intent in user_text_lower:
base_score += 0.3
# Keyword bonuses
keyword_bonuses = {
"write": ["create", "generate", "compose", "draft", "author", "produce"],
"analyze": ["check", "review", "examine", "evaluate", "assess", "study"],
"seo": ["optimize", "rank", "search", "keywords", "meta", "visibility"],
"social": ["post", "share", "engage", "campaign", "viral", "audience"],
"plan": ["schedule", "organize", "strategy", "roadmap", "timeline"],
"research": ["study", "investigate", "explore", "discover", "find"]
}
if intent in keyword_bonuses:
for keyword in keyword_bonuses[intent]:
if keyword in user_text_lower:
base_score += 0.1
# Context bonuses
if context:
# Recent tool usage
recent_tools = context.get('tool_usage_history', [])[-3:]
if any(tool in user_text_lower for tool in recent_tools):
base_score += 0.15
# User preferences
user_prefs = context.get('user_preferences', {})
if user_prefs.get('industry') and user_prefs['industry'].lower() in user_text_lower:
base_score += 0.1
# Urgency bonus
urgency_keywords = ["urgent", "asap", "quickly", "fast", "immediate", "now"]
if any(keyword in user_text_lower for keyword in urgency_keywords):
base_score += 0.1
return min(base_score, 1.0)
def _get_context_based_suggestions(self, context: Dict[str, Any], user_intent: str) -> List[Dict[str, Any]]:
"""Get tool suggestions based on conversation context."""
context_tools = []
if not context:
return context_tools
# Recent tool usage patterns
recent_tools = context.get('tool_usage_history', [])
if recent_tools:
# Suggest complementary tools
last_tool = recent_tools[-1] if recent_tools else None
complementary_tools = self._get_complementary_tools(last_tool)
for tool in complementary_tools:
context_tools.append({
"tool": tool,
"category": self._get_tool_category(tool),
"confidence": 0.6,
"intent_match": "context",
"reason": f"Complements recent use of {last_tool}"
})
# Active workflows
active_workflows = context.get('active_workflows', [])
if active_workflows:
# Suggest tools for current workflow steps
for workflow in active_workflows:
workflow_tools = self._get_workflow_tools(workflow)
for tool in workflow_tools:
context_tools.append({
"tool": tool,
"category": self._get_tool_category(tool),
"confidence": 0.7,
"intent_match": "workflow",
"reason": f"Next step in {workflow} workflow"
})
# User preferences
user_prefs = context.get('user_preferences', {})
if user_prefs.get('content_preferences'):
pref_tools = self._get_preference_based_tools(user_prefs['content_preferences'])
for tool in pref_tools:
context_tools.append({
"tool": tool,
"category": self._get_tool_category(tool),
"confidence": 0.65,
"intent_match": "preference",
"reason": "Based on your content preferences"
})
return context_tools
def _get_complementary_tools(self, last_tool: str) -> List[str]:
"""Get tools that complement the last used tool."""
complementary_mapping = {
"ai_blog_writer": ["seo_optimizer", "meta_generator", "content_gap_analysis"],
"content_gap_analysis": ["ai_blog_writer", "keyword_research", "competitor_analysis"],
"technical_seo": ["on_page_seo", "content_optimizer", "performance_analyzer"],
"linkedin_writer": ["social_campaign", "content_calendar", "hashtag_research"],
"competitor_analysis": ["content_gap_analysis", "keyword_research", "strategy_planner"],
"keyword_research": ["ai_blog_writer", "content_gap_analysis", "seo_optimizer"]
}
return complementary_mapping.get(last_tool, [])
def _get_workflow_tools(self, workflow: str) -> List[str]:
"""Get tools associated with a specific workflow."""
workflow_tools = {
"blog_creation_workflow": ["keyword_research", "ai_blog_writer", "seo_optimizer"],
"competitor_analysis_workflow": ["competitor_analysis", "content_gap_analysis"],
"social_media_workflow": ["linkedin_writer", "facebook_writer", "social_campaign"],
"seo_audit_workflow": ["technical_seo", "on_page_seo", "competitor_analysis"]
}
return workflow_tools.get(workflow, [])
def _get_preference_based_tools(self, content_preferences: List[str]) -> List[str]:
"""Get tools based on user content preferences."""
preference_tools = []
for pref in content_preferences:
if pref in ["blog", "article"]:
preference_tools.extend(["ai_blog_writer", "seo_optimizer"])
elif pref in ["social", "post"]:
preference_tools.extend(["linkedin_writer", "facebook_writer"])
elif pref in ["seo", "optimization"]:
preference_tools.extend(["technical_seo", "on_page_seo"])
return list(set(preference_tools)) # Remove duplicates
def get_tool_info(self, tool_name: str) -> Dict[str, Any]:
"""Get detailed information about a specific tool."""
tool_info = {
"ai_blog_writer": {
"name": "AI Blog Writer",
"description": "Create comprehensive, SEO-optimized blog posts",
"category": "content_creation",
"use_cases": ["Blog posts", "Articles", "Long-form content"],
"estimated_time": "5-10 minutes"
},
"content_gap_analysis": {
"name": "Content Gap Analysis",
"description": "Identify content opportunities vs competitors",
"category": "seo_tools",
"use_cases": ["Competitor research", "Content strategy", "SEO planning"],
"estimated_time": "10-15 minutes"
},
"technical_seo": {
"name": "Technical SEO Crawler",
"description": "Comprehensive technical SEO audit",
"category": "seo_tools",
"use_cases": ["Site audits", "Technical issues", "Performance analysis"],
"estimated_time": "15-20 minutes"
},
"linkedin_writer": {
"name": "LinkedIn Writer",
"description": "Create professional LinkedIn content",
"category": "social_media",
"use_cases": ["LinkedIn posts", "Professional articles", "Networking content"],
"estimated_time": "3-5 minutes"
}
}
return tool_info.get(tool_name, {
"name": tool_name.replace('_', ' ').title(),
"description": f"ALwrity {tool_name.replace('_', ' ')} tool",
"category": self._get_tool_category(tool_name),
"use_cases": ["Content creation", "Analysis", "Optimization"],
"estimated_time": "5-10 minutes"
})
def get_category_tools(self, category: str) -> List[str]:
"""Get all tools in a specific category."""
return self.tool_categories.get(category, [])
def get_all_categories(self) -> List[str]:
"""Get all available tool categories."""
return list(self.tool_categories.keys())

171
ToBeMigrated/chatbot_custom/core/workflow_engine.py
View File

@@ -1,171 +0,0 @@
"""
Workflow Engine for Enhanced ALwrity Chatbot.
Handles multi-tool workflows and automation for complex content creation tasks.
"""
from typing import Dict, List, Any
class WorkflowEngine:
"""Handles multi-tool workflows and automation."""
def __init__(self):
self.workflows = {
"blog_creation_workflow": {
"name": "Complete Blog Creation",
"description": "From idea to published blog post",
"steps": [
{"tool": "keyword_research", "name": "Keyword Research"},
{"tool": "content_gap_analysis", "name": "Content Gap Analysis"},
{"tool": "blog_writing", "name": "Blog Writing"},
{"tool": "seo_optimization", "name": "SEO Optimization"},
{"tool": "meta_generation", "name": "Meta Tags Generation"}
]
},
"competitor_analysis_workflow": {
"name": "Competitor Content Strategy",
"description": "Analyze competitors and create content plan",
"steps": [
{"tool": "competitor_analysis", "name": "Competitor Analysis"},
{"tool": "content_gap_analysis", "name": "Content Gap Analysis"},
{"tool": "content_calendar", "name": "Content Calendar Creation"},
{"tool": "content_ideas", "name": "Content Ideas Generation"}
]
},
"social_media_workflow": {
"name": "Social Media Campaign",
"description": "Create comprehensive social media content",
"steps": [
{"tool": "audience_analysis", "name": "Audience Analysis"},
{"tool": "content_planning", "name": "Content Planning"},
{"tool": "social_content_creation", "name": "Social Content Creation"},
{"tool": "hashtag_research", "name": "Hashtag Research"}
]
},
"seo_audit_workflow": {
"name": "Complete SEO Audit",
"description": "Comprehensive website SEO analysis and optimization",
"steps": [
{"tool": "technical_seo", "name": "Technical SEO Analysis"},
{"tool": "on_page_seo", "name": "On-Page SEO Review"},
{"tool": "content_gap_analysis", "name": "Content Gap Analysis"},
{"tool": "competitor_seo", "name": "Competitor SEO Analysis"},
{"tool": "optimization_plan", "name": "SEO Optimization Plan"}
]
},
"content_strategy_workflow": {
"name": "Content Strategy Development",
"description": "Develop comprehensive content strategy from research to execution",
"steps": [
{"tool": "market_research", "name": "Market Research"},
{"tool": "audience_analysis", "name": "Audience Analysis"},
{"tool": "competitor_analysis", "name": "Competitor Analysis"},
{"tool": "content_pillars", "name": "Content Pillars Definition"},
{"tool": "content_calendar", "name": "Content Calendar Creation"}
]
}
}
def suggest_workflows(self, user_intent: str) -> List[Dict[str, Any]]:
"""Suggest relevant workflows based on user intent."""
relevant_workflows = []
user_intent_lower = user_intent.lower()
# Blog and content creation
if any(word in user_intent_lower for word in ['blog', 'article', 'post', 'write', 'content']):
relevant_workflows.append(self.workflows["blog_creation_workflow"])
# Competitor and market analysis
if any(word in user_intent_lower for word in ['competitor', 'analysis', 'research', 'market']):
relevant_workflows.append(self.workflows["competitor_analysis_workflow"])
# Social media
if any(word in user_intent_lower for word in ['social', 'facebook', 'linkedin', 'campaign', 'instagram', 'twitter']):
relevant_workflows.append(self.workflows["social_media_workflow"])
# SEO related
if any(word in user_intent_lower for word in ['seo', 'optimize', 'rank', 'search', 'audit']):
relevant_workflows.append(self.workflows["seo_audit_workflow"])
# Strategy and planning
if any(word in user_intent_lower for word in ['strategy', 'plan', 'roadmap', 'framework']):
relevant_workflows.append(self.workflows["content_strategy_workflow"])
return relevant_workflows
def get_workflow(self, workflow_id: str) -> Dict[str, Any]:
"""Get a specific workflow by ID."""
return self.workflows.get(workflow_id)
def get_all_workflows(self) -> Dict[str, Dict[str, Any]]:
"""Get all available workflows."""
return self.workflows
def create_custom_workflow(self, name: str, description: str, steps: List[Dict[str, str]]) -> str:
"""Create a custom workflow."""
workflow_id = f"custom_{name.lower().replace(' ', '_')}"
self.workflows[workflow_id] = {
"name": name,
"description": description,
"steps": steps,
"custom": True
}
return workflow_id
def get_workflow_progress(self, workflow_id: str, completed_steps: List[str]) -> Dict[str, Any]:
"""Get progress information for a workflow."""
workflow = self.workflows.get(workflow_id)
if not workflow:
return {"error": "Workflow not found"}
total_steps = len(workflow["steps"])
completed_count = len(completed_steps)
progress_percentage = (completed_count / total_steps) * 100 if total_steps > 0 else 0
next_step = None
if completed_count < total_steps:
next_step = workflow["steps"][completed_count]
return {
"workflow_name": workflow["name"],
"total_steps": total_steps,
"completed_steps": completed_count,
"progress_percentage": progress_percentage,
"next_step": next_step,
"is_complete": completed_count >= total_steps
}
def get_step_details(self, workflow_id: str, step_index: int) -> Dict[str, Any]:
"""Get detailed information about a specific workflow step."""
workflow = self.workflows.get(workflow_id)
if not workflow or step_index >= len(workflow["steps"]):
return {"error": "Workflow or step not found"}
step = workflow["steps"][step_index]
# Add detailed descriptions for each tool
step_descriptions = {
"keyword_research": "Research and identify target keywords for your content",
"content_gap_analysis": "Analyze competitor content to find opportunities",
"blog_writing": "Create high-quality, SEO-optimized blog content",
"seo_optimization": "Optimize content for search engines",
"meta_generation": "Generate meta titles and descriptions",
"competitor_analysis": "Analyze competitor strategies and performance",
"content_calendar": "Plan and schedule content publication",
"content_ideas": "Generate creative content ideas and topics",
"audience_analysis": "Research and define target audience",
"content_planning": "Plan content strategy and themes",
"social_content_creation": "Create platform-specific social media content",
"hashtag_research": "Research relevant hashtags for social media",
"technical_seo": "Analyze technical SEO aspects of website",
"on_page_seo": "Review and optimize on-page SEO elements"
}
return {
"tool": step["tool"],
"name": step["name"],
"description": step_descriptions.get(step["tool"], "Execute this workflow step"),
"step_number": step_index + 1,
"total_steps": len(workflow["steps"])
}

1156
ToBeMigrated/chatbot_custom/enhanced_alwrity_chatbot.py
View File

File diff suppressed because it is too large Load Diff

12
ToBeMigrated/chatbot_custom/ui/__init__.py
View File

@@ -1,12 +0,0 @@
"""
UI Components for Enhanced ALwrity Chatbot.
This package contains modular UI components for the Streamlit interface:
- sidebar: Intelligent sidebar with dashboard and quick tools
"""
from .sidebar import SidebarManager
__all__ = [
'SidebarManager'
]

396
ToBeMigrated/chatbot_custom/ui/sidebar.py
View File

@@ -1,396 +0,0 @@
"""
Sidebar Manager for Enhanced ALwrity Chatbot.
Manages the intelligent sidebar with dashboard, quick tools, and user analytics.
"""
import streamlit as st
from typing import Dict, List, Any, Optional
from datetime import datetime
class SidebarManager:
"""Manages the enhanced sidebar interface."""
def __init__(self, context_manager, workflow_engine, tool_router):
self.context_manager = context_manager
self.workflow_engine = workflow_engine
self.tool_router = tool_router
def render_sidebar(self) -> Dict[str, Any]:
"""Render the complete sidebar interface."""
sidebar_data = {}
with st.sidebar:
# Header
st.markdown("# 🚀 ALwrity Hub")
st.markdown("---")
# Dashboard section
sidebar_data.update(self._render_dashboard())
# Quick tools section
sidebar_data.update(self._render_quick_tools())
# Active workflows section
sidebar_data.update(self._render_active_workflows())
# User preferences section
sidebar_data.update(self._render_user_preferences())
# Analytics section
sidebar_data.update(self._render_analytics())
# Export/Import section
sidebar_data.update(self._render_export_import())
return sidebar_data
def _render_dashboard(self) -> Dict[str, Any]:
"""Render the dashboard section."""
st.markdown("## 📊 Dashboard")
# Get user analytics
analytics = self.context_manager.get_user_analytics()
# Key metrics in columns
col1, col2 = st.columns(2)
with col1:
st.metric(
label="Total Interactions",
value=analytics.get("total_interactions", 0)
)
st.metric(
label="Active Workflows",
value=analytics.get("active_workflows_count", 0)
)
with col2:
st.metric(
label="Workflows Completed",
value=analytics.get("workflows_completed", 0)
)
st.metric(
label="Conversation Turns",
value=analytics.get("conversation_turns", 0)
)
# Most used tools
most_used_tools = analytics.get("most_used_tools", [])
if most_used_tools:
st.markdown("**🔧 Most Used Tools:**")
for tool, count in most_used_tools[:3]:
st.markdown(f"• {tool}: {count} times")
st.markdown("---")
return {"dashboard_rendered": True}
def _render_quick_tools(self) -> Dict[str, Any]:
"""Render the quick tools section."""
st.markdown("## ⚡ Quick Tools")
quick_actions = {}
# Content creation tools
st.markdown("**✍️ Content Creation**")
col1, col2 = st.columns(2)
with col1:
if st.button("📝 Blog Writer", key="quick_blog"):
quick_actions["action"] = "blog_writer"
if st.button("📱 Social Post", key="quick_social"):
quick_actions["action"] = "social_post"
with col2:
if st.button("📧 Email Writer", key="quick_email"):
quick_actions["action"] = "email_writer"
if st.button("📖 Story Writer", key="quick_story"):
quick_actions["action"] = "story_writer"
# SEO tools
st.markdown("**🔍 SEO Tools**")
col1, col2 = st.columns(2)
with col1:
if st.button("🔧 Technical SEO", key="quick_tech_seo"):
quick_actions["action"] = "technical_seo"
if st.button("📊 Content Gap", key="quick_content_gap"):
quick_actions["action"] = "content_gap"
with col2:
if st.button("🎯 Keyword Research", key="quick_keywords"):
quick_actions["action"] = "keyword_research"
if st.button("🏆 Competitor Analysis", key="quick_competitor"):
quick_actions["action"] = "competitor_analysis"
# Analysis tools
st.markdown("**📈 Analysis**")
col1, col2 = st.columns(2)
with col1:
if st.button("🌐 Website Analyzer", key="quick_website"):
quick_actions["action"] = "website_analyzer"
if st.button("📋 On-Page SEO", key="quick_onpage"):
quick_actions["action"] = "onpage_seo"
with col2:
if st.button("🔗 URL SEO Check", key="quick_url_seo"):
quick_actions["action"] = "url_seo_check"
if st.button("📱 Social Analyzer", key="quick_social_analyzer"):
quick_actions["action"] = "social_analyzer"
st.markdown("---")
return {"quick_actions": quick_actions}
def _render_active_workflows(self) -> Dict[str, Any]:
"""Render the active workflows section."""
st.markdown("## 🔄 Active Workflows")
workflow_actions = {}
active_workflows = self.context_manager.get_active_workflows()
paused_workflows = self.context_manager.get_paused_workflows()
if active_workflows:
for workflow in active_workflows:
with st.expander(f"🟢 {workflow.workflow_name}"):
# Progress bar
progress = workflow.current_step / workflow.total_steps
st.progress(progress)
st.markdown(f"Step {workflow.current_step}/{workflow.total_steps}")
# Action buttons
col1, col2 = st.columns(2)
with col1:
if st.button("⏸️ Pause", key=f"pause_{workflow.workflow_id}"):
workflow_actions["pause"] = workflow.workflow_id
with col2:
if st.button("▶️ Continue", key=f"continue_{workflow.workflow_id}"):
workflow_actions["continue"] = workflow.workflow_id
if paused_workflows:
st.markdown("**⏸️ Paused Workflows:**")
for workflow in paused_workflows:
col1, col2 = st.columns([3, 1])
with col1:
st.markdown(f"• {workflow.workflow_name}")
with col2:
if st.button("▶️", key=f"resume_{workflow.workflow_id}"):
workflow_actions["resume"] = workflow.workflow_id
# Start new workflow
st.markdown("**🆕 Start New Workflow:**")
available_workflows = list(self.workflow_engine.workflows.keys())
selected_workflow = st.selectbox(
"Choose workflow:",
[""] + available_workflows,
key="new_workflow_select"
)
if selected_workflow and st.button("🚀 Start Workflow", key="start_new_workflow"):
workflow_actions["start"] = selected_workflow
st.markdown("---")
return {"workflow_actions": workflow_actions}
def _render_user_preferences(self) -> Dict[str, Any]:
"""Render the user preferences section."""
st.markdown("## ⚙️ Preferences")
preferences_updated = {}
current_prefs = self.context_manager.user_preferences
with st.expander("🎨 Content Preferences"):
# Tone preference
tone = st.selectbox(
"Preferred Tone:",
["professional", "casual", "friendly", "formal", "creative"],
index=["professional", "casual", "friendly", "formal", "creative"].index(
current_prefs.preferred_tone
),
key="pref_tone"
)
# Length preference
length = st.selectbox(
"Preferred Length:",
["short", "medium", "long", "comprehensive"],
index=["short", "medium", "long", "comprehensive"].index(
current_prefs.preferred_length
),
key="pref_length"
)
# Industry focus
industry_focus = st.multiselect(
"Industry Focus:",
["Technology", "Healthcare", "Finance", "Education", "Marketing",
"E-commerce", "Travel", "Food", "Fashion", "Real Estate"],
default=current_prefs.industry_focus,
key="pref_industry"
)
# Content preferences
content_prefs = st.multiselect(
"Content Types:",
["Blog Posts", "Social Media", "Email Marketing", "Technical Writing",
"Creative Writing", "SEO Content", "Product Descriptions", "News Articles"],
default=current_prefs.content_preferences,
key="pref_content_types"
)
if st.button("💾 Save Preferences", key="save_preferences"):
preferences_updated = {
"preferred_tone": tone,
"preferred_length": length,
"industry_focus": industry_focus,
"content_preferences": content_prefs
}
st.markdown("---")
return {"preferences_updated": preferences_updated}
def _render_analytics(self) -> Dict[str, Any]:
"""Render the analytics section."""
st.markdown("## 📈 Analytics")
analytics = self.context_manager.get_user_analytics()
with st.expander("📊 Usage Statistics"):
# Recent activity pattern
recent_activity = analytics.get("recent_activity_pattern", {})
if recent_activity:
st.markdown("**Recent Activity:**")
for date, count in list(recent_activity.items())[-7:]: # Last 7 days
st.markdown(f"• {date}: {count} interactions")
# Tool usage breakdown
most_used_tools = analytics.get("most_used_tools", [])
if most_used_tools:
st.markdown("**Tool Usage Breakdown:**")
for tool, count in most_used_tools:
percentage = (count / analytics.get("total_interactions", 1)) * 100
st.markdown(f"• {tool}: {count} ({percentage:.1f}%)")
# Context summary
with st.expander("🧠 Context Summary"):
context_summary = self.context_manager.get_context_summary()
st.text(context_summary)
st.markdown("---")
return {"analytics_viewed": True}
def _render_export_import(self) -> Dict[str, Any]:
"""Render the export/import section."""
st.markdown("## 💾 Data Management")
export_actions = {}
with st.expander("📤 Export Data"):
export_format = st.selectbox(
"Export Format:",
["JSON", "TXT"],
key="export_format"
)
if st.button("📥 Export Conversation History", key="export_history"):
export_actions["export"] = {
"type": "conversation_history",
"format": export_format.lower()
}
if st.button("📊 Export Analytics", key="export_analytics"):
export_actions["export"] = {
"type": "analytics",
"format": export_format.lower()
}
with st.expander("🗑️ Data Cleanup"):
cleanup_days = st.number_input(
"Keep data for (days):",
min_value=1,
max_value=365,
value=30,
key="cleanup_days"
)
if st.button("🧹 Cleanup Old Data", key="cleanup_data"):
export_actions["cleanup"] = cleanup_days
if st.button("⚠️ Reset All Data", key="reset_data"):
if st.checkbox("I understand this will delete all data", key="confirm_reset"):
export_actions["reset"] = True
return {"export_actions": export_actions}
def render_workflow_suggestions(self, intent_analysis: Dict[str, Any]) -> Optional[str]:
"""Render workflow suggestions based on intent analysis."""
suggested_workflows = intent_analysis.get("suggested_workflows", [])
if suggested_workflows:
st.sidebar.markdown("## 💡 Suggested Workflows")
for workflow in suggested_workflows[:3]: # Show top 3 suggestions
workflow_info = self.workflow_engine.get_workflow(workflow)
if workflow_info:
with st.sidebar.expander(f"🔄 {workflow_info['name']}"):
st.markdown(f"**Description:** {workflow_info['description']}")
st.markdown(f"**Steps:** {len(workflow_info['steps'])}")
if st.button(f"Start {workflow_info['name']}",
key=f"suggest_{workflow}"):
return workflow
return None
def render_tool_suggestions(self, intent_analysis: Dict[str, Any]) -> Optional[str]:
"""Render tool suggestions based on intent analysis."""
suggested_tools = intent_analysis.get("suggested_tools", [])
if suggested_tools:
st.sidebar.markdown("## 🛠️ Suggested Tools")
# Group tools by category
tool_categories = self.tool_router.tool_categories
categorized_tools = {}
for tool in suggested_tools[:6]: # Show top 6 suggestions
for category, tools in tool_categories.items():
if tool in tools:
if category not in categorized_tools:
categorized_tools[category] = []
categorized_tools[category].append(tool)
break
for category, tools in categorized_tools.items():
st.sidebar.markdown(f"**{category.title()}:**")
for tool in tools:
if st.sidebar.button(f"🚀 {tool.replace('_', ' ').title()}",
key=f"suggest_tool_{tool}"):
return tool
return None
def show_notification(self, message: str, type: str = "info"):
"""Show a notification in the sidebar."""
if type == "success":
st.sidebar.success(message)
elif type == "error":
st.sidebar.error(message)
elif type == "warning":
st.sidebar.warning(message)
else:
st.sidebar.info(message)
def get_sidebar_state(self) -> Dict[str, Any]:
"""Get current sidebar state for persistence."""
return {
"last_updated": datetime.now().isoformat(),
"active_sections": st.session_state.get("sidebar_sections", []),
"user_preferences": self.context_manager.user_preferences.__dict__
}

164
ToBeMigrated/database/__init__.py
View File

@@ -1,164 +0,0 @@
"""
Database Package for ALwrity
============================
This package provides database models and services for managing data
in the ALwrity application, including Twitter-specific functionality.
Main Components:
- models.py: Core application database models
- twitter_models.py: Twitter-specific database models
- twitter_service.py: High-level Twitter database service
- twitter_init.py: Database initialization and management utilities
Usage:
# Initialize Twitter database
from lib.database import initialize_twitter_database
initialize_twitter_database()
# Use Twitter database service
from lib.database import twitter_db
user = twitter_db.create_or_update_user(user_data)
# Use Twitter models directly
from lib.database.twitter_models import TwitterUser, Tweet
"""
# Import core models
from .models import (
SEOData, ContentType, Platform, ScheduleStatus,
ContentItem, Schedule, create_engine, init_db, get_session
)
# Import Twitter-specific components
try:
from .twitter_models import (
# Models
TwitterUser, Tweet, ScheduledTweet, TwitterAnalytics,
TweetAnalytics, EngagementData, AudienceInsight,
HashtagPerformance, ContentTemplate, TwitterSettings,
# Enums and Data Classes
TwitterAccountType, TweetType, TweetStatus, EngagementType,
AnalyticsTimeframe, ContentCategory, TwitterCredentials, TweetMetrics,
# Database functions
get_twitter_engine, init_twitter_db, get_twitter_session,
create_twitter_user, update_user_metrics, create_tweet_record,
update_tweet_metrics, calculate_virality_score, get_user_analytics_summary
)
from .twitter_service import TwitterDatabaseService, twitter_db
from .twitter_init import (
TwitterDatabaseInitializer, initialize_twitter_database,
check_twitter_database_health
)
TWITTER_AVAILABLE = True
except ImportError as e:
# Twitter components not available (missing dependencies)
TWITTER_AVAILABLE = False
print(f"Warning: Twitter database components not available: {e}")
# Package metadata
__version__ = "1.0.0"
__author__ = "ALwrity Team"
# Export main components
__all__ = [
# Core models
'SEOData', 'ContentType', 'Platform', 'ScheduleStatus',
'ContentItem', 'Schedule', 'create_engine', 'init_db', 'get_session',
# Twitter availability flag
'TWITTER_AVAILABLE',
]
# Add Twitter exports if available
if TWITTER_AVAILABLE:
__all__.extend([
# Twitter Models
'TwitterUser', 'Tweet', 'ScheduledTweet', 'TwitterAnalytics',
'TweetAnalytics', 'EngagementData', 'AudienceInsight',
'HashtagPerformance', 'ContentTemplate', 'TwitterSettings',
# Twitter Enums and Data Classes
'TwitterAccountType', 'TweetType', 'TweetStatus', 'EngagementType',
'AnalyticsTimeframe', 'ContentCategory', 'TwitterCredentials', 'TweetMetrics',
# Twitter Database Functions
'get_twitter_engine', 'init_twitter_db', 'get_twitter_session',
'create_twitter_user', 'update_user_metrics', 'create_tweet_record',
'update_tweet_metrics', 'calculate_virality_score', 'get_user_analytics_summary',
# Twitter Service
'TwitterDatabaseService', 'twitter_db',
# Twitter Initialization
'TwitterDatabaseInitializer', 'initialize_twitter_database',
'check_twitter_database_health'
])
def setup_database(db_url: str = "sqlite:///alwrity.db", twitter_db_url: str = "sqlite:///twitter_data.db"):
"""
Setup both core and Twitter databases.
Args:
db_url: URL for the core database
twitter_db_url: URL for the Twitter database
Returns:
dict: Setup results
"""
results = {
'core_db': False,
'twitter_db': False,
'errors': []
}
try:
# Initialize core database
engine = create_engine(db_url)
init_db(engine)
results['core_db'] = True
except Exception as e:
results['errors'].append(f"Core database setup failed: {e}")
if TWITTER_AVAILABLE:
try:
# Initialize Twitter database
success = initialize_twitter_database(twitter_db_url)
results['twitter_db'] = success
if not success:
results['errors'].append("Twitter database initialization failed")
except Exception as e:
results['errors'].append(f"Twitter database setup failed: {e}")
else:
results['errors'].append("Twitter database components not available")
return results
def get_database_info():
"""
Get information about available database components.
Returns:
dict: Database component information
"""
info = {
'core_models_available': True,
'twitter_models_available': TWITTER_AVAILABLE,
'version': __version__
}
if TWITTER_AVAILABLE:
try:
# Get Twitter database stats if service is available
stats = twitter_db.get_database_stats()
info['twitter_stats'] = stats
except Exception as e:
info['twitter_stats_error'] = str(e)
return info

105
ToBeMigrated/database/models.py
View File

@@ -1,105 +0,0 @@
from sqlalchemy import (
create_engine, Column, Integer, String, Text, DateTime, Enum, ForeignKey, JSON
)
from sqlalchemy.orm import declarative_base, relationship, sessionmaker
from datetime import datetime
import enum
from dataclasses import dataclass
from typing import List, Dict, Any
Base = declarative_base()
# --- DATACLASSES ---
@dataclass
class SEOData:
title: str = ""
meta_description: str = ""
keywords: List[str] = None
structured_data: Dict[str, Any] = None
def __post_init__(self):
if self.keywords is None:
self.keywords = []
if self.structured_data is None:
self.structured_data = {}
# --- ENUMS ---
class ContentType(enum.Enum):
BLOG_POST = "blog_post"
SOCIAL_MEDIA = "social_media"
VIDEO = "video"
NEWSLETTER = "newsletter"
class Platform(enum.Enum):
WEBSITE = "website"
INSTAGRAM = "instagram"
TWITTER = "twitter"
LINKEDIN = "linkedin"
FACEBOOK = "facebook"
class ScheduleStatus(enum.Enum):
SCHEDULED = "scheduled"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
CANCELLED = "cancelled"
# --- MODELS ---
class ContentItem(Base):
__tablename__ = "content_items"
id = Column(Integer, primary_key=True)
title = Column(String, nullable=False)
description = Column(Text)
content_type = Column(Enum(ContentType), nullable=False)
platforms = Column(JSON, nullable=False) # List of platforms (as strings)
publish_date = Column(DateTime, nullable=False)
status = Column(String, default="draft")
author = Column(String)
tags = Column(JSON, default=list)
notes = Column(Text)
seo_data = Column(JSON, default=dict)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
schedules = relationship("Schedule", back_populates="content_item", cascade="all, delete-orphan")
class Schedule(Base):
__tablename__ = "schedules"
id = Column(Integer, primary_key=True)
content_item_id = Column(Integer, ForeignKey("content_items.id"), nullable=False)
scheduled_time = Column(DateTime, nullable=False)
status = Column(Enum(ScheduleStatus), default=ScheduleStatus.SCHEDULED)
recurrence = Column(String) # e.g., 'none', 'daily', 'weekly'
priority = Column(Integer, default=1)
result = Column(Text)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
content_item = relationship("ContentItem", back_populates="schedules")
# --- DB INIT & SESSION ---
def get_engine(db_url="sqlite:///content_scheduler.db"):
return create_engine(db_url, echo=False)
def init_db(engine):
Base.metadata.create_all(engine)
def get_session(engine):
Session = sessionmaker(bind=engine)
return Session()
__all__ = [
'ContentItem',
'ContentType',
'Platform',
'SEOData',
'get_engine',
'get_session',
'init_db',
]

524
ToBeMigrated/database/twitter_init.py
View File

@@ -1,524 +0,0 @@
"""
Twitter Database Initialization and Migration Script
===================================================
This module provides utilities for initializing the Twitter database,
handling schema migrations, and managing database setup.
Features:
- Database initialization and table creation
- Schema migration utilities
- Data seeding for development/testing
- Database health checks and maintenance
"""
import os
import logging
from typing import Dict, Any, List, Optional
from datetime import datetime
import json
from pathlib import Path
from sqlalchemy import create_engine, text, inspect
from sqlalchemy.orm import sessionmaker
from sqlalchemy.exc import SQLAlchemyError
from .twitter_models import (
Base, TwitterUser, Tweet, ScheduledTweet, TwitterAnalytics,
TweetAnalytics, EngagementData, AudienceInsight, HashtagPerformance,
ContentTemplate, TwitterSettings, TwitterAccountType, TweetType,
TweetStatus, EngagementType, AnalyticsTimeframe, ContentCategory
)
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class TwitterDatabaseInitializer:
"""
Handles Twitter database initialization and management.
"""
def __init__(self, db_url: str = "sqlite:///twitter_data.db"):
"""Initialize the database initializer."""
self.db_url = db_url
self.engine = create_engine(db_url, echo=False)
self.SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=self.engine)
# Create database directory if using SQLite
if db_url.startswith('sqlite:///'):
db_path = db_url.replace('sqlite:///', '')
os.makedirs(os.path.dirname(os.path.abspath(db_path)), exist_ok=True)
def initialize_database(self, force_recreate: bool = False) -> bool:
"""
Initialize the Twitter database with all required tables.
Args:
force_recreate: If True, drop existing tables and recreate
Returns:
bool: True if successful, False otherwise
"""
try:
if force_recreate:
logger.info("Dropping existing tables...")
Base.metadata.drop_all(bind=self.engine)
logger.info("Creating Twitter database tables...")
Base.metadata.create_all(bind=self.engine)
# Verify tables were created
inspector = inspect(self.engine)
tables = inspector.get_table_names()
expected_tables = [
'twitter_users', 'tweets', 'scheduled_tweets', 'twitter_analytics',
'tweet_analytics', 'engagement_data', 'audience_insights',
'hashtag_performance', 'content_templates', 'twitter_settings'
]
missing_tables = [table for table in expected_tables if table not in tables]
if missing_tables:
logger.error(f"Missing tables: {missing_tables}")
return False
logger.info(f"Successfully created {len(tables)} tables")
# Create indexes for better performance
self._create_indexes()
# Seed initial data if needed
self._seed_initial_data()
logger.info("Twitter database initialization completed successfully")
return True
except Exception as e:
logger.error(f"Error initializing database: {e}")
return False
def _create_indexes(self):
"""Create database indexes for better query performance."""
try:
with self.engine.connect() as conn:
# User indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_users_user_id ON twitter_users(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_users_twitter_user_id ON twitter_users(twitter_user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_users_username ON twitter_users(username)"))
# Tweet indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweets_user_id ON tweets(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweets_status ON tweets(status)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweets_posted_at ON tweets(posted_at)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweets_tweet_id ON tweets(tweet_id)"))
# Scheduled tweet indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_scheduled_tweets_user_id ON scheduled_tweets(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_scheduled_tweets_status ON scheduled_tweets(status)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_scheduled_tweets_scheduled_time ON scheduled_tweets(scheduled_time)"))
# Analytics indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_analytics_user_id ON twitter_analytics(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_analytics_date ON twitter_analytics(date)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_twitter_analytics_timeframe ON twitter_analytics(timeframe)"))
# Tweet analytics indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweet_analytics_tweet_id ON tweet_analytics(tweet_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_tweet_analytics_recorded_at ON tweet_analytics(recorded_at)"))
# Engagement data indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_engagement_data_tweet_id ON engagement_data(tweet_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_engagement_data_occurred_at ON engagement_data(occurred_at)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_engagement_data_type ON engagement_data(engagement_type)"))
# Hashtag performance indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_hashtag_performance_user_id ON hashtag_performance(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_hashtag_performance_hashtag ON hashtag_performance(hashtag)"))
# Content template indexes
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_content_templates_user_id ON content_templates(user_id)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_content_templates_category ON content_templates(category)"))
conn.execute(text("CREATE INDEX IF NOT EXISTS idx_content_templates_is_active ON content_templates(is_active)"))
conn.commit()
logger.info("Database indexes created successfully")
except Exception as e:
logger.error(f"Error creating indexes: {e}")
def _seed_initial_data(self):
"""Seed the database with initial data for development/testing."""
try:
session = self.SessionLocal()
# Check if we already have data
if session.query(TwitterUser).count() > 0:
logger.info("Database already contains data, skipping seeding")
session.close()
return
# Create sample content templates
sample_templates = [
{
'name': 'Daily Motivation',
'description': 'Motivational quotes and thoughts',
'template_text': 'Start your day with this thought: {quote} #motivation #success',
'category': ContentCategory.PERSONAL,
'variables': ['quote'],
'default_hashtags': ['#motivation', '#success', '#mindset'],
'ai_prompt': 'Generate an inspiring motivational quote',
'ai_tone': 'inspirational',
'ai_target_audience': 'professionals and entrepreneurs'
},
{
'name': 'Tech News Share',
'description': 'Template for sharing tech news',
'template_text': 'Interesting development in {topic}: {summary} {link} #tech #innovation',
'category': ContentCategory.EDUCATIONAL,
'variables': ['topic', 'summary', 'link'],
'default_hashtags': ['#tech', '#innovation', '#technology'],
'ai_prompt': 'Summarize this tech news in an engaging way',
'ai_tone': 'informative',
'ai_target_audience': 'tech enthusiasts and professionals'
},
{
'name': 'Question Engagement',
'description': 'Template for asking engaging questions',
'template_text': 'Quick question for my followers: {question} What do you think? #community #discussion',
'category': ContentCategory.QUESTION,
'variables': ['question'],
'default_hashtags': ['#community', '#discussion', '#question'],
'ai_prompt': 'Generate an engaging question for social media',
'ai_tone': 'conversational',
'ai_target_audience': 'general audience'
},
{
'name': 'Product Update',
'description': 'Template for product announcements',
'template_text': 'Excited to share: {update} {details} #product #update #announcement',
'category': ContentCategory.PROMOTIONAL,
'variables': ['update', 'details'],
'default_hashtags': ['#product', '#update', '#announcement'],
'ai_prompt': 'Write an exciting product update announcement',
'ai_tone': 'enthusiastic',
'ai_target_audience': 'customers and prospects'
}
]
# Note: We can't create templates without a user, so we'll skip this for now
# In a real scenario, templates would be created when users are added
session.close()
logger.info("Initial data seeding completed")
except Exception as e:
logger.error(f"Error seeding initial data: {e}")
def check_database_health(self) -> Dict[str, Any]:
"""
Check the health and status of the Twitter database.
Returns:
Dict containing health check results
"""
health_status = {
'status': 'healthy',
'timestamp': datetime.utcnow().isoformat(),
'tables': {},
'indexes': {},
'issues': []
}
try:
inspector = inspect(self.engine)
# Check table existence and row counts
expected_tables = [
'twitter_users', 'tweets', 'scheduled_tweets', 'twitter_analytics',
'tweet_analytics', 'engagement_data', 'audience_insights',
'hashtag_performance', 'content_templates', 'twitter_settings'
]
session = self.SessionLocal()
for table_name in expected_tables:
if table_name in inspector.get_table_names():
# Get row count
try:
result = session.execute(text(f"SELECT COUNT(*) FROM {table_name}"))
count = result.scalar()
health_status['tables'][table_name] = {
'exists': True,
'row_count': count
}
except Exception as e:
health_status['tables'][table_name] = {
'exists': True,
'row_count': 'error',
'error': str(e)
}
health_status['issues'].append(f"Error counting rows in {table_name}: {e}")
else:
health_status['tables'][table_name] = {'exists': False}
health_status['issues'].append(f"Missing table: {table_name}")
# Check indexes
for table_name in inspector.get_table_names():
indexes = inspector.get_indexes(table_name)
health_status['indexes'][table_name] = len(indexes)
session.close()
# Set overall status
if health_status['issues']:
health_status['status'] = 'issues_found'
return health_status
except Exception as e:
health_status['status'] = 'error'
health_status['error'] = str(e)
logger.error(f"Error checking database health: {e}")
return health_status
def backup_database(self, backup_path: str) -> bool:
"""
Create a backup of the database.
Args:
backup_path: Path where to save the backup
Returns:
bool: True if successful, False otherwise
"""
try:
if not self.db_url.startswith('sqlite:///'):
logger.error("Backup currently only supported for SQLite databases")
return False
# Get the database file path
db_file = self.db_url.replace('sqlite:///', '')
if not os.path.exists(db_file):
logger.error(f"Database file not found: {db_file}")
return False
# Create backup directory if it doesn't exist
os.makedirs(os.path.dirname(backup_path), exist_ok=True)
# Copy the database file
import shutil
shutil.copy2(db_file, backup_path)
logger.info(f"Database backed up to: {backup_path}")
return True
except Exception as e:
logger.error(f"Error backing up database: {e}")
return False
def restore_database(self, backup_path: str) -> bool:
"""
Restore database from a backup.
Args:
backup_path: Path to the backup file
Returns:
bool: True if successful, False otherwise
"""
try:
if not self.db_url.startswith('sqlite:///'):
logger.error("Restore currently only supported for SQLite databases")
return False
if not os.path.exists(backup_path):
logger.error(f"Backup file not found: {backup_path}")
return False
# Get the database file path
db_file = self.db_url.replace('sqlite:///', '')
# Copy the backup file to the database location
import shutil
shutil.copy2(backup_path, db_file)
logger.info(f"Database restored from: {backup_path}")
return True
except Exception as e:
logger.error(f"Error restoring database: {e}")
return False
def migrate_schema(self, migration_scripts: List[str]) -> bool:
"""
Apply schema migration scripts.
Args:
migration_scripts: List of SQL migration scripts
Returns:
bool: True if successful, False otherwise
"""
try:
with self.engine.connect() as conn:
# Create migration tracking table if it doesn't exist
conn.execute(text("""
CREATE TABLE IF NOT EXISTS schema_migrations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
migration_name TEXT NOT NULL UNIQUE,
applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
"""))
for script in migration_scripts:
# Check if migration was already applied
result = conn.execute(text(
"SELECT COUNT(*) FROM schema_migrations WHERE migration_name = :name"
), {"name": script})
if result.scalar() == 0:
# Apply migration
logger.info(f"Applying migration: {script}")
# Read and execute migration script
script_path = Path(script)
if script_path.exists():
with open(script_path, 'r') as f:
migration_sql = f.read()
conn.execute(text(migration_sql))
# Record migration as applied
conn.execute(text(
"INSERT INTO schema_migrations (migration_name) VALUES (:name)"
), {"name": script})
else:
logger.error(f"Migration script not found: {script}")
return False
else:
logger.info(f"Migration already applied: {script}")
conn.commit()
logger.info("Schema migration completed successfully")
return True
except Exception as e:
logger.error(f"Error applying schema migration: {e}")
return False
def cleanup_old_data(self, days: int = 90) -> Dict[str, int]:
"""
Clean up old data to maintain database performance.
Args:
days: Number of days to keep data for
Returns:
Dict with cleanup statistics
"""
try:
cutoff_date = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
cutoff_date = cutoff_date.replace(day=cutoff_date.day - days)
session = self.SessionLocal()
# Count records to be deleted
old_tweet_analytics = session.query(TweetAnalytics).filter(
TweetAnalytics.recorded_at < cutoff_date
).count()
old_engagement_data = session.query(EngagementData).filter(
EngagementData.occurred_at < cutoff_date
).count()
# Delete old records
session.query(TweetAnalytics).filter(
TweetAnalytics.recorded_at < cutoff_date
).delete()
session.query(EngagementData).filter(
EngagementData.occurred_at < cutoff_date
).delete()
session.commit()
session.close()
cleanup_stats = {
'tweet_analytics_deleted': old_tweet_analytics,
'engagement_data_deleted': old_engagement_data,
'cutoff_date': cutoff_date.isoformat()
}
logger.info(f"Cleanup completed: {cleanup_stats}")
return cleanup_stats
except Exception as e:
logger.error(f"Error during cleanup: {e}")
return {'error': str(e)}
def initialize_twitter_database(db_url: str = "sqlite:///twitter_data.db", force_recreate: bool = False) -> bool:
"""
Convenience function to initialize the Twitter database.
Args:
db_url: Database URL
force_recreate: Whether to recreate existing tables
Returns:
bool: True if successful, False otherwise
"""
initializer = TwitterDatabaseInitializer(db_url)
return initializer.initialize_database(force_recreate)
def check_twitter_database_health(db_url: str = "sqlite:///twitter_data.db") -> Dict[str, Any]:
"""
Convenience function to check Twitter database health.
Args:
db_url: Database URL
Returns:
Dict with health check results
"""
initializer = TwitterDatabaseInitializer(db_url)
return initializer.check_database_health()
if __name__ == "__main__":
# Command line interface for database management
import argparse
parser = argparse.ArgumentParser(description="Twitter Database Management")
parser.add_argument("--db-url", default="sqlite:///twitter_data.db", help="Database URL")
parser.add_argument("--init", action="store_true", help="Initialize database")
parser.add_argument("--force", action="store_true", help="Force recreate tables")
parser.add_argument("--health", action="store_true", help="Check database health")
parser.add_argument("--backup", help="Create database backup")
parser.add_argument("--restore", help="Restore from backup")
parser.add_argument("--cleanup", type=int, help="Cleanup data older than N days")
args = parser.parse_args()
initializer = TwitterDatabaseInitializer(args.db_url)
if args.init:
success = initializer.initialize_database(args.force)
print(f"Database initialization: {'SUCCESS' if success else 'FAILED'}")
if args.health:
health = initializer.check_database_health()
print(json.dumps(health, indent=2))
if args.backup:
success = initializer.backup_database(args.backup)
print(f"Database backup: {'SUCCESS' if success else 'FAILED'}")
if args.restore:
success = initializer.restore_database(args.restore)
print(f"Database restore: {'SUCCESS' if success else 'FAILED'}")
if args.cleanup:
stats = initializer.cleanup_old_data(args.cleanup)
print(f"Cleanup completed: {stats}")

791
ToBeMigrated/database/twitter_models.py
View File

@@ -1,791 +0,0 @@
"""
Twitter Database Models for ALwrity
===================================
This module defines SQLAlchemy models for storing Twitter-related data including:
- User profiles and authentication
- Tweet content and metadata
- Analytics and engagement metrics
- Scheduling and automation data
- Performance tracking and insights
This allows the application to store Twitter data locally and reduce API calls
while providing rich analytics and historical data to users.
"""
from sqlalchemy import (
create_engine, Column, Integer, String, Text, DateTime, Boolean, Float,
Enum, ForeignKey, JSON, BigInteger, Index, UniqueConstraint
)
from sqlalchemy.orm import declarative_base, relationship, sessionmaker
from datetime import datetime, timedelta
import enum
from dataclasses import dataclass
from typing import List, Dict, Any, Optional
import json
Base = declarative_base()
# --- ENUMS ---
class TwitterAccountType(enum.Enum):
PERSONAL = "personal"
BUSINESS = "business"
CREATOR = "creator"
BRAND = "brand"
class TweetType(enum.Enum):
ORIGINAL = "original"
REPLY = "reply"
RETWEET = "retweet"
QUOTE_TWEET = "quote_tweet"
THREAD = "thread"
class TweetStatus(enum.Enum):
DRAFT = "draft"
SCHEDULED = "scheduled"
POSTED = "posted"
FAILED = "failed"
DELETED = "deleted"
class EngagementType(enum.Enum):
LIKE = "like"
RETWEET = "retweet"
REPLY = "reply"
QUOTE_TWEET = "quote_tweet"
BOOKMARK = "bookmark"
IMPRESSION = "impression"
PROFILE_CLICK = "profile_click"
URL_CLICK = "url_click"
HASHTAG_CLICK = "hashtag_click"
MENTION_CLICK = "mention_click"
class AnalyticsTimeframe(enum.Enum):
HOURLY = "hourly"
DAILY = "daily"
WEEKLY = "weekly"
MONTHLY = "monthly"
class ContentCategory(enum.Enum):
EDUCATIONAL = "educational"
PROMOTIONAL = "promotional"
PERSONAL = "personal"
NEWS = "news"
ENTERTAINMENT = "entertainment"
QUESTION = "question"
POLL = "poll"
THREAD = "thread"
# --- DATACLASSES ---
@dataclass
class TwitterCredentials:
"""Dataclass for Twitter API credentials"""
api_key: str = ""
api_secret: str = ""
access_token: str = ""
access_token_secret: str = ""
bearer_token: str = ""
def to_dict(self) -> Dict[str, str]:
return {
'api_key': self.api_key,
'api_secret': self.api_secret,
'access_token': self.access_token,
'access_token_secret': self.access_token_secret,
'bearer_token': self.bearer_token
}
@classmethod
def from_dict(cls, data: Dict[str, str]) -> 'TwitterCredentials':
return cls(
api_key=data.get('api_key', ''),
api_secret=data.get('api_secret', ''),
access_token=data.get('access_token', ''),
access_token_secret=data.get('access_token_secret', ''),
bearer_token=data.get('bearer_token', '')
)
@dataclass
class TweetMetrics:
"""Dataclass for tweet performance metrics"""
likes: int = 0
retweets: int = 0
replies: int = 0
quotes: int = 0
bookmarks: int = 0
impressions: int = 0
profile_clicks: int = 0
url_clicks: int = 0
hashtag_clicks: int = 0
engagement_rate: float = 0.0
reach: int = 0
def to_dict(self) -> Dict[str, Any]:
return {
'likes': self.likes,
'retweets': self.retweets,
'replies': self.replies,
'quotes': self.quotes,
'bookmarks': self.bookmarks,
'impressions': self.impressions,
'profile_clicks': self.profile_clicks,
'url_clicks': self.url_clicks,
'hashtag_clicks': self.hashtag_clicks,
'engagement_rate': self.engagement_rate,
'reach': self.reach
}
# --- MODELS ---
class TwitterUser(Base):
"""
Stores Twitter user profile information and authentication data.
This reduces API calls for user profile information.
"""
__tablename__ = "twitter_users"
id = Column(Integer, primary_key=True)
user_id = Column(String, nullable=False, unique=True) # ALwrity user ID
twitter_user_id = Column(BigInteger, nullable=False, unique=True) # Twitter user ID
username = Column(String, nullable=False, index=True) # @username
display_name = Column(String, nullable=False)
bio = Column(Text)
location = Column(String)
website = Column(String)
profile_image_url = Column(String)
banner_image_url = Column(String)
# Account metrics
followers_count = Column(Integer, default=0)
following_count = Column(Integer, default=0)
tweet_count = Column(Integer, default=0)
listed_count = Column(Integer, default=0)
# Account details
account_type = Column(Enum(TwitterAccountType), default=TwitterAccountType.PERSONAL)
verified = Column(Boolean, default=False)
protected = Column(Boolean, default=False)
created_at_twitter = Column(DateTime) # When Twitter account was created
# Authentication and API data
credentials_encrypted = Column(Text) # Encrypted JSON of TwitterCredentials
api_rate_limit_remaining = Column(Integer, default=0)
api_rate_limit_reset = Column(DateTime)
last_api_call = Column(DateTime)
# Metadata
is_active = Column(Boolean, default=True)
last_sync = Column(DateTime, default=datetime.utcnow)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
tweets = relationship("Tweet", back_populates="user", cascade="all, delete-orphan")
analytics = relationship("TwitterAnalytics", back_populates="user", cascade="all, delete-orphan")
scheduled_tweets = relationship("ScheduledTweet", back_populates="user", cascade="all, delete-orphan")
engagement_data = relationship("EngagementData", back_populates="user", cascade="all, delete-orphan")
audience_insights = relationship("AudienceInsight", back_populates="user", cascade="all, delete-orphan")
# Indexes
__table_args__ = (
Index('idx_twitter_user_username', 'username'),
Index('idx_twitter_user_sync', 'last_sync'),
Index('idx_twitter_user_active', 'is_active'),
)
class Tweet(Base):
"""
Stores tweet content, metadata, and performance data.
Includes both posted tweets and drafts.
"""
__tablename__ = "tweets"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
tweet_id = Column(BigInteger, unique=True, index=True) # Twitter tweet ID (null for drafts)
# Content
text = Column(Text, nullable=False)
hashtags = Column(JSON, default=list) # List of hashtags
mentions = Column(JSON, default=list) # List of mentioned users
urls = Column(JSON, default=list) # List of URLs in tweet
media_urls = Column(JSON, default=list) # List of media URLs
# Tweet metadata
tweet_type = Column(Enum(TweetType), default=TweetType.ORIGINAL)
status = Column(Enum(TweetStatus), default=TweetStatus.DRAFT)
category = Column(Enum(ContentCategory))
# Engagement metrics (updated periodically)
likes_count = Column(Integer, default=0)
retweets_count = Column(Integer, default=0)
replies_count = Column(Integer, default=0)
quotes_count = Column(Integer, default=0)
bookmarks_count = Column(Integer, default=0)
impressions_count = Column(Integer, default=0)
# Performance metrics
engagement_rate = Column(Float, default=0.0)
reach = Column(Integer, default=0)
click_through_rate = Column(Float, default=0.0)
# AI and generation data
ai_generated = Column(Boolean, default=False)
ai_model_used = Column(String) # Which AI model generated this
ai_prompt = Column(Text) # Original prompt used
ai_confidence_score = Column(Float) # AI confidence in content quality
generation_metadata = Column(JSON, default=dict) # Additional AI metadata
# Scheduling and posting
scheduled_for = Column(DateTime)
posted_at = Column(DateTime)
last_metrics_update = Column(DateTime)
# Thread information
thread_id = Column(String) # For grouping thread tweets
thread_position = Column(Integer) # Position in thread (1, 2, 3...)
parent_tweet_id = Column(BigInteger) # For replies
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser", back_populates="tweets")
analytics = relationship("TweetAnalytics", back_populates="tweet", cascade="all, delete-orphan")
# Indexes
__table_args__ = (
Index('idx_tweet_user_status', 'user_id', 'status'),
Index('idx_tweet_posted_at', 'posted_at'),
Index('idx_tweet_engagement', 'engagement_rate'),
Index('idx_tweet_thread', 'thread_id'),
)
class ScheduledTweet(Base):
"""
Stores scheduled tweets with automation settings.
"""
__tablename__ = "scheduled_tweets"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
tweet_id = Column(Integer, ForeignKey("tweets.id"), nullable=False)
# Scheduling details
scheduled_time = Column(DateTime, nullable=False)
timezone = Column(String, default="UTC")
recurrence_pattern = Column(String) # cron-like pattern for recurring tweets
# Automation settings
auto_optimize_time = Column(Boolean, default=False) # AI-optimize posting time
auto_add_hashtags = Column(Boolean, default=False)
auto_add_emojis = Column(Boolean, default=False)
# Status and execution
status = Column(Enum(TweetStatus), default=TweetStatus.SCHEDULED)
attempts = Column(Integer, default=0)
last_attempt = Column(DateTime)
error_message = Column(Text)
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser", back_populates="scheduled_tweets")
tweet = relationship("Tweet")
# Indexes
__table_args__ = (
Index('idx_scheduled_time', 'scheduled_time'),
Index('idx_scheduled_status', 'status'),
)
class TwitterAnalytics(Base):
"""
Stores aggregated Twitter analytics data for users.
Updated periodically to track account performance over time.
"""
__tablename__ = "twitter_analytics"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
# Time period
date = Column(DateTime, nullable=False)
timeframe = Column(Enum(AnalyticsTimeframe), nullable=False)
# Account metrics
followers_gained = Column(Integer, default=0)
followers_lost = Column(Integer, default=0)
net_follower_change = Column(Integer, default=0)
following_change = Column(Integer, default=0)
# Content metrics
tweets_posted = Column(Integer, default=0)
total_impressions = Column(Integer, default=0)
total_engagements = Column(Integer, default=0)
total_likes = Column(Integer, default=0)
total_retweets = Column(Integer, default=0)
total_replies = Column(Integer, default=0)
total_quotes = Column(Integer, default=0)
# Performance metrics
average_engagement_rate = Column(Float, default=0.0)
top_tweet_id = Column(BigInteger) # Best performing tweet
top_tweet_engagement = Column(Integer, default=0)
# Audience metrics
profile_visits = Column(Integer, default=0)
mention_count = Column(Integer, default=0)
hashtag_performance = Column(JSON, default=dict) # Top hashtags and their performance
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser", back_populates="analytics")
# Indexes
__table_args__ = (
Index('idx_analytics_user_date', 'user_id', 'date'),
Index('idx_analytics_timeframe', 'timeframe'),
UniqueConstraint('user_id', 'date', 'timeframe', name='uq_user_date_timeframe'),
)
class TweetAnalytics(Base):
"""
Stores detailed analytics for individual tweets.
Updated periodically to track tweet performance over time.
"""
__tablename__ = "tweet_analytics"
id = Column(Integer, primary_key=True)
tweet_id = Column(Integer, ForeignKey("tweets.id"), nullable=False)
# Time period
recorded_at = Column(DateTime, nullable=False, default=datetime.utcnow)
# Engagement metrics
likes = Column(Integer, default=0)
retweets = Column(Integer, default=0)
replies = Column(Integer, default=0)
quotes = Column(Integer, default=0)
bookmarks = Column(Integer, default=0)
# Reach metrics
impressions = Column(Integer, default=0)
reach = Column(Integer, default=0)
profile_clicks = Column(Integer, default=0)
# Click metrics
url_clicks = Column(Integer, default=0)
hashtag_clicks = Column(Integer, default=0)
mention_clicks = Column(Integer, default=0)
media_views = Column(Integer, default=0)
# Calculated metrics
engagement_rate = Column(Float, default=0.0)
click_through_rate = Column(Float, default=0.0)
virality_score = Column(Float, default=0.0) # Custom metric for viral potential
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
# Relationships
tweet = relationship("Tweet", back_populates="analytics")
# Indexes
__table_args__ = (
Index('idx_tweet_analytics_recorded', 'recorded_at'),
Index('idx_tweet_analytics_engagement', 'engagement_rate'),
)
class EngagementData(Base):
"""
Stores individual engagement events for detailed analysis.
"""
__tablename__ = "engagement_data"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
tweet_id = Column(Integer, ForeignKey("tweets.id"))
# Engagement details
engagement_type = Column(Enum(EngagementType), nullable=False)
engaging_user_id = Column(BigInteger) # Twitter ID of user who engaged
engaging_username = Column(String)
# Metadata
occurred_at = Column(DateTime, nullable=False)
created_at = Column(DateTime, default=datetime.utcnow)
# Relationships
user = relationship("TwitterUser", back_populates="engagement_data")
tweet = relationship("Tweet")
# Indexes
__table_args__ = (
Index('idx_engagement_user_type', 'user_id', 'engagement_type'),
Index('idx_engagement_occurred', 'occurred_at'),
)
class AudienceInsight(Base):
"""
Stores audience demographics and behavior insights.
"""
__tablename__ = "audience_insights"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
# Time period
date = Column(DateTime, nullable=False)
# Demographics (aggregated data)
top_locations = Column(JSON, default=list) # Top follower locations
age_demographics = Column(JSON, default=dict) # Age distribution
gender_demographics = Column(JSON, default=dict) # Gender distribution
language_demographics = Column(JSON, default=dict) # Language distribution
# Behavior insights
most_active_hours = Column(JSON, default=list) # When audience is most active
top_interests = Column(JSON, default=list) # Audience interests
engagement_patterns = Column(JSON, default=dict) # How audience engages
# Content preferences
preferred_content_types = Column(JSON, default=dict)
top_hashtags_used = Column(JSON, default=list)
response_rate_by_content = Column(JSON, default=dict)
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser", back_populates="audience_insights")
# Indexes
__table_args__ = (
Index('idx_audience_user_date', 'user_id', 'date'),
)
class HashtagPerformance(Base):
"""
Tracks performance of hashtags used by the user.
"""
__tablename__ = "hashtag_performance"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
# Hashtag details
hashtag = Column(String, nullable=False, index=True)
usage_count = Column(Integer, default=0)
# Performance metrics
total_impressions = Column(Integer, default=0)
total_engagements = Column(Integer, default=0)
average_engagement_rate = Column(Float, default=0.0)
# Best performing tweet with this hashtag
best_tweet_id = Column(Integer, ForeignKey("tweets.id"))
best_tweet_engagement = Column(Integer, default=0)
# Time tracking
first_used = Column(DateTime)
last_used = Column(DateTime)
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser")
best_tweet = relationship("Tweet")
# Indexes
__table_args__ = (
Index('idx_hashtag_user_performance', 'user_id', 'average_engagement_rate'),
UniqueConstraint('user_id', 'hashtag', name='uq_user_hashtag'),
)
class ContentTemplate(Base):
"""
Stores reusable tweet templates and AI prompts.
"""
__tablename__ = "content_templates"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False)
# Template details
name = Column(String, nullable=False)
description = Column(Text)
template_text = Column(Text, nullable=False)
category = Column(Enum(ContentCategory))
# Template variables and settings
variables = Column(JSON, default=list) # List of template variables
default_hashtags = Column(JSON, default=list)
suggested_times = Column(JSON, default=list) # Best times to post this type
# AI settings
ai_prompt = Column(Text) # AI prompt for generating content
ai_tone = Column(String) # Tone for AI generation
ai_target_audience = Column(String)
# Usage tracking
usage_count = Column(Integer, default=0)
last_used = Column(DateTime)
average_performance = Column(Float, default=0.0)
# Metadata
is_active = Column(Boolean, default=True)
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser")
# Indexes
__table_args__ = (
Index('idx_template_user_category', 'user_id', 'category'),
Index('idx_template_performance', 'average_performance'),
)
class TwitterSettings(Base):
"""
Stores user-specific Twitter settings and preferences.
"""
__tablename__ = "twitter_settings"
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey("twitter_users.id"), nullable=False, unique=True)
# Posting preferences
default_hashtags = Column(JSON, default=list)
auto_add_hashtags = Column(Boolean, default=False)
auto_add_emojis = Column(Boolean, default=False)
max_hashtags_per_tweet = Column(Integer, default=2)
# Scheduling preferences
preferred_posting_times = Column(JSON, default=list)
timezone = Column(String, default="UTC")
auto_optimize_timing = Column(Boolean, default=False)
# AI preferences
ai_tone_preference = Column(String, default="casual")
ai_target_audience = Column(String, default="general")
ai_creativity_level = Column(Float, default=0.7) # 0-1 scale
# Analytics preferences
analytics_frequency = Column(String, default="daily") # hourly, daily, weekly
track_competitor_hashtags = Column(JSON, default=list)
notification_preferences = Column(JSON, default=dict)
# Content preferences
content_categories = Column(JSON, default=list) # Preferred content types
avoid_topics = Column(JSON, default=list) # Topics to avoid
brand_keywords = Column(JSON, default=list) # Brand-related keywords
# Automation settings
auto_retweet_keywords = Column(JSON, default=list)
auto_like_keywords = Column(JSON, default=list)
auto_follow_back = Column(Boolean, default=False)
# Metadata
created_at = Column(DateTime, default=datetime.utcnow)
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
# Relationships
user = relationship("TwitterUser")
# --- DATABASE FUNCTIONS ---
def get_twitter_engine(db_url: str = "sqlite:///twitter_data.db"):
"""Create and return database engine for Twitter data."""
return create_engine(db_url, echo=False)
def init_twitter_db(engine):
"""Initialize Twitter database tables."""
Base.metadata.create_all(engine)
def get_twitter_session(engine):
"""Create and return database session for Twitter data."""
Session = sessionmaker(bind=engine)
return Session()
def create_twitter_user(session, user_data: Dict[str, Any]) -> TwitterUser:
"""Create a new Twitter user record."""
twitter_user = TwitterUser(
user_id=user_data['user_id'],
twitter_user_id=user_data['twitter_user_id'],
username=user_data['username'],
display_name=user_data['display_name'],
bio=user_data.get('bio', ''),
location=user_data.get('location', ''),
website=user_data.get('website', ''),
profile_image_url=user_data.get('profile_image_url', ''),
banner_image_url=user_data.get('banner_image_url', ''),
followers_count=user_data.get('followers_count', 0),
following_count=user_data.get('following_count', 0),
tweet_count=user_data.get('tweet_count', 0),
verified=user_data.get('verified', False),
protected=user_data.get('protected', False),
created_at_twitter=user_data.get('created_at_twitter'),
credentials_encrypted=user_data.get('credentials_encrypted', ''),
)
session.add(twitter_user)
session.commit()
return twitter_user
def update_user_metrics(session, user_id: int, metrics: Dict[str, Any]):
"""Update user metrics from Twitter API."""
user = session.query(TwitterUser).filter_by(id=user_id).first()
if user:
user.followers_count = metrics.get('followers_count', user.followers_count)
user.following_count = metrics.get('following_count', user.following_count)
user.tweet_count = metrics.get('tweet_count', user.tweet_count)
user.last_sync = datetime.utcnow()
session.commit()
def create_tweet_record(session, tweet_data: Dict[str, Any]) -> Tweet:
"""Create a new tweet record."""
# Handle both 'text' and 'content' field names for compatibility
text_content = tweet_data.get('text') or tweet_data.get('content')
if not text_content:
raise ValueError("Tweet must have either 'text' or 'content' field")
tweet = Tweet(
user_id=tweet_data['user_id'],
tweet_id=tweet_data.get('tweet_id'),
text=text_content,
hashtags=tweet_data.get('hashtags', []),
mentions=tweet_data.get('mentions', []),
urls=tweet_data.get('urls', []),
media_urls=tweet_data.get('media_urls', []),
tweet_type=TweetType(tweet_data.get('tweet_type', 'original')),
status=TweetStatus(tweet_data.get('status', 'draft')),
category=ContentCategory(tweet_data['category']) if tweet_data.get('category') else None,
ai_generated=tweet_data.get('ai_generated', False),
ai_model_used=tweet_data.get('ai_model_used'),
ai_prompt=tweet_data.get('ai_prompt'),
ai_confidence_score=tweet_data.get('ai_confidence_score'),
generation_metadata=tweet_data.get('generation_metadata', {}),
scheduled_for=tweet_data.get('scheduled_for'),
posted_at=tweet_data.get('posted_at'),
thread_id=tweet_data.get('thread_id'),
thread_position=tweet_data.get('thread_position'),
parent_tweet_id=tweet_data.get('parent_tweet_id'),
)
session.add(tweet)
session.commit()
return tweet
def update_tweet_metrics(session, tweet_id: int, metrics: TweetMetrics):
"""Update tweet metrics from Twitter API."""
tweet = session.query(Tweet).filter_by(id=tweet_id).first()
if tweet:
tweet.likes_count = metrics.likes
tweet.retweets_count = metrics.retweets
tweet.replies_count = metrics.replies
tweet.quotes_count = metrics.quotes
tweet.bookmarks_count = metrics.bookmarks
tweet.impressions_count = metrics.impressions
tweet.engagement_rate = metrics.engagement_rate
tweet.reach = metrics.reach
tweet.last_metrics_update = datetime.utcnow()
session.commit()
# Also create analytics record
analytics = TweetAnalytics(
tweet_id=tweet_id,
likes=metrics.likes,
retweets=metrics.retweets,
replies=metrics.replies,
quotes=metrics.quotes,
bookmarks=metrics.bookmarks,
impressions=metrics.impressions,
reach=metrics.reach,
engagement_rate=metrics.engagement_rate,
click_through_rate=metrics.url_clicks / max(metrics.impressions, 1) * 100,
virality_score=calculate_virality_score(metrics)
)
session.add(analytics)
session.commit()
def calculate_virality_score(metrics: TweetMetrics) -> float:
"""Calculate a custom virality score based on engagement metrics."""
if metrics.impressions == 0:
return 0.0
# Weight different engagement types
engagement_score = (
metrics.likes * 1.0 +
metrics.retweets * 3.0 + # Retweets are more valuable
metrics.replies * 2.0 +
metrics.quotes * 2.5 +
metrics.bookmarks * 1.5
)
# Normalize by impressions and scale
virality = (engagement_score / metrics.impressions) * 100
return min(virality, 100.0) # Cap at 100
def get_user_analytics_summary(session, user_id: int, days: int = 30) -> Dict[str, Any]:
"""Get analytics summary for a user over specified days."""
from sqlalchemy import func
start_date = datetime.utcnow() - timedelta(days=days)
# Get tweet metrics
tweet_stats = session.query(
func.count(Tweet.id).label('total_tweets'),
func.avg(Tweet.engagement_rate).label('avg_engagement'),
func.sum(Tweet.likes_count).label('total_likes'),
func.sum(Tweet.retweets_count).label('total_retweets'),
func.sum(Tweet.impressions_count).label('total_impressions')
).filter(
Tweet.user_id == user_id,
Tweet.posted_at >= start_date,
Tweet.status == TweetStatus.POSTED
).first()
# Get follower growth
user = session.query(TwitterUser).filter_by(id=user_id).first()
return {
'total_tweets': tweet_stats.total_tweets or 0,
'average_engagement_rate': float(tweet_stats.avg_engagement or 0),
'total_likes': tweet_stats.total_likes or 0,
'total_retweets': tweet_stats.total_retweets or 0,
'total_impressions': tweet_stats.total_impressions or 0,
'current_followers': user.followers_count if user else 0,
'period_days': days
}
# Export all models and functions
__all__ = [
# Models
'TwitterUser', 'Tweet', 'ScheduledTweet', 'TwitterAnalytics', 'TweetAnalytics',
'EngagementData', 'AudienceInsight', 'HashtagPerformance', 'ContentTemplate',
'TwitterSettings',
# Enums
'TwitterAccountType', 'TweetType', 'TweetStatus', 'EngagementType',
'AnalyticsTimeframe', 'ContentCategory',
# Dataclasses
'TwitterCredentials', 'TweetMetrics',
# Functions
'get_twitter_engine', 'init_twitter_db', 'get_twitter_session',
'create_twitter_user', 'update_user_metrics', 'create_tweet_record',
'update_tweet_metrics', 'calculate_virality_score', 'get_user_analytics_summary'
]

766
ToBeMigrated/database/twitter_service.py
View File

@@ -1,766 +0,0 @@
"""
Twitter Database Service Layer
=============================
This module provides high-level service functions for managing Twitter data
in the database. It acts as an interface between the application and the
database models, providing convenient methods for common operations.
Key Features:
- User profile management and synchronization
- Tweet creation, updating, and analytics tracking
- Scheduled tweet management
- Analytics data aggregation and reporting
- Hashtag performance tracking
- Audience insights management
"""
import logging
from typing import Dict, List, Any, Optional, Tuple
from datetime import datetime, timedelta
from sqlalchemy.orm import Session
from sqlalchemy import func, desc, and_, or_
import json
from cryptography.fernet import Fernet
import os
from .twitter_models import (
TwitterUser, Tweet, ScheduledTweet, TwitterAnalytics, TweetAnalytics,
EngagementData, AudienceInsight, HashtagPerformance, ContentTemplate,
TwitterSettings, TwitterCredentials, TweetMetrics,
TwitterAccountType, TweetType, TweetStatus, EngagementType,
AnalyticsTimeframe, ContentCategory,
get_twitter_engine, init_twitter_db, get_twitter_session,
create_twitter_user, update_user_metrics, create_tweet_record,
update_tweet_metrics, calculate_virality_score, get_user_analytics_summary
)
# Configure logging
logger = logging.getLogger(__name__)
class TwitterDatabaseService:
"""
High-level service for managing Twitter data in the database.
"""
def __init__(self, db_url: str = "sqlite:///twitter_data.db", encryption_key: Optional[str] = None):
"""Initialize the Twitter database service."""
self.engine = get_twitter_engine(db_url)
self.encryption_key = encryption_key or self._get_or_create_encryption_key()
self.cipher = Fernet(self.encryption_key.encode() if isinstance(self.encryption_key, str) else self.encryption_key)
# Initialize database
init_twitter_db(self.engine)
logger.info("Twitter database service initialized")
def _get_or_create_encryption_key(self) -> str:
"""Get or create encryption key for sensitive data."""
key_file = "twitter_encryption.key"
if os.path.exists(key_file):
with open(key_file, 'rb') as f:
return f.read()
else:
key = Fernet.generate_key()
with open(key_file, 'wb') as f:
f.write(key)
return key
def _encrypt_credentials(self, credentials: TwitterCredentials) -> str:
"""Encrypt Twitter credentials for secure storage."""
credentials_json = json.dumps(credentials.to_dict())
encrypted = self.cipher.encrypt(credentials_json.encode())
return encrypted.decode()
def _decrypt_credentials(self, encrypted_credentials: str) -> TwitterCredentials:
"""Decrypt Twitter credentials from storage."""
try:
decrypted = self.cipher.decrypt(encrypted_credentials.encode())
credentials_dict = json.loads(decrypted.decode())
return TwitterCredentials.from_dict(credentials_dict)
except Exception as e:
logger.error(f"Failed to decrypt credentials: {e}")
return TwitterCredentials()
def get_session(self) -> Session:
"""Get a database session."""
return get_twitter_session(self.engine)
# --- USER MANAGEMENT ---
def create_or_update_user(self, user_data: Dict[str, Any]) -> TwitterUser:
"""Create a new Twitter user or update existing one."""
session = self.get_session()
try:
# Check if user already exists
existing_user = session.query(TwitterUser).filter_by(
user_id=user_data['user_id']
).first()
if existing_user:
# Update existing user
for key, value in user_data.items():
if hasattr(existing_user, key) and key != 'id':
setattr(existing_user, key, value)
existing_user.updated_at = datetime.utcnow()
session.commit()
logger.info(f"Updated Twitter user: {existing_user.username}")
return existing_user
else:
# Create new user
twitter_user = create_twitter_user(session, user_data)
logger.info(f"Created new Twitter user: {twitter_user.username}")
return twitter_user
except Exception as e:
session.rollback()
logger.error(f"Error creating/updating user: {e}")
raise
finally:
session.close()
def save_user_credentials(self, user_id: str, credentials: TwitterCredentials) -> bool:
"""Save encrypted Twitter credentials for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if user:
encrypted_creds = self._encrypt_credentials(credentials)
user.credentials_encrypted = encrypted_creds
user.updated_at = datetime.utcnow()
session.commit()
logger.info(f"Saved credentials for user: {user.username}")
return True
else:
logger.error(f"User not found: {user_id}")
return False
except Exception as e:
session.rollback()
logger.error(f"Error saving credentials: {e}")
return False
finally:
session.close()
def get_user_credentials(self, user_id: str) -> Optional[TwitterCredentials]:
"""Get decrypted Twitter credentials for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if user and user.credentials_encrypted:
return self._decrypt_credentials(user.credentials_encrypted)
return None
except Exception as e:
logger.error(f"Error getting credentials: {e}")
return None
finally:
session.close()
def get_user_by_id(self, user_id: str) -> Optional[TwitterUser]:
"""Get Twitter user by ALwrity user ID."""
session = self.get_session()
try:
return session.query(TwitterUser).filter_by(user_id=user_id).first()
finally:
session.close()
def get_user_by_twitter_id(self, twitter_user_id: int) -> Optional[TwitterUser]:
"""Get Twitter user by Twitter user ID."""
session = self.get_session()
try:
return session.query(TwitterUser).filter_by(twitter_user_id=twitter_user_id).first()
finally:
session.close()
def update_user_profile(self, user_id: str, profile_data: Dict[str, Any]) -> bool:
"""Update user profile information from Twitter API."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if user:
update_user_metrics(session, user.id, profile_data)
logger.info(f"Updated profile for user: {user.username}")
return True
return False
except Exception as e:
session.rollback()
logger.error(f"Error updating user profile: {e}")
return False
finally:
session.close()
# --- TWEET MANAGEMENT ---
def save_tweet(self, tweet_data: Dict[str, Any]) -> Tweet:
"""Save a tweet to the database."""
session = self.get_session()
try:
tweet = create_tweet_record(session, tweet_data)
logger.info(f"Saved tweet: {tweet.id}")
return tweet
except Exception as e:
session.rollback()
logger.error(f"Error saving tweet: {e}")
raise
finally:
session.close()
def update_tweet_status(self, tweet_id: int, status: TweetStatus, twitter_tweet_id: Optional[int] = None) -> bool:
"""Update tweet status (e.g., from draft to posted)."""
session = self.get_session()
try:
tweet = session.query(Tweet).filter_by(id=tweet_id).first()
if tweet:
tweet.status = status
if twitter_tweet_id:
tweet.tweet_id = twitter_tweet_id
if status == TweetStatus.POSTED:
tweet.posted_at = datetime.utcnow()
tweet.updated_at = datetime.utcnow()
session.commit()
logger.info(f"Updated tweet {tweet_id} status to {status.value}")
return True
return False
except Exception as e:
session.rollback()
logger.error(f"Error updating tweet status: {e}")
return False
finally:
session.close()
def get_user_tweets(self, user_id: str, status: Optional[TweetStatus] = None, limit: int = 50) -> List[Tweet]:
"""Get tweets for a user, optionally filtered by status."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return []
query = session.query(Tweet).filter_by(user_id=user.id)
if status:
query = query.filter_by(status=status)
return query.order_by(desc(Tweet.created_at)).limit(limit).all()
finally:
session.close()
def get_tweet_by_id(self, tweet_id: int) -> Optional[Tweet]:
"""Get tweet by database ID."""
session = self.get_session()
try:
return session.query(Tweet).filter_by(id=tweet_id).first()
finally:
session.close()
def get_tweet_by_twitter_id(self, twitter_tweet_id: int) -> Optional[Tweet]:
"""Get tweet by Twitter tweet ID."""
session = self.get_session()
try:
return session.query(Tweet).filter_by(tweet_id=twitter_tweet_id).first()
finally:
session.close()
def update_tweet_analytics(self, tweet_id: int, metrics: TweetMetrics) -> bool:
"""Update tweet analytics from Twitter API."""
session = self.get_session()
try:
update_tweet_metrics(session, tweet_id, metrics)
logger.info(f"Updated analytics for tweet: {tweet_id}")
return True
except Exception as e:
session.rollback()
logger.error(f"Error updating tweet analytics: {e}")
return False
finally:
session.close()
def get_top_performing_tweets(self, user_id: str, days: int = 30, limit: int = 10) -> List[Tweet]:
"""Get top performing tweets for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return []
start_date = datetime.utcnow() - timedelta(days=days)
return session.query(Tweet).filter(
and_(
Tweet.user_id == user.id,
Tweet.status == TweetStatus.POSTED,
Tweet.posted_at >= start_date
)
).order_by(desc(Tweet.engagement_rate)).limit(limit).all()
finally:
session.close()
# --- SCHEDULED TWEETS ---
def schedule_tweet(self, tweet_id: int, scheduled_time: datetime, settings: Dict[str, Any] = None) -> ScheduledTweet:
"""Schedule a tweet for posting."""
session = self.get_session()
try:
tweet = session.query(Tweet).filter_by(id=tweet_id).first()
if not tweet:
raise ValueError(f"Tweet {tweet_id} not found")
scheduled_tweet = ScheduledTweet(
user_id=tweet.user_id,
tweet_id=tweet_id,
scheduled_time=scheduled_time,
timezone=settings.get('timezone', 'UTC'),
auto_optimize_time=settings.get('auto_optimize_time', False),
auto_add_hashtags=settings.get('auto_add_hashtags', False),
auto_add_emojis=settings.get('auto_add_emojis', False)
)
session.add(scheduled_tweet)
# Update tweet status
tweet.status = TweetStatus.SCHEDULED
tweet.scheduled_for = scheduled_time
session.commit()
logger.info(f"Scheduled tweet {tweet_id} for {scheduled_time}")
return scheduled_tweet
except Exception as e:
session.rollback()
logger.error(f"Error scheduling tweet: {e}")
raise
finally:
session.close()
def get_pending_scheduled_tweets(self, user_id: Optional[str] = None) -> List[ScheduledTweet]:
"""Get tweets scheduled for posting."""
session = self.get_session()
try:
query = session.query(ScheduledTweet).filter(
and_(
ScheduledTweet.status == TweetStatus.SCHEDULED,
ScheduledTweet.scheduled_time <= datetime.utcnow()
)
)
if user_id:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if user:
query = query.filter_by(user_id=user.id)
return query.order_by(ScheduledTweet.scheduled_time).all()
finally:
session.close()
def mark_scheduled_tweet_posted(self, scheduled_tweet_id: int, twitter_tweet_id: int) -> bool:
"""Mark a scheduled tweet as posted."""
session = self.get_session()
try:
scheduled_tweet = session.query(ScheduledTweet).filter_by(id=scheduled_tweet_id).first()
if scheduled_tweet:
scheduled_tweet.status = TweetStatus.POSTED
# Update the associated tweet
tweet = session.query(Tweet).filter_by(id=scheduled_tweet.tweet_id).first()
if tweet:
tweet.status = TweetStatus.POSTED
tweet.tweet_id = twitter_tweet_id
tweet.posted_at = datetime.utcnow()
session.commit()
logger.info(f"Marked scheduled tweet {scheduled_tweet_id} as posted")
return True
return False
except Exception as e:
session.rollback()
logger.error(f"Error marking scheduled tweet as posted: {e}")
return False
finally:
session.close()
# --- ANALYTICS ---
def save_daily_analytics(self, user_id: str, analytics_data: Dict[str, Any]) -> TwitterAnalytics:
"""Save daily analytics data for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
raise ValueError(f"User {user_id} not found")
# Check if analytics for today already exist
today = datetime.utcnow().date()
existing = session.query(TwitterAnalytics).filter(
and_(
TwitterAnalytics.user_id == user.id,
func.date(TwitterAnalytics.date) == today,
TwitterAnalytics.timeframe == AnalyticsTimeframe.DAILY
)
).first()
if existing:
# Update existing record
for key, value in analytics_data.items():
if hasattr(existing, key):
setattr(existing, key, value)
existing.updated_at = datetime.utcnow()
session.commit()
return existing
else:
# Create new record
analytics = TwitterAnalytics(
user_id=user.id,
date=datetime.utcnow(),
timeframe=AnalyticsTimeframe.DAILY,
**analytics_data
)
session.add(analytics)
session.commit()
logger.info(f"Saved daily analytics for user: {user.username}")
return analytics
except Exception as e:
session.rollback()
logger.error(f"Error saving analytics: {e}")
raise
finally:
session.close()
def get_analytics_summary(self, user_id: str, days: int = 30) -> Dict[str, Any]:
"""Get comprehensive analytics summary for a user."""
session = self.get_session()
try:
return get_user_analytics_summary(session, user_id, days)
finally:
session.close()
def get_engagement_trends(self, user_id: str, days: int = 30) -> List[Dict[str, Any]]:
"""Get engagement trends over time."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return []
start_date = datetime.utcnow() - timedelta(days=days)
analytics = session.query(TwitterAnalytics).filter(
and_(
TwitterAnalytics.user_id == user.id,
TwitterAnalytics.date >= start_date,
TwitterAnalytics.timeframe == AnalyticsTimeframe.DAILY
)
).order_by(TwitterAnalytics.date).all()
return [
{
'date': a.date.isoformat(),
'engagement_rate': a.average_engagement_rate,
'total_engagements': a.total_engagements,
'impressions': a.total_impressions,
'followers_change': a.net_follower_change
}
for a in analytics
]
finally:
session.close()
# --- HASHTAG PERFORMANCE ---
def track_hashtag_performance(self, user_id: str, hashtag: str, tweet_id: int, engagement_metrics: Dict[str, Any]) -> bool:
"""Track performance of a hashtag."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return False
# Get or create hashtag performance record
hashtag_perf = session.query(HashtagPerformance).filter(
and_(
HashtagPerformance.user_id == user.id,
HashtagPerformance.hashtag == hashtag
)
).first()
if hashtag_perf:
# Update existing record
hashtag_perf.usage_count += 1
hashtag_perf.total_impressions += engagement_metrics.get('impressions', 0)
hashtag_perf.total_engagements += engagement_metrics.get('engagements', 0)
hashtag_perf.last_used = datetime.utcnow()
# Update average engagement rate
if hashtag_perf.usage_count > 0:
hashtag_perf.average_engagement_rate = (
hashtag_perf.total_engagements / hashtag_perf.total_impressions * 100
if hashtag_perf.total_impressions > 0 else 0
)
# Update best performing tweet if this one is better
current_engagement = engagement_metrics.get('engagements', 0)
if current_engagement > hashtag_perf.best_tweet_engagement:
hashtag_perf.best_tweet_id = tweet_id
hashtag_perf.best_tweet_engagement = current_engagement
else:
# Create new record
hashtag_perf = HashtagPerformance(
user_id=user.id,
hashtag=hashtag,
usage_count=1,
total_impressions=engagement_metrics.get('impressions', 0),
total_engagements=engagement_metrics.get('engagements', 0),
average_engagement_rate=(
engagement_metrics.get('engagements', 0) /
max(engagement_metrics.get('impressions', 1), 1) * 100
),
best_tweet_id=tweet_id,
best_tweet_engagement=engagement_metrics.get('engagements', 0),
first_used=datetime.utcnow(),
last_used=datetime.utcnow()
)
session.add(hashtag_perf)
session.commit()
return True
except Exception as e:
session.rollback()
logger.error(f"Error tracking hashtag performance: {e}")
return False
finally:
session.close()
def get_top_hashtags(self, user_id: str, limit: int = 10) -> List[HashtagPerformance]:
"""Get top performing hashtags for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return []
return session.query(HashtagPerformance).filter_by(
user_id=user.id
).order_by(desc(HashtagPerformance.average_engagement_rate)).limit(limit).all()
finally:
session.close()
# --- CONTENT TEMPLATES ---
def save_content_template(self, user_id: str, template_data: Dict[str, Any]) -> ContentTemplate:
"""Save a content template."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
raise ValueError(f"User {user_id} not found")
template = ContentTemplate(
user_id=user.id,
name=template_data['name'],
description=template_data.get('description', ''),
template_text=template_data['template_text'],
category=ContentCategory(template_data['category']) if template_data.get('category') else None,
variables=template_data.get('variables', []),
default_hashtags=template_data.get('default_hashtags', []),
ai_prompt=template_data.get('ai_prompt', ''),
ai_tone=template_data.get('ai_tone', ''),
ai_target_audience=template_data.get('ai_target_audience', '')
)
session.add(template)
session.commit()
logger.info(f"Saved content template: {template.name}")
return template
except Exception as e:
session.rollback()
logger.error(f"Error saving content template: {e}")
raise
finally:
session.close()
def get_user_templates(self, user_id: str, category: Optional[ContentCategory] = None) -> List[ContentTemplate]:
"""Get content templates for a user."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return []
query = session.query(ContentTemplate).filter(
and_(
ContentTemplate.user_id == user.id,
ContentTemplate.is_active == True
)
)
if category:
query = query.filter_by(category=category)
return query.order_by(desc(ContentTemplate.average_performance)).all()
finally:
session.close()
# --- SETTINGS ---
def save_user_settings(self, user_id: str, settings_data: Dict[str, Any]) -> TwitterSettings:
"""Save user Twitter settings."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
raise ValueError(f"User {user_id} not found")
# Check if settings already exist
existing_settings = session.query(TwitterSettings).filter_by(user_id=user.id).first()
if existing_settings:
# Update existing settings
for key, value in settings_data.items():
if hasattr(existing_settings, key):
setattr(existing_settings, key, value)
existing_settings.updated_at = datetime.utcnow()
session.commit()
return existing_settings
else:
# Create new settings
settings = TwitterSettings(
user_id=user.id,
**settings_data
)
session.add(settings)
session.commit()
logger.info(f"Saved settings for user: {user.username}")
return settings
except Exception as e:
session.rollback()
logger.error(f"Error saving user settings: {e}")
raise
finally:
session.close()
def get_user_settings(self, user_id: str) -> Optional[TwitterSettings]:
"""Get user Twitter settings."""
session = self.get_session()
try:
user = session.query(TwitterUser).filter_by(user_id=user_id).first()
if not user:
return None
return session.query(TwitterSettings).filter_by(user_id=user.id).first()
finally:
session.close()
# --- UTILITY METHODS ---
def cleanup_old_data(self, days_old: int = 30) -> Dict[str, int]:
"""
Clean up old data to maintain database performance.
Args:
days_old: Number of days old data to keep
Returns:
Dictionary with cleanup statistics
"""
try:
cutoff_date = datetime.utcnow() - timedelta(days=days_old)
with self.get_session() as session:
# Clean up old analytics data
old_analytics = session.query(TwitterAnalytics).filter(
TwitterAnalytics.created_at < cutoff_date
).count()
session.query(TwitterAnalytics).filter(
TwitterAnalytics.created_at < cutoff_date
).delete()
# Clean up old tweet analytics
old_tweet_analytics = session.query(TweetAnalytics).filter(
TweetAnalytics.created_at < cutoff_date
).count()
session.query(TweetAnalytics).filter(
TweetAnalytics.created_at < cutoff_date
).delete()
session.commit()
stats = {
'old_analytics_removed': old_analytics,
'old_tweet_analytics_removed': old_tweet_analytics,
'cutoff_date': cutoff_date.isoformat()
}
logger.info(f"Cleaned up old data: {stats}")
return stats
except Exception as e:
logger.error(f"Error cleaning up old data: {e}")
return {'error': str(e)}
def get_database_stats(self) -> Dict[str, int]:
"""
Get database statistics.
Returns:
Dictionary with database statistics
"""
try:
with self.get_session() as session:
stats = {
'total_users': session.query(TwitterUser).count(),
'total_tweets': session.query(Tweet).count(),
'posted_tweets': session.query(Tweet).filter(
Tweet.status == TweetStatus.POSTED
).count(),
'scheduled_tweets': session.query(ScheduledTweet).filter(
ScheduledTweet.status == TweetStatus.SCHEDULED
).count(),
'total_analytics_records': session.query(TwitterAnalytics).count(),
'total_templates': session.query(ContentTemplate).count()
}
return stats
except Exception as e:
logger.error(f"Error getting database stats: {e}")
return {'error': str(e)}
def close(self):
"""
Close database connections and clean up resources.
"""
try:
if hasattr(self, 'engine') and self.engine:
self.engine.dispose()
logger.info("Database connections closed successfully")
except Exception as e:
logger.error(f"Error closing database connections: {e}")
# Create a global instance for easy access
twitter_db = TwitterDatabaseService()
# Export the service and key functions
__all__ = [
'TwitterDatabaseService',
'twitter_db'
]

99
ToBeMigrated/utils/ai_research.py
View File

@@ -1,99 +0,0 @@
"""AI research module for topic analysis and research."""
import asyncio
from typing import Dict, Any
from loguru import logger
import sys
from ..web_crawlers.async_web_crawler import AsyncWebCrawlerService
from ..gpt_providers.text_generation.main_text_generation import llm_text_gen
# Configure logger
logger.remove()
logger.add(
"logs/ai_research.log",
rotation="500 MB",
retention="10 days",
level="DEBUG",
format="{time:YYYY-MM-DD HH:mm:ss} | {level} | {message}"
)
logger.add(
sys.stdout,
level="INFO",
format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{message}</cyan>"
)
def research_topic(topic: str) -> Dict[str, Any]:
"""
Research a topic using web crawling and AI analysis.
Args:
topic (str): The topic to research
Returns:
Dict[str, Any]: Research results including overview, findings, and recommendations
"""
try:
logger.info(f"[research_topic] Starting research for topic: {topic}")
# Initialize web crawler
async def analyze_topic():
async with AsyncWebCrawlerService() as crawler:
# Perform web research
search_results = await crawler.crawl_website(topic)
if not search_results.get('success'):
return {
'success': False,
'error': search_results.get('error', 'Research failed')
}
# Analyze content with LLM
analysis = await crawler.analyze_content_with_llm(
search_results['content'],
api_key=None, # Should be passed from config
gpt_provider="google" # Should be configurable
)
# Structure the response
return {
'success': True,
'data': {
'research': {
'overview': {
'topic': topic,
'scope': analysis.get('topics', []),
'methodology': 'Web crawling and AI analysis'
},
'data_quality': {
'is_reliable': bool(analysis.get('seo_score', 0) > 0.7)
},
'analysis_quality': {
'is_thorough': bool(len(analysis.get('key_insights', [])) > 5)
},
'recommendations': analysis.get('recommendations', []),
'next_steps': analysis.get('priority_areas', [])
}
}
}
# Run the async analysis
results = asyncio.run(analyze_topic())
if not results.get('success'):
error_msg = results.get('error', 'Research failed')
logger.error(f"[research_topic] Research failed: {error_msg}")
return {
'success': False,
'error': error_msg
}
logger.info("[research_topic] Research completed successfully")
return results
except Exception as e:
error_msg = f"Research failed: {str(e)}"
logger.error(f"[research_topic] {error_msg}")
return {
'success': False,
'error': str(e)
}

54
ToBeMigrated/utils/api_key_manager/__init__.py
View File

@@ -1,54 +0,0 @@
"""API Key Manager package for ALwrity."""
from .manager import APIKeyManager
from .api_key_manager import render, check_onboarding_completion, get_onboarding_status, reset_onboarding
from .onboarding_progress import (
OnboardingProgress,
get_onboarding_progress,
render_progress_indicator,
render_resume_message,
StepStatus,
StepData
)
from .validation import check_all_api_keys
from .components.base import (
render_step_indicator,
render_navigation_buttons,
render_step_validation,
render_resume_options
)
# Export all public components
__all__ = [
# Main classes
'APIKeyManager',
'OnboardingProgress',
'StepStatus',
'StepData',
# Main functions
'render',
'check_onboarding_completion',
'get_onboarding_status',
'reset_onboarding',
'get_onboarding_progress',
# UI components
'render_progress_indicator',
'render_resume_message',
'render_step_indicator',
'render_navigation_buttons',
'render_step_validation',
'render_resume_options',
# Validation
'check_all_api_keys'
]
# Version information
__version__ = "2.0.0"
__author__ = "ALwrity Team"
__description__ = "Comprehensive API key management and onboarding system for ALwrity"
# Note: FastAPI endpoints have been moved to the backend/ directory
# for better separation of concerns and enterprise architecture.

42
ToBeMigrated/utils/api_key_manager/ai_research.py
View File

@@ -1,42 +0,0 @@
"""AI research functionality for API key manager."""
from loguru import logger
import asyncio
from typing import Dict, Any, Optional
async def research_topic(topic: str, api_keys: Dict[str, str]) -> Dict[str, Any]:
"""
Research a topic using available AI services.
Args:
topic (str): The topic to research
api_keys (Dict[str, str]): Dictionary of API keys for different services
Returns:
Dict[str, Any]: Research results and metadata
"""
try:
logger.info(f"Starting research on topic: {topic}")
# TODO: Implement actual research functionality using available API keys
# This is a placeholder implementation
results = {
"topic": topic,
"status": "success",
"data": {
"summary": f"Research summary for {topic}",
"key_points": ["Point 1", "Point 2", "Point 3"],
"sources": ["Source 1", "Source 2"]
}
}
logger.info("Research completed successfully")
return results
except Exception as e:
logger.error(f"Error during research: {str(e)}")
return {
"topic": topic,
"status": "error",
"error": str(e)
}

178
ToBeMigrated/utils/api_key_manager/components/README.md
View File

@@ -1,178 +0,0 @@
# ALwrity Setup Components Guide
## Overview
The ALwrity Setup Components are the building blocks that guide you through setting up your content creation environment. Each component is designed to help you configure specific aspects of ALwrity for optimal content creation.
## Core Components
### 1. Website Setup (`website_setup.py`)
**Purpose**: Configure your website's basic information and analyze its current state
**Features**:
- **URL Configuration**: Set up your website's URL
- **Analysis Options**:
- Basic Analysis: Quick overview of your website
- Full Analysis with SEO: Comprehensive website and SEO analysis
- **Analysis Results**:
- Basic Metrics: Status, content type, title, meta description
- Content Analysis: Word count, headings, images, links
- SEO Analysis: SEO score, meta tags, content quality
- Technical SEO: Mobile friendliness, page speed, technical issues
- Strategy Recommendations: Actionable improvements
### 2. AI Research Setup (`ai_research_setup.py`)
**Purpose**: Configure AI-powered research tools for content creation
**Features**:
- **Traditional Search**:
- SerpAPI integration for real-time search results
- Access to structured data and knowledge graphs
- News articles and related questions
- **AI Deep Research**:
- Tavily AI for semantic understanding
- Metaphor/Exa for neural search capabilities
- Advanced research features
### 3. AI Providers (`ai_providers.py`)
**Purpose**: Set up your preferred AI content generation services
**Supported Providers**:
- **OpenAI (GPT models)**
- Advanced language models
- Creative content generation
- Context-aware responses
- **Google (Gemini Pro)**
- Balanced content creation
- Factual accuracy
- Multilingual support
- **Anthropic (Claude)**
- Professional writing
- Detailed analysis
- Ethical considerations
- **DeepSeek**
- Technical content
- Specialized knowledge
- Efficient processing
### 4. Personalization Setup (`personalization_setup.py`)
**Purpose**: Customize your content creation experience
**Features**:
- **Writing Style**:
- Tone preferences
- Voice settings
- Content structure
- **Brand Configuration**:
- Brand voice
- Style guidelines
- Content templates
### 5. ALwrity Integrations (`alwrity_integrations.py`)
**Purpose**: Connect additional tools and services
**Features**:
- **Third-party Services**:
- Analytics integration
- Social media tools
- Content management systems
- **Workflow Automation**:
- Publishing tools
- Content scheduling
- Distribution channels
### 6. Final Setup (`final_setup.py`)
**Purpose**: Complete and verify your configuration
**Features**:
- **Configuration Review**:
- Settings verification
- Connection testing
- Setup completion
- **Validation**:
- API key verification
- Service connectivity
- System readiness
## Base Components
### 1. Navigation (`base.py`)
**Purpose**: Provide consistent navigation throughout the setup process
**Features**:
- Step indicators
- Navigation buttons
- Progress tracking
- Back/forward controls
## How to Use the Components
### 1. Starting the Setup
1. Launch ALwrity
2. Navigate to the Setup section
3. Follow the guided wizard process
### 2. Component Navigation
- Use the step indicator to track progress
- Navigate between components using buttons
- Save progress automatically
- Return to previous steps if needed
### 3. Configuration Process
1. **Enter Information**: Fill in required details
2. **Verify Settings**: Review your inputs
3. **Test Connections**: Ensure everything works
4. **Complete Setup**: Finalize your configuration
## Best Practices
### 1. Before Setup
- Gather all necessary API keys
- Review provider documentation
- Plan your configuration
- Backup existing settings
### 2. During Setup
- Follow the wizard steps
- Verify each configuration
- Test connections
- Save progress regularly
### 3. After Setup
- Review all settings
- Test functionality
- Document configurations
- Monitor usage
## Troubleshooting
### 1. Common Issues
- Invalid API keys
- Connection problems
- Configuration errors
- Setup interruptions
### 2. Solutions
- Key verification
- Connection testing
- Error logging
- Support resources
## Need Help?
If you encounter any issues during setup:
1. Check the error messages
2. Review the documentation
3. Verify your API keys
4. Contact ALwrity support
---
*Note: Each component is designed to help you set up a specific aspect of ALwrity. Follow the setup wizard in order to ensure all components are properly configured for optimal content creation.*

22
ToBeMigrated/utils/api_key_manager/components/__init__.py
View File

@@ -1,22 +0,0 @@
"""API key manager components package."""
from .ai_research_setup import render_ai_research_setup
from .ai_research import render_ai_research
from .ai_providers import render_ai_providers
from .final_setup import render_final_setup
from .personalization_setup import render_personalization_setup
from .alwrity_integrations import render_alwrity_integrations
from .base import render_navigation_buttons, render_step_indicator
from .website_setup import render_website_setup
__all__ = [
'render_ai_research_setup',
'render_ai_research',
'render_ai_providers',
'render_final_setup',
'render_personalization_setup',
'render_alwrity_integrations',
'render_navigation_buttons',
'render_step_indicator',
'render_website_setup'
]

137
ToBeMigrated/utils/api_key_manager/components/ai_research.py
View File

@@ -1,137 +0,0 @@
"""AI Research setup component."""
import streamlit as st
from typing import Dict, Any
from loguru import logger
from ..manager import APIKeyManager
from .base import render_navigation_buttons, render_step_indicator
def render_ai_research(api_key_manager: APIKeyManager) -> Dict[str, Any]:
"""Render the AI Research setup step."""
try:
st.markdown("""
<div class='setup-header'>
<h2>🔍 AI Research Configuration</h2>
<p>Configure your research preferences and provide user information</p>
</div>
""", unsafe_allow_html=True)
# Create tabs for different sections
tabs = st.tabs(["User Information", "Research Preferences"])
changes_made = False
has_valid_info = False
validation_message = ""
with tabs[0]:
st.markdown("### User Information")
st.markdown("Please provide your details for personalized research experience")
# User Information Card
with st.container():
st.markdown("""
<div class="user-info-card">
<div class="user-info-header">
<div class="user-info-icon">👤</div>
<div class="user-info-title">Personal Details</div>
</div>
<div class="user-info-content">
<p>Your information helps us customize the research experience.</p>
</div>
</div>
""", unsafe_allow_html=True)
# User Input Fields with Streamlit Components
full_name = st.text_input("Full Name", key="full_name",
help="Enter your full name as you'd like it to appear")
email = st.text_input("Email Address", key="email",
help="Enter your business email address")
company = st.text_input("Company/Organization", key="company",
help="Enter your company or organization name")
role = st.selectbox("Role",
["Content Creator", "Marketing Manager", "Business Owner", "Other"],
help="Select your primary role")
with tabs[1]:
st.markdown("### Research Preferences")
st.markdown("Configure how AI assists with your research")
# Research Preferences Card
with st.container():
st.markdown("""
<div class="research-prefs-card">
<div class="research-prefs-header">
<div class="research-prefs-icon">🎯</div>
<div class="research-prefs-title">Research Settings</div>
</div>
</div>
""", unsafe_allow_html=True)
# Research Preferences Settings
research_depth = st.select_slider(
"Research Depth",
options=["Basic", "Standard", "Deep", "Comprehensive"],
value="Standard",
help="Choose how detailed you want the AI research to be"
)
st.markdown("#### Content Types")
content_types = st.multiselect(
"Select content types to focus on",
["Blog Posts", "Social Media", "Technical Articles", "News", "Academic Papers"],
default=["Blog Posts", "Social Media"],
help="Choose what types of content you want to research"
)
auto_research = st.toggle(
"Enable Automated Research",
help="Automatically start research when content topics are added"
)
# Validate inputs
if all([full_name, email, company]):
changes_made = True
has_valid_info = True
validation_message = "✅ User information completed successfully"
else:
validation_message = "⚠️ Please fill in all required fields to continue"
# Display validation message
if validation_message:
if "✅" in validation_message:
st.success(validation_message)
else:
st.warning(validation_message)
# Navigation buttons
if render_navigation_buttons(3, 6, changes_made):
if has_valid_info:
# Store user information in session state
st.session_state['user_info'] = {
'full_name': full_name,
'email': email,
'company': company,
'role': role,
'research_preferences': {
'depth': research_depth,
'content_types': content_types,
'auto_research': auto_research
}
}
# Update progress and move to next step
st.session_state['current_step'] = 4
st.rerun()
else:
st.error("Please complete all required fields to continue")
return {"current_step": 3, "changes_made": changes_made}
except Exception as e:
error_msg = f"Error in AI research setup: {str(e)}"
logger.error(f"[render_ai_research] {error_msg}")
st.error(error_msg)
return {"current_step": 3, "error": error_msg}

188
ToBeMigrated/utils/api_key_manager/components/personalization.py
View File

@@ -1,188 +0,0 @@
"""Personalization setup component."""
import streamlit as st
from typing import Dict, Any
from loguru import logger
from ..manager import APIKeyManager
from .base import render_navigation_buttons, render_step_indicator
def render_personalization(api_key_manager: APIKeyManager) -> Dict[str, Any]:
"""Render the personalization setup step."""
try:
st.markdown("""
<div class='setup-header'>
<h2>🎨 Personalization Settings</h2>
<p>Customize your content generation experience</p>
</div>
""", unsafe_allow_html=True)
# Create tabs for different sections
tabs = st.tabs(["Content Style", "Brand Voice", "Advanced Settings"])
changes_made = False
has_valid_settings = False
validation_message = ""
with tabs[0]:
st.markdown("### Content Style")
st.markdown("Define your preferred content style and tone")
# Content Style Card
with st.container():
st.markdown("""
<div class="style-card">
<div class="style-header">
<div class="style-icon">✨</div>
<div class="style-title">Writing Style</div>
</div>
<div class="style-content">
<p>Choose how you want your content to be written.</p>
</div>
</div>
""", unsafe_allow_html=True)
# Style Settings
writing_style = st.selectbox(
"Writing Style",
["Professional", "Casual", "Technical", "Conversational", "Academic"],
help="Select your preferred writing style"
)
tone = st.select_slider(
"Content Tone",
options=["Formal", "Semi-Formal", "Neutral", "Friendly", "Humorous"],
value="Neutral",
help="Choose the tone for your content"
)
content_length = st.select_slider(
"Content Length",
options=["Concise", "Standard", "Detailed", "Comprehensive"],
value="Standard",
help="Select your preferred content length"
)
with tabs[1]:
st.markdown("### Brand Voice")
st.markdown("Configure your brand's unique voice and personality")
# Brand Voice Card
with st.container():
st.markdown("""
<div class="brand-card">
<div class="brand-header">
<div class="brand-icon">🎯</div>
<div class="brand-title">Brand Identity</div>
</div>
<div class="brand-content">
<p>Define your brand's personality and voice.</p>
</div>
</div>
""", unsafe_allow_html=True)
# Brand Settings
brand_personality = st.multiselect(
"Brand Personality Traits",
["Professional", "Innovative", "Friendly", "Trustworthy", "Creative", "Expert"],
default=["Professional", "Trustworthy"],
help="Select traits that best describe your brand"
)
brand_voice = st.text_area(
"Brand Voice Description",
help="Describe how your brand should sound in content"
)
keywords = st.text_input(
"Brand Keywords",
help="Enter key terms that should be used in your content"
)
with tabs[2]:
st.markdown("### Advanced Settings")
st.markdown("Fine-tune your content generation preferences")
# Advanced Settings Card
with st.container():
st.markdown("""
<div class="advanced-card">
<div class="advanced-header">
<div class="advanced-icon">⚙️</div>
<div class="advanced-title">Advanced Options</div>
</div>
<div class="advanced-content">
<p>Configure advanced content generation settings.</p>
</div>
</div>
""", unsafe_allow_html=True)
# Advanced Settings
seo_optimization = st.toggle(
"Enable SEO Optimization",
help="Automatically optimize content for search engines"
)
readability_level = st.select_slider(
"Readability Level",
options=["Simple", "Standard", "Advanced", "Expert"],
value="Standard",
help="Choose the complexity level of your content"
)
content_structure = st.multiselect(
"Content Structure",
["Introduction", "Key Points", "Examples", "Conclusion", "Call-to-Action"],
default=["Introduction", "Key Points", "Conclusion"],
help="Select required content sections"
)
# Validate settings
if all([writing_style, tone, content_length, brand_personality]):
changes_made = True
has_valid_settings = True
validation_message = "✅ Personalization settings completed successfully"
else:
validation_message = "⚠️ Please complete all required settings to continue"
# Display validation message
if validation_message:
if "✅" in validation_message:
st.success(validation_message)
else:
st.warning(validation_message)
# Navigation buttons
if render_navigation_buttons(4, 6, changes_made):
if has_valid_settings:
# Store personalization settings in session state
st.session_state['personalization'] = {
'content_style': {
'writing_style': writing_style,
'tone': tone,
'content_length': content_length
},
'brand_voice': {
'personality': brand_personality,
'voice_description': brand_voice,
'keywords': keywords
},
'advanced_settings': {
'seo_optimization': seo_optimization,
'readability_level': readability_level,
'content_structure': content_structure
}
}
# Update progress and move to next step
st.session_state['current_step'] = 5
st.rerun()
else:
st.error("Please complete all required settings to continue")
return {"current_step": 4, "changes_made": changes_made}
except Exception as e:
error_msg = f"Error in personalization setup: {str(e)}"
logger.error(f"[render_personalization] {error_msg}")
st.error(error_msg)
return {"current_step": 4, "error": error_msg}

79
ToBeMigrated/utils/content_generators.py
View File

@@ -1,79 +0,0 @@
import streamlit as st
from lib.alwrity_ui.similar_analysis import competitor_analysis
from lib.alwrity_ui.keyword_web_researcher import do_web_research
def content_planning_tools():
# A custom CSS for compact layout
st.markdown("""
<style>
/* Reduce top padding of main container */
.main .block-container {
padding-top: 0rem !important;
padding-bottom: 1rem !important;
}
/* Reduce spacing between elements */
.stTabs {
margin-top: 0.5rem !important;
}
/* Make markdown text more compact */
.element-container {
margin-bottom: 0.5rem !important;
}
/* Adjust subheader margins */
.stMarkdown h3 {
margin-top: 0 !important;
margin-bottom: 0.5rem !important;
}
</style>
""", unsafe_allow_html=True)
# Make description more compact using a smaller font
st.markdown("""
<div style='font-size: 0.9em; margin-bottom: 0.5rem;'>
<strong>Alwrity content Ideation & Planning</strong>: Provide few keywords to do comprehensive web research.
Provide few keywords to get Google, Neural, pytrends analysis. Know keywords, blog titles to target.
Generate months long content calendar around given keywords.
</div>
""", unsafe_allow_html=True)
# Create tabs with reduced spacing
tab_keywords, tab_competitor, tab_calendar = st.tabs([
"🔍 Keywords Researcher",
"📊 Competitor Analysis",
"📅 Content Calendar Ideator"
])
# Keywords Researcher tab
with tab_keywords:
do_web_research()
# Competitor Analysis tab
with tab_competitor:
competitor_analysis()
# Content Calendar Ideator tab
with tab_calendar:
st.info("🚧 **Content Calendar & Planning Dashboard**")
st.markdown("""
<div style='background-color: #f0f2f6; padding: 15px; border-radius: 5px; margin-bottom: 20px;'>
<h3 style='margin-top: 0;'>📅 Content Calendar & Planning Dashboard</h3>
<p>The Content Calendar Dashboard provides:</p>
<ul>
<li>AI-powered content planning and generation</li>
<li>Multi-platform content scheduling</li>
<li>Content optimization tools</li>
<li>A/B testing capabilities</li>
<li>Performance analytics</li>
</ul>
</div>
""", unsafe_allow_html=True)
# Initialize and render the dashboard directly
from lib.ai_seo_tools.content_calendar.ui.dashboard import ContentCalendarDashboard
dashboard = ContentCalendarDashboard()
dashboard.render()

113
ToBeMigrated/utils/take_url_screenshot.py
View File

@@ -1,113 +0,0 @@
import os
import sys
import datetime
import subprocess
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from PIL import Image
from selenium import webdriver
from PIL import Image
import shutil
from screenshotone import Client, TakeOptions
from pathlib import Path
from dotenv import load_dotenv
load_dotenv(Path('../.env'))
from loguru import logger
logger.remove()
logger.add(sys.stdout,
colorize=True,
format="<level>{level}</level>|<green>{file}:{line}:{function}</green>| {message}"
)
def screenshot_api(url, generated_image_filepath):
""" Use screenshotone API to take company webpage screenshots """
try:
# create API client
client = Client(os.getenv('SCREENSHOTONE_ACCESS_KEY'), os.getenv('SCREENSHOTONE_SECRET_KEY'))
# set up options
options = (TakeOptions.url(url)
.format("png")
.viewport_width(1024)
.viewport_height(768)
.block_cookie_banners(True)
.block_chats(True))
# generate the screenshot URL and share it with a user
#url = client.generate_take_url(options)
# or render a screenshot and download the image as stream
image = client.take(options)
# store the screenshot the example.png file
with open(generated_image_filepath, 'wb', encoding="utf-8") as result_file:
shutil.copyfileobj(image, result_file)
# Display the screenshot using Image.show
image = Image.open(generated_image_filepath)
image.show()
# Wait for 2 seconds (adjust the delay as needed)
sleep(2)
# Close the image window
image.close()
except Exception as err:
print(f"Failed in screenshotone api: {err}")
generated_image_filepath = take_screenshot(url, generated_image_filepath)
return generated_image_filepath
def take_screenshot(url, generated_image_filepath):
# Create a webdriver instance in headless mode
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome(options=options)
logger.debug(f"Taking screenshot of url: {url}")
try:
# Navigate to the given url
driver.get(url)
# Optionally, increase the delay to ensure all content is loaded
sleep(2)
# Explicitly wait for the page to load (adjust timeout as needed)
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, "body")))
# Set a larger window size
driver.set_window_size(1200, 800)
# Take a screenshot of the webpage
screenshot = driver.get_screenshot_as_png()
# Save the screenshot to a file
with open(generated_image_filepath, "wb", encoding="utf-8") as f:
f.write(screenshot)
# Display the screenshot using Image.show
image = Image.open(generated_image_filepath)
image.show()
# Wait for 2 seconds (adjust the delay as needed)
sleep(2)
# Close the image window using subprocess (platform-dependent)
subprocess.run(["pkill", "-f", "display"]) # Adjust based on your platform and viewer
# If using macOS, you can use the following:
# subprocess.run(["osascript", "-e", 'tell application "Preview" to close every window'])
# If using Windows, you can use the following:
# subprocess.run(["taskkill", "/F", "/IM", "Microsoft.Photos.exe"])
logger.debug(f"Screenshot successfully stored at: {generated_image_filepath}")
return generated_image_filepath
finally:
# Close the webdriver instance
driver.quit()

181
ToBeMigrated/utils/website_analyzer/README.md
View File

@@ -1,181 +0,0 @@
# Website Analyzer Module
A comprehensive website analysis toolkit that provides detailed insights into website performance, SEO metrics, and content quality. This module combines traditional web analysis techniques with AI-powered content evaluation to deliver actionable recommendations.
## Features
### 1. Comprehensive Website Analysis
- Basic website information extraction
- SSL/TLS certificate validation
- DNS record analysis
- WHOIS information retrieval
- Content analysis and structure evaluation
- Performance metrics assessment
### 2. Advanced SEO Analysis
- Meta tag optimization analysis
- Content quality evaluation
- Keyword density analysis
- Readability scoring
- Heading structure analysis
- AI-powered content recommendations
### 3. Technical Infrastructure
- Asynchronous web crawling
- Multi-threaded analysis
- Robust error handling
- Comprehensive logging
- Type-safe data models
## Module Structure
### 1. `analyzer.py`
The main analysis engine that provides comprehensive website analysis.
#### Key Components:
- `WebsiteAnalyzer` class
- URL validation
- Basic website information extraction
- SSL/TLS certificate checking
- DNS record analysis
- WHOIS information retrieval
- Content analysis
- Performance metrics assessment
#### Features:
- Concurrent analysis using ThreadPoolExecutor
- Robust error handling and logging
- User-agent simulation for reliable scraping
- Timeout handling for requests
- Comprehensive result formatting
### 2. `seo_analyzer.py`
Specialized SEO analysis module with AI integration.
#### Key Components:
- `extract_content()`: Fetches and parses webpage content
- `analyze_meta_tags()`: Evaluates meta tags and SEO elements
- `analyze_content_with_ai()`: AI-powered content analysis
- `analyze_seo()`: Main SEO analysis function
#### Features:
- Meta tag optimization analysis
- Content quality scoring
- Keyword density analysis
- Readability evaluation
- AI-powered recommendations
- Weighted scoring system
### 3. `models.py`
Data models for structured analysis results.
#### Key Components:
- `SEORecommendation`: Individual SEO recommendations
- `MetaTagAnalysis`: Meta tag analysis results
- `ContentAnalysis`: Content analysis metrics
- `SEOAnalysisResult`: Complete analysis results
#### Features:
- Type-safe data structures
- Clear data organization
- Easy serialization/deserialization
- Comprehensive documentation
## Usage Examples
### Basic Website Analysis
```python
from website_analyzer import analyze_website
# Analyze a website
results = analyze_website("https://example.com")
# Access analysis results
if results["success"]:
data = results["data"]
print(f"Domain: {data['domain']}")
print(f"SSL Info: {data['analysis']['ssl_info']}")
print(f"Content Info: {data['analysis']['content_info']}")
```
### SEO Analysis
```python
from website_analyzer.seo_analyzer import analyze_seo
# Perform SEO analysis
seo_results = analyze_seo("https://example.com", "your-openai-api-key")
# Access SEO results
if seo_results.success:
print(f"Overall Score: {seo_results.overall_score}")
print(f"Meta Tags: {seo_results.meta_tags}")
print(f"Content Analysis: {seo_results.content}")
print(f"Recommendations: {seo_results.recommendations}")
```
## Dependencies
- `requests`: HTTP requests
- `beautifulsoup4`: HTML parsing
- `python-whois`: WHOIS information
- `dnspython`: DNS record analysis
- `openai`: AI-powered analysis
- `loguru`: Logging
- `typing`: Type hints
- `dataclasses`: Data models
## Error Handling
The module implements comprehensive error handling:
- URL validation
- Request timeouts
- Connection errors
- Parsing errors
- API errors
- DNS resolution errors
- SSL/TLS errors
All errors are logged and returned in a structured format for easy handling.
## Logging
The module uses `loguru` for logging with the following features:
- File rotation (500 MB)
- 10-day retention
- Debug level logging
- Structured log format
- Both file and stdout output
## Best Practices
1. **API Key Management**
- Store API keys securely
- Use environment variables
- Implement rate limiting
2. **Error Handling**
- Always check success status
- Handle errors gracefully
- Log errors appropriately
3. **Performance**
- Use concurrent analysis
- Implement timeouts
- Cache results when possible
4. **Rate Limiting**
- Respect website robots.txt
- Implement delays between requests
- Use appropriate user agents
## Contributing
1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request
## License
This module is part of the ALwrity project and is licensed under the MIT License.

6
ToBeMigrated/utils/website_analyzer/__init__.py
View File

@@ -1,6 +0,0 @@
"""Website analyzer module for AI-powered website analysis."""
from .analyzer import analyze_website, WebsiteAnalyzer
from .models import SEOAnalysisResult
__all__ = ['analyze_website', 'WebsiteAnalyzer', 'SEOAnalysisResult']

697
ToBeMigrated/utils/website_analyzer/analyzer.py
View File

@@ -1,697 +0,0 @@
"""Website and SEO analysis module."""
import asyncio
from typing import Dict, List, Optional, Tuple
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import streamlit as st
import re
from loguru import logger
from ...web_crawlers.async_web_crawler import AsyncWebCrawlerService
from ...gpt_providers.text_generation.main_text_generation import llm_text_gen
import os
import sys
import logging
import json
from datetime import datetime
import requests
import ssl
import socket
import whois
import dns.resolver
from requests.exceptions import RequestException
from concurrent.futures import ThreadPoolExecutor
from .models import (
SEOAnalysisResult,
MetaTagAnalysis,
ContentAnalysis,
SEORecommendation
)
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.StreamHandler(),
logging.FileHandler('logs/website_analyzer.log')
]
)
# Create a logger for the website analyzer
logger = logging.getLogger(__name__)
# Create a separate logger for scraping operations
scraping_logger = logging.getLogger('website_analyzer.scraping')
scraping_logger.setLevel(logging.WARNING)
class WebsiteAnalyzer:
def __init__(self):
self.session = requests.Session()
self.session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
})
logger.info("WebsiteAnalyzer initialized")
def analyze_website(self, url: str) -> Dict:
"""
Perform comprehensive analysis of a website.
Args:
url (str): The URL to analyze
Returns:
Dict: Analysis results including various metrics and checks
"""
logger.info(f"Starting analysis for URL: {url}")
try:
# Validate URL
if not self._validate_url(url):
error_msg = f"Invalid URL format: {url}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {"stage": "url_validation"}
}
# Basic URL parsing
parsed_url = urlparse(url)
domain = parsed_url.netloc
# Initialize results dictionary
results = {
"url": url,
"domain": domain,
"timestamp": datetime.now().isoformat(),
"analysis": {}
}
# Perform various analyses
with ThreadPoolExecutor(max_workers=4) as executor:
logger.info("Starting parallel analysis tasks")
# Basic website info
logger.info("Starting basic info analysis")
basic_info = executor.submit(self._get_basic_info, url).result()
if "error" in basic_info:
error_msg = f"Basic info analysis failed: {basic_info['error']}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {
"stage": "basic_info",
"details": basic_info.get("error_details", {})
}
}
results["analysis"]["basic_info"] = basic_info
# SSL/TLS info
logger.info("Starting SSL analysis")
ssl_info = executor.submit(self._check_ssl, domain).result()
results["analysis"]["ssl_info"] = ssl_info
# DNS info
logger.info("Starting DNS analysis")
dns_info = executor.submit(self._check_dns, domain).result()
results["analysis"]["dns_info"] = dns_info
# WHOIS info
logger.info("Starting WHOIS analysis")
whois_info = executor.submit(self._get_whois_info, domain).result()
results["analysis"]["whois_info"] = whois_info
# Content analysis
logger.info("Starting content analysis")
content_info = executor.submit(self._analyze_content, url).result()
if "error" in content_info:
error_msg = f"Content analysis failed: {content_info['error']}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {
"stage": "content_analysis",
"details": content_info.get("error_details", {})
}
}
results["analysis"]["content_info"] = content_info
# Performance metrics
logger.info("Starting performance analysis")
performance = executor.submit(self._check_performance, url).result()
if "error" in performance:
error_msg = f"Performance analysis failed: {performance['error']}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {
"stage": "performance_analysis",
"details": performance.get("error_details", {})
}
}
results["analysis"]["performance"] = performance
# SEO analysis
logger.info("Starting SEO analysis")
seo_analysis = executor.submit(self._analyze_seo, url).result()
if "error" in seo_analysis:
error_msg = f"SEO analysis failed: {seo_analysis['error']}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {
"stage": "seo_analysis",
"details": seo_analysis.get("error_details", {})
}
}
results["analysis"]["seo_info"] = seo_analysis
logger.info(f"Analysis completed successfully for {url}")
logger.debug(f"Final results: {json.dumps(results, indent=2)}")
return {
"success": True,
"data": results
}
except Exception as e:
error_msg = f"Error during website analysis: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
}
}
def _validate_url(self, url: str) -> bool:
"""Validate URL format."""
try:
result = urlparse(url)
return all([result.scheme, result.netloc])
except Exception as e:
logger.error(f"URL validation error: {str(e)}")
return False
def _get_basic_info(self, url: str) -> Dict:
"""Get basic website information."""
scraping_logger.debug(f"Getting basic info for {url}")
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
return {
"status_code": response.status_code,
"content_type": response.headers.get('content-type', ''),
"title": soup.title.string if soup.title else '',
"meta_description": self._get_meta_description(soup),
"headers": dict(response.headers),
"robots_txt": self._get_robots_txt(url),
"sitemap": self._get_sitemap(url)
}
except requests.exceptions.RequestException as e:
error_msg = f"Request error in basic info: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"error": error_msg,
"error_details": {
"type": "RequestException",
"status_code": getattr(e.response, 'status_code', None) if hasattr(e, 'response') else None,
"url": url
}
}
except Exception as e:
error_msg = f"Error getting basic info: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
}
}
def _check_ssl(self, domain: str) -> Dict:
"""Check SSL/TLS certificate information."""
scraping_logger.debug(f"Checking SSL for {domain}")
try:
context = ssl.create_default_context()
with socket.create_connection((domain, 443)) as sock:
with context.wrap_socket(sock, server_hostname=domain) as ssock:
cert = ssock.getpeercert()
return {
"has_ssl": True,
"issuer": dict(x[0] for x in cert['issuer']),
"expiry": datetime.strptime(cert['notAfter'], '%b %d %H:%M:%S %Y %Z').isoformat(),
"version": cert['version'],
"subject": dict(x[0] for x in cert['subject'])
}
except Exception as e:
logger.error(f"SSL check error: {str(e)}", exc_info=True)
return {"has_ssl": False, "error": str(e)}
def _check_dns(self, domain: str) -> Dict:
"""Check DNS records."""
scraping_logger.debug(f"Checking DNS for {domain}")
try:
records = {}
for record_type in ['A', 'AAAA', 'MX', 'NS', 'TXT']:
try:
answers = dns.resolver.resolve(domain, record_type)
records[record_type] = [str(rdata) for rdata in answers]
except dns.resolver.NoAnswer:
records[record_type] = []
except Exception as e:
scraping_logger.warning(f"Error resolving {record_type} record: {str(e)}")
records[record_type] = []
return records
except Exception as e:
logger.error(f"DNS check error: {str(e)}", exc_info=True)
return {"error": str(e)}
def _get_whois_info(self, domain: str) -> Dict:
"""Get WHOIS information for a domain."""
scraping_logger.debug(f"Getting WHOIS info for {domain}")
try:
w = whois.whois(domain)
def format_date(date_value):
if isinstance(date_value, list):
return date_value[0].isoformat() if date_value else 'Unknown'
return date_value.isoformat() if date_value else 'Unknown'
return {
'registrar': w.registrar if hasattr(w, 'registrar') else 'Unknown',
'creation_date': format_date(w.creation_date),
'expiration_date': format_date(w.expiration_date),
'updated_date': format_date(w.updated_date) if hasattr(w, 'updated_date') else 'Unknown',
'name_servers': w.name_servers if hasattr(w, 'name_servers') else [],
'domain_name': w.domain_name if hasattr(w, 'domain_name') else domain,
'text': w.text if hasattr(w, 'text') else ''
}
except Exception as e:
logger.error(f"WHOIS check error: {str(e)}")
return {
'registrar': 'Unknown',
'creation_date': 'Unknown',
'expiration_date': 'Unknown',
'updated_date': 'Unknown',
'name_servers': [],
'domain_name': domain,
'text': ''
}
def _analyze_content(self, url: str) -> Dict:
"""Analyze website content."""
scraping_logger.debug(f"Analyzing content for {url}")
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
# Get all text content
text_content = soup.get_text()
# Count words
words = re.findall(r'\w+', text_content.lower())
word_count = len(words)
# Count headings
headings = soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6'])
heading_counts = {
'h1': len(soup.find_all('h1')),
'h2': len(soup.find_all('h2')),
'h3': len(soup.find_all('h3')),
'h4': len(soup.find_all('h4')),
'h5': len(soup.find_all('h5')),
'h6': len(soup.find_all('h6'))
}
# Count images
images = soup.find_all('img')
# Count links
links = soup.find_all('a')
# Count paragraphs
paragraphs = soup.find_all('p')
return {
"word_count": word_count,
"heading_count": len(headings),
"heading_structure": heading_counts,
"image_count": len(images),
"link_count": len(links),
"paragraph_count": len(paragraphs),
"has_meta_description": bool(self._get_meta_description(soup)),
"has_robots_txt": bool(self._get_robots_txt(url)),
"has_sitemap": bool(self._get_sitemap(url))
}
except requests.exceptions.RequestException as e:
logger.error(f"Request error in content analysis: {str(e)}", exc_info=True)
return {
"word_count": 0,
"heading_count": 0,
"heading_structure": {'h1': 0, 'h2': 0, 'h3': 0, 'h4': 0, 'h5': 0, 'h6': 0},
"image_count": 0,
"link_count": 0,
"paragraph_count": 0,
"has_meta_description": False,
"has_robots_txt": False,
"has_sitemap": False,
"error": str(e)
}
except Exception as e:
logger.error(f"Content analysis error: {str(e)}", exc_info=True)
return {
"word_count": 0,
"heading_count": 0,
"heading_structure": {'h1': 0, 'h2': 0, 'h3': 0, 'h4': 0, 'h5': 0, 'h6': 0},
"image_count": 0,
"link_count": 0,
"paragraph_count": 0,
"has_meta_description": False,
"has_robots_txt": False,
"has_sitemap": False,
"error": str(e)
}
def _check_performance(self, url: str) -> Dict:
"""Check website performance metrics."""
scraping_logger.debug(f"Checking performance for {url}")
try:
start_time = datetime.now()
response = self.session.get(url, timeout=10)
end_time = datetime.now()
load_time = (end_time - start_time).total_seconds()
return {
"load_time": load_time,
"status_code": response.status_code,
"content_length": len(response.content),
"headers": dict(response.headers),
"response_time": response.elapsed.total_seconds()
}
except requests.exceptions.RequestException as e:
logger.error(f"Request error in performance check: {str(e)}", exc_info=True)
return {
"load_time": 0,
"status_code": 0,
"content_length": 0,
"headers": {},
"response_time": 0,
"error": str(e)
}
except Exception as e:
logger.error(f"Performance check error: {str(e)}", exc_info=True)
return {
"load_time": 0,
"status_code": 0,
"content_length": 0,
"headers": {},
"response_time": 0,
"error": str(e)
}
def _get_meta_description(self, soup: BeautifulSoup) -> Optional[str]:
"""Extract meta description from HTML."""
meta_desc = soup.find('meta', attrs={'name': 'description'})
return meta_desc.get('content') if meta_desc else None
def _get_robots_txt(self, url: str) -> Optional[str]:
"""Get robots.txt content."""
try:
robots_url = f"{url.rstrip('/')}/robots.txt"
response = self.session.get(robots_url, timeout=5)
if response.status_code == 200:
return response.text
except Exception as e:
scraping_logger.warning(f"Error fetching robots.txt: {str(e)}")
return None
def _get_sitemap(self, url: str) -> Optional[str]:
"""Get sitemap.xml content."""
try:
sitemap_url = f"{url.rstrip('/')}/sitemap.xml"
response = self.session.get(sitemap_url, timeout=5)
if response.status_code == 200:
return response.text
except Exception as e:
scraping_logger.warning(f"Error fetching sitemap.xml: {str(e)}")
return None
def _analyze_seo(self, url: str) -> Dict:
"""Analyze website SEO."""
try:
# Extract content
content, soup, extract_errors = self._extract_content(url)
if not content or not soup:
return {
"error": "Failed to extract content",
"error_details": {"errors": extract_errors}
}
# Analyze meta tags
meta_analysis = self._analyze_meta_tags(soup)
# Analyze content with AI
content_analysis, recommendations = self._analyze_content_with_ai(content)
# Calculate overall score
meta_score = sum([
1 if meta_analysis.title['status'] == 'good' else 0,
1 if meta_analysis.description['status'] == 'good' else 0,
1 if meta_analysis.keywords['status'] == 'good' else 0,
1 if meta_analysis.has_robots else 0,
1 if meta_analysis.has_sitemap else 0
]) * 20 # Scale to 100
overall_score = (
meta_score * 0.3 + # 30% weight for meta tags
content_analysis.readability_score * 0.3 + # 30% weight for readability
content_analysis.content_quality_score * 0.4 # 40% weight for content quality
)
return {
"overall_score": overall_score,
"meta_tags": meta_analysis.__dict__,
"content": content_analysis.__dict__,
"recommendations": [rec.__dict__ for rec in recommendations]
}
except Exception as e:
error_msg = f"Error in SEO analysis: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
}
}
def _extract_content(self, url: str) -> Tuple[Optional[str], Optional[BeautifulSoup], List[str]]:
"""Extract content from URL."""
errors = []
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
soup = BeautifulSoup(response.text, 'html.parser')
return response.text, soup, errors
except requests.RequestException as e:
error_msg = f"Error fetching URL: {str(e)}"
logger.error(error_msg)
errors.append(error_msg)
return None, None, errors
def _analyze_meta_tags(self, soup: BeautifulSoup) -> MetaTagAnalysis:
"""Analyze meta tags using BeautifulSoup."""
# Title analysis
title = soup.title.string if soup.title else ""
title_analysis = {
'status': 'good' if title and 30 <= len(title) <= 60 else 'needs_improvement',
'value': title,
'recommendation': '' if title and 30 <= len(title) <= 60 else 'Title should be between 30-60 characters'
}
# Meta description analysis
meta_desc = soup.find('meta', attrs={'name': 'description'})
desc = meta_desc.get('content', '') if meta_desc else ""
desc_analysis = {
'status': 'good' if desc and 120 <= len(desc) <= 160 else 'needs_improvement',
'value': desc,
'recommendation': '' if desc and 120 <= len(desc) <= 160 else 'Description should be between 120-160 characters'
}
# Keywords analysis
meta_keywords = soup.find('meta', attrs={'name': 'keywords'})
keywords = meta_keywords.get('content', '') if meta_keywords else ""
keywords_analysis = {
'status': 'good' if keywords else 'needs_improvement',
'value': keywords,
'recommendation': '' if keywords else 'Add relevant keywords meta tag'
}
return MetaTagAnalysis(
title=title_analysis,
description=desc_analysis,
keywords=keywords_analysis,
has_robots=bool(soup.find('meta', attrs={'name': 'robots'})),
has_sitemap=bool(soup.find('link', attrs={'rel': 'sitemap'}))
)
def _analyze_content_with_ai(self, content: str) -> Tuple[ContentAnalysis, List[SEORecommendation]]:
"""Analyze content using AI."""
try:
# Prepare prompt for content analysis
prompt = f"""Analyze the following webpage content for SEO and provide a structured analysis:
Content: {content[:4000]}... # Truncate to avoid token limits
Provide analysis in the following format:
1. Word count
2. Heading structure analysis
3. Keyword density for main topics
4. Readability score (0-100)
5. Content quality score (0-100)
6. List of SEO recommendations with priority (high/medium/low), category, issue, recommendation, and impact
Format the response as JSON."""
try:
# Get AI analysis using llm_text_gen
analysis = llm_text_gen(
prompt=prompt,
system_prompt="You are an SEO expert analyzing website content.",
response_format="json_object"
)
if not analysis:
logger.error("Empty response from AI analysis")
return self._get_fallback_analysis(content)
# Create ContentAnalysis object
content_analysis = ContentAnalysis(
word_count=len(content.split()),
headings_structure=analysis.get('heading_structure', {}),
keyword_density=analysis.get('keyword_density', {}),
readability_score=analysis.get('readability_score', 0),
content_quality_score=analysis.get('content_quality_score', 0)
)
# Create recommendations
recommendations = [
SEORecommendation(
priority=rec['priority'],
category=rec['category'],
issue=rec['issue'],
recommendation=rec['recommendation'],
impact=rec['impact']
)
for rec in analysis.get('recommendations', [])
]
return content_analysis, recommendations
except Exception as e:
logger.error(f"Error in AI analysis: {str(e)}")
return self._get_fallback_analysis(content)
except Exception as e:
logger.error(f"Error in AI analysis setup: {str(e)}")
return self._get_fallback_analysis(content)
def _get_fallback_analysis(self, content: str) -> Tuple[ContentAnalysis, List[SEORecommendation]]:
"""Provide fallback analysis when AI analysis is not available."""
try:
# Basic content analysis
words = content.split()
word_count = len(words)
# Simple readability score based on word count
readability_score = min(100, max(0, word_count / 10))
# Basic content quality score
content_quality_score = min(100, max(0, word_count / 20))
# Create basic recommendations
recommendations = [
SEORecommendation(
priority="high",
category="content",
issue="AI analysis unavailable",
recommendation="Consider running the analysis again with a valid API key for more detailed insights",
impact="Limited analysis capabilities"
)
]
return ContentAnalysis(
word_count=word_count,
headings_structure={},
keyword_density={},
readability_score=readability_score,
content_quality_score=content_quality_score
), recommendations
except Exception as e:
logger.error(f"Error in fallback analysis: {str(e)}")
return ContentAnalysis(
word_count=0,
headings_structure={},
keyword_density={},
readability_score=0,
content_quality_score=0
), []
def analyze_website(url: str) -> Dict:
"""
Analyze a website and return comprehensive results.
Args:
url (str): The URL to analyze
Returns:
Dict: Analysis results including various metrics and checks
"""
logger.info(f"Starting website analysis for URL: {url}")
try:
analyzer = WebsiteAnalyzer()
results = analyzer.analyze_website(url)
# Add success status to results
if "error" in results:
error_msg = f"Error in base analysis: {results['error']}"
logger.error(error_msg)
logger.error(f"Error details: {json.dumps(results.get('error_details', {}), indent=2)}")
return {
"success": False,
"error": error_msg,
"error_details": results.get("error_details", {})
}
# Add success status and wrap results
logger.info("Analysis completed successfully")
logger.debug(f"Analysis results: {json.dumps(results, indent=2)}")
return {
"success": True,
"data": results
}
except Exception as e:
error_msg = f"Error in analyze_website: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
}
}

134
ToBeMigrated/utils/website_analyzer/content_gap_analyzer.py
View File

@@ -1,134 +0,0 @@
from typing import Dict
import json
class ContentGapAnalyzer:
def __init__(self, analyzer):
self.analyzer = analyzer
def analyze(self, url: str) -> Dict:
"""
Analyze content gaps for a given URL.
Args:
url (str): The URL to analyze
Returns:
Dict: Analysis results including content gaps and recommendations
"""
try:
# Get base analysis
logger.info(f"Starting content gap analysis for URL: {url}")
base_analysis = self.analyzer.analyze_website(url)
# Check for errors in base analysis
if not base_analysis.get("success", False):
error_msg = base_analysis.get("error", "Unknown error in website analysis")
error_details = base_analysis.get("error_details", {})
logger.error(f"Base analysis failed: {error_msg}")
logger.error(f"Error details: {json.dumps(error_details, indent=2)}")
return {
"success": False,
"error": error_msg,
"error_details": error_details,
"stage": "base_analysis"
}
# Extract required sections
analysis_data = base_analysis.get("data", {}).get("analysis", {})
required_sections = ["content_info", "basic_info", "performance"]
missing_sections = [section for section in required_sections if section not in analysis_data]
if missing_sections:
error_msg = f"Missing required analysis sections: {', '.join(missing_sections)}"
logger.error(error_msg)
logger.error(f"Available sections: {list(analysis_data.keys())}")
return {
"success": False,
"error": error_msg,
"error_details": {
"missing_sections": missing_sections,
"available_sections": list(analysis_data.keys())
},
"stage": "section_validation"
}
# Extract content metrics
try:
content_info = analysis_data["content_info"]
basic_info = analysis_data["basic_info"]
performance = analysis_data["performance"]
except KeyError as e:
error_msg = f"Error extracting analysis section: {str(e)}"
logger.error(error_msg)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": "KeyError",
"missing_key": str(e),
"available_keys": list(analysis_data.keys())
},
"stage": "data_extraction"
}
# Analyze content gaps
try:
gaps = self._analyze_content_gaps(content_info, basic_info, performance)
except Exception as e:
error_msg = f"Error analyzing content gaps: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
},
"stage": "gap_analysis"
}
# Generate recommendations
try:
recommendations = self._generate_recommendations(gaps)
except Exception as e:
error_msg = f"Error generating recommendations: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
},
"stage": "recommendation_generation"
}
return {
"success": True,
"data": {
"content_gaps": gaps,
"recommendations": recommendations,
"metrics": {
"word_count": content_info.get("word_count", 0),
"heading_count": content_info.get("heading_count", 0),
"image_count": content_info.get("image_count", 0),
"link_count": content_info.get("link_count", 0),
"paragraph_count": content_info.get("paragraph_count", 0),
"load_time": performance.get("load_time", 0),
"response_time": performance.get("response_time", 0)
}
}
}
except Exception as e:
error_msg = f"Error in content gap analysis: {str(e)}"
logger.error(error_msg, exc_info=True)
return {
"success": False,
"error": error_msg,
"error_details": {
"type": type(e).__name__,
"traceback": str(e.__traceback__)
},
"stage": "general"
}

BIN
ToBeMigrated/workspace/AskAlwrity-min.ico
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 79 KiB

BIN
ToBeMigrated/workspace/alwrity_ai_writer.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 270 KiB

BIN
ToBeMigrated/workspace/alwrity_blog_post_writer.gif
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 1.3 MiB

BIN
ToBeMigrated/workspace/alwrity_logo.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 188 KiB

5
tatus
View File

@@ -1,5 +0,0 @@
cc8f9cd2e (HEAD -> main, origin/cleanup/remove-cache-files, cleanup/remove-cache-files) Clean up: Remove all cache files and add comprehensive .gitignore
c19fc3f22 (origin/main, origin/HEAD) ALwrity Prompts - AI Integration Plan
5efee4235 Added citation and quality metrics to the content editor.
10b50f973 Alwrity Copilot Integration for LinkedIn Writer
64944104a merge: LinkedIn Writer PR #223 - resolve conflicts and integrate with existing routers