kunthawat/ALwrity
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ALwrity onboarding final step

Browse Source
This commit is contained in:
ajaysi
2025-10-10 23:19:28 +05:30
parent e3daebec16
commit b1ebe1034e
38 changed files with 4867 additions and 770 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
docs/API_KEY_FLOW_DIAGRAM.md Normal file
View File

@@ -0,0 +1,370 @@
# API Key Management Flow Diagrams
## 🏠 Local Development Mode
```
┌─────────────────────────────────────────────────────────────────────┐
│ LOCAL DEVELOPMENT │
│ (DEBUG=true) │
└─────────────────────────────────────────────────────────────────────┘
Developer completes onboarding
│
├─> Frontend: Save API keys
│ └─> POST /api/onboarding/api-keys (gemini, exa, copilotkit)
│
├─> Backend: Process API keys
│ │
│ ├─> Save to PostgreSQL database
│ │ └─> onboarding_sessions (user_id)
│ │ └─> api_keys (provider, key)
│ │
│ └─> Save to backend/.env file [DEV MODE ONLY]
│ ├─> GEMINI_API_KEY=xxx
│ ├─> EXA_API_KEY=xxx
│ └─> COPILOTKIT_API_KEY=xxx
│
└─> Frontend: Save CopilotKit to frontend/.env
└─> REACT_APP_COPILOTKIT_API_KEY=xxx
Developer generates content
│
├─> Service calls user_api_keys(user_id=None)
│ │
│ └─> Detects DEV mode (DEBUG=true)
│ └─> Reads from backend/.env file
│ └─> Returns all keys
│
└─> Content generated using developer's keys
└─> All costs → Developer's API account
✅ Advantages:
• Quick setup (keys persist in .env)
• No database required for basic dev
• Single developer = single set of keys
• Keys survive server restarts
```
---
## 🌐 Production Mode (Multi-User)
```
┌─────────────────────────────────────────────────────────────────────┐
│ PRODUCTION (VERCEL + RENDER) │
│ (DEBUG=false, DEPLOY_ENV=render) │
└─────────────────────────────────────────────────────────────────────┘
Alpha Tester A visits https://alwrity-ai.vercel.app
│
├─> Completes onboarding
│ └─> Enters API keys:
│ ├─> GEMINI_API_KEY=tester_a_key
│ ├─> EXA_API_KEY=tester_a_exa
│ └─> COPILOTKIT_API_KEY=tester_a_copilot
│
├─> Frontend: Save API keys
│ ├─> POST /api/onboarding/api-keys (gemini, exa, copilotkit)
│ └─> Save to localStorage (CopilotKit)
│
└─> Backend: Process API keys
├─> Save to PostgreSQL database ONLY [PROD MODE]
│ └─> onboarding_sessions
│ ├─> user_id = "user_clerk_tester_a"
│ └─> api_keys
│ ├─> (session_id, "gemini", "tester_a_key")
│ ├─> (session_id, "exa", "tester_a_exa")
│ └─> (session_id, "copilotkit", "tester_a_copilot")
│
└─> [SKIP] ❌ Do NOT save to .env file (multi-user conflict!)
Alpha Tester A generates blog content
│
├─> Request to /api/blog/generate
│ └─> Headers: Authorization: Bearer <tester_a_clerk_token>
│
├─> Auth Middleware extracts user_id = "user_clerk_tester_a"
│
├─> BlogService calls user_api_keys("user_clerk_tester_a")
│ │
│ ├─> Detects PROD mode (DEPLOY_ENV=render)
│ │
│ └─> Query database:
│ SELECT key FROM api_keys
│ WHERE session_id = (
│ SELECT id FROM onboarding_sessions
│ WHERE user_id = 'user_clerk_tester_a'
│ )
│ └─> Returns: {"gemini": "tester_a_key", "exa": "tester_a_exa"}
│
└─> Content generated using Tester A's Gemini key
└─> All costs → Tester A's Gemini account
────────────────────────────────────────────────────────────────────────
SIMULTANEOUSLY...
Alpha Tester B visits https://alwrity-ai.vercel.app
│
├─> Completes onboarding
│ └─> Enters API keys:
│ ├─> GEMINI_API_KEY=tester_b_key
│ ├─> EXA_API_KEY=tester_b_exa
│ └─> COPILOTKIT_API_KEY=tester_b_copilot
│
└─> Backend: Save to database
└─> onboarding_sessions
├─> user_id = "user_clerk_tester_b"
└─> api_keys
├─> (session_id, "gemini", "tester_b_key") [SEPARATE!]
├─> (session_id, "exa", "tester_b_exa")
└─> (session_id, "copilotkit", "tester_b_copilot")
Alpha Tester B generates blog content
│
├─> Request to /api/blog/generate
│ └─> Headers: Authorization: Bearer <tester_b_clerk_token>
│
├─> Auth Middleware extracts user_id = "user_clerk_tester_b"
│
├─> BlogService calls user_api_keys("user_clerk_tester_b")
│ │
│ └─> Query database:
│ WHERE user_id = 'user_clerk_tester_b' [DIFFERENT!]
│ └─> Returns: {"gemini": "tester_b_key", "exa": "tester_b_exa"}
│
└─> Content generated using Tester B's Gemini key
└─> All costs → Tester B's Gemini account
✅ User Isolation:
• Tester A's keys ≠ Tester B's keys
• Tester A's costs ≠ Tester B's costs
• Completely isolated in database
• You (owner) pay nothing! 🎉
```
---
## 🔄 Environment Detection Logic
```
┌─────────────────────────────────────────────────────────────────────┐
│ ENVIRONMENT DETECTION │
└─────────────────────────────────────────────────────────────────────┘
When user_api_keys(user_id) is called:
┌──────────────────────────────────┐
│ Check environment variables │
└──────────────────────────────────┘
│
├─> DEBUG=true OR DEPLOY_ENV=None
│ │
│ ├─> DEVELOPMENT MODE
│ │ └─> Read from backend/.env file
│ │ └─> os.getenv('GEMINI_API_KEY')
│ │
│ └─> Log: "[DEV MODE] Using .env file"
│
└─> DEBUG=false AND DEPLOY_ENV=render
│
├─> PRODUCTION MODE
│ └─> Read from database
│ └─> SELECT key FROM api_keys WHERE user_id=?
│
└─> Log: "[PROD MODE] Using database for user {user_id}"
Example configurations:
Local Development:
┌─────────────────────────────┐
│ backend/.env │
├─────────────────────────────┤
│ DEBUG=true │
│ GEMINI_API_KEY=dev_key │
│ EXA_API_KEY=dev_exa │
└─────────────────────────────┘
Render Production:
┌─────────────────────────────┐
│ Environment Variables │
├─────────────────────────────┤
│ DEBUG=false │
│ DEPLOY_ENV=render │
│ DATABASE_URL=postgresql:// │
└─────────────────────────────┘
```
---
## 📊 Database Schema Visualization
```
┌─────────────────────────────────────────────────────────────────────┐
│ DATABASE SCHEMA │
└─────────────────────────────────────────────────────────────────────┘
onboarding_sessions
┌────────────┬──────────────────────────┬─────────────┬──────────┐
│ id (PK) │ user_id (UNIQUE) │ current_step│ progress │
├────────────┼──────────────────────────┼─────────────┼──────────┤
│ 1 │ user_clerk_tester_a │ 6 │ 100.0 │
│ 2 │ user_clerk_tester_b │ 6 │ 100.0 │
│ 3 │ user_clerk_tester_c │ 3 │ 50.0 │
└────────────┴──────────────────────────┴─────────────┴──────────┘
api_keys
┌────────────┬────────────┬──────────────┬────────────────────────┐
│ id (PK) │ session_id │ provider │ key │
│ │ (FK) │ │ │
├────────────┼────────────┼──────────────┼────────────────────────┤
│ 1 │ 1 │ gemini │ tester_a_gemini_key │ ← Tester A
│ 2 │ 1 │ exa │ tester_a_exa_key │ ← Tester A
│ 3 │ 1 │ copilotkit │ tester_a_copilot_key │ ← Tester A
├────────────┼────────────┼──────────────┼────────────────────────┤
│ 4 │ 2 │ gemini │ tester_b_gemini_key │ ← Tester B
│ 5 │ 2 │ exa │ tester_b_exa_key │ ← Tester B
│ 6 │ 2 │ copilotkit │ tester_b_copilot_key │ ← Tester B
├────────────┼────────────┼──────────────┼────────────────────────┤
│ 7 │ 3 │ gemini │ tester_c_gemini_key │ ← Tester C
│ 8 │ 3 │ exa │ tester_c_exa_key │ ← Tester C
└────────────┴────────────┴──────────────┴────────────────────────┘
Query to get Tester A's Gemini key:
SELECT k.key
FROM api_keys k
JOIN onboarding_sessions s ON k.session_id = s.id
WHERE s.user_id = 'user_clerk_tester_a'
AND k.provider = 'gemini'
Result: 'tester_a_gemini_key'
Query to get Tester B's Gemini key:
SELECT k.key
FROM api_keys k
JOIN onboarding_sessions s ON k.session_id = s.id
WHERE s.user_id = 'user_clerk_tester_b'
AND k.provider = 'gemini'
Result: 'tester_b_gemini_key' [DIFFERENT!]
```
---
## 🔐 Security & Isolation
```
┌─────────────────────────────────────────────────────────────────────┐
│ USER ISOLATION GUARANTEE │
└─────────────────────────────────────────────────────────────────────┘
Scenario: Both Tester A and Tester B generate content simultaneously
Tester A's Request Thread:
┌────────────────────────────────────────────┐
│ 1. Auth: user_id = "user_clerk_tester_a" │
│ 2. Fetch keys: WHERE user_id = tester_a │
│ 3. Get: gemini_key = "tester_a_key" │
│ 4. Generate with tester_a_key │
│ 5. Response to Tester A │
└────────────────────────────────────────────┘
↓
[Database]
↑
┌────────────────────────────────────────────┐
│ 1. Auth: user_id = "user_clerk_tester_b" │
│ 2. Fetch keys: WHERE user_id = tester_b │
│ 3. Get: gemini_key = "tester_b_key" │
│ 4. Generate with tester_b_key │
│ 5. Response to Tester B │
└────────────────────────────────────────────┘
Tester B's Request Thread:
✅ Guarantees:
• Tester A NEVER sees Tester B's keys
• Tester B NEVER sees Tester A's keys
• Tester A's costs charged to Tester A
• Tester B's costs charged to Tester B
• Database enforces isolation via user_id
• Clerk auth ensures correct user_id
```
---
## 💰 Cost Distribution
```
┌─────────────────────────────────────────────────────────────────────┐
│ WHO PAYS FOR WHAT? │
└─────────────────────────────────────────────────────────────────────┘
Local Development (You):
Your API Keys → Your Costs
┌─────────────────────────────────────────────┐
│ Developer generates 100 blog posts │
│ Uses: GEMINI_API_KEY from .env │
│ Cost: $5.00 → Charged to developer's │
│ Google Cloud account │
└─────────────────────────────────────────────┘
Production (Alpha Testers):
Their API Keys → Their Costs
┌─────────────────────────────────────────────┐
│ Tester A generates 50 blog posts │
│ Uses: tester_a_gemini_key from database │
│ Cost: $2.50 → Charged to Tester A's │
│ Google Cloud account │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
│ Tester B generates 200 blog posts │
│ Uses: tester_b_gemini_key from database │
│ Cost: $10.00 → Charged to Tester B's │
│ Google Cloud account │
└─────────────────────────────────────────────┘
┌─────────────────────────────────────────────┐
│ You (owner) host infrastructure │
│ Render: Free tier / $7/month │
│ Vercel: Free tier │
│ Database: Render free tier │
│ Cost: $0 - $7/month (infrastructure only) │
└─────────────────────────────────────────────┘
Total monthly cost for you with 100 alpha testers:
Infrastructure: $0 - $7
API usage: $0 (testers pay their own!)
────────────────────────────
Total: $0 - $7/month 🎉
```
---
## 🎯 Summary
| Aspect | Local Dev | Production |
|--------|-----------|------------|
| **Environment** | `DEBUG=true` | `DEPLOY_ENV=render` |
| **Key Storage** | `.env` file + DB | Database only |
| **Key Retrieval** | `os.getenv()` | Database query |
| **User Isolation** | Not needed | Full isolation |
| **Cost Bearer** | You (developer) | Each tester |
| **Scalability** | 1 developer | Unlimited users |
| **Setup Effort** | Low (persist .env) | Low (onboard once) |
**Architecture Principle:**
> Development convenience with `.env` files, production isolation with database. Best of both worlds! 🚀

326
docs/API_KEY_INJECTION_EXPLAINED.md Normal file
View File

@@ -0,0 +1,326 @@
# API Key Injection - How It Works in Production
## 🎯 The Problem You Identified
**Question:** "For production, when we read APIs from database, how will they be exported to the environment?"
**Answer:** They are **temporarily injected** into `os.environ` for each request, then immediately cleaned up.
---
## 🔍 The Challenge
### **Existing Code Pattern:**
Most of your codebase uses this pattern:
```python
import os
import google.generativeai as genai
def generate_content(prompt: str):
# Expects GEMINI_API_KEY in environment
gemini_key = os.getenv('GEMINI_API_KEY')
genai.configure(api_key=gemini_key)
# ...
```
### **Production Problem:**
```
User A's request:
↓
os.getenv('GEMINI_API_KEY') → ??? (User A's key in database, not in os.environ)
User B's request (simultaneous):
↓
os.getenv('GEMINI_API_KEY') → ??? (User B's key in database, not in os.environ)
```
**Issue:** `os.environ` is global, but we need user-specific keys!
---
## ✅ The Solution: Request-Scoped Injection
### **How It Works:**
```
1. Request arrives with Authorization: Bearer <user_a_token>
↓
2. API Key Injection Middleware extracts user_id from token
↓
3. Fetch User A's keys from database
↓
4. Temporarily inject into os.environ:
- GEMINI_API_KEY = user_a_gemini_key
- EXA_API_KEY = user_a_exa_key
↓
5. Process request (all os.getenv() calls get User A's keys)
↓
6. Request completes
↓
7. IMMEDIATELY clean up os.environ (remove User A's keys)
```
### **Key Insight:**
**The injection is request-scoped, not global:**
- User A's keys exist in `os.environ` ONLY during User A's request
- Immediately removed after response sent
- User B's request gets User B's keys injected
- No overlap, no conflict!
---
## 🏗️ Architecture
### **Middleware Flow:**
```
FastAPI Request Pipeline:
┌─────────────────────────────────────────────────────────────┐
│ 1. Rate Limit Middleware │
│ └─> Check rate limits │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 2. API Key Injection Middleware (NEW!) │
│ ├─> Extract user_id from Authorization header │
│ ├─> Fetch user's API keys from database │
│ ├─> Inject into os.environ (temporarily) │
│ │ ├─> GEMINI_API_KEY = user_specific_key │
│ │ ├─> EXA_API_KEY = user_specific_key │
│ │ └─> COPILOTKIT_API_KEY = user_specific_key │
│ └─> [Request processed with user-specific keys] │
│ ↓ │
│ ├─> [Response generated] │
│ └─> CLEANUP: Remove injected keys from os.environ │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Your Endpoint (e.g., /api/blog/generate) │
│ └─> Calls service that uses os.getenv('GEMINI_API_KEY') │
│ └─> Gets user-specific key! ✅ │
└─────────────────────────────────────────────────────────────┘
```
---
## 💻 Code Example
### **The Middleware:**
```python
async def __call__(self, request: Request, call_next):
# 1. Extract user_id from token
user_id = extract_user_from_token(request)
if not user_id or DEPLOY_ENV == 'local':
return await call_next(request) # Skip in local mode
# 2. Get user-specific keys from database
with user_api_keys(user_id) as user_keys:
# 3. Save original environment (if any)
original_gemini = os.environ.get('GEMINI_API_KEY')
original_exa = os.environ.get('EXA_API_KEY')
# 4. Inject user-specific keys
os.environ['GEMINI_API_KEY'] = user_keys['gemini']
os.environ['EXA_API_KEY'] = user_keys['exa']
try:
# 5. Process request with user-specific keys
response = await call_next(request)
return response
finally:
# 6. CRITICAL: Restore original environment
if original_gemini is None:
del os.environ['GEMINI_API_KEY']
else:
os.environ['GEMINI_API_KEY'] = original_gemini
if original_exa is None:
del os.environ['EXA_API_KEY']
else:
os.environ['EXA_API_KEY'] = original_exa
```
---
## 📊 Concurrent Requests Example
### **Scenario: Two Users Generate Content Simultaneously**
```
TIME: 00:00:000
User A request arrives
├─> Extract user_id = "user_a"
├─> Fetch keys from DB: gemini_key = "key_a_123"
├─> os.environ['GEMINI_API_KEY'] = "key_a_123"
│
├─> TIME: 00:00:050 (50ms later)
│ User B request arrives
│ ├─> Extract user_id = "user_b"
│ ├─> Fetch keys from DB: gemini_key = "key_b_456"
│ ├─> os.environ['GEMINI_API_KEY'] = "key_b_456" ← Overwrites!
│ │
│ ├─> User B's request processes
│ │ os.getenv('GEMINI_API_KEY') → "key_b_456" ✅
│ │
│ └─> TIME: 00:00:100
│ User B response sent
│ os.environ['GEMINI_API_KEY'] restored
│
└─> TIME: 00:00:120
User A's request processes
os.getenv('GEMINI_API_KEY') → ??? (Could be wrong!)
```
**⚠️ PROBLEM: Race condition!**
---
## 🔒 Thread Safety Solution
Python's asyncio in FastAPI handles this correctly:
```python
# FastAPI uses asyncio, which is single-threaded
# Each request is processed in sequence (no parallel execution)
# So the injection is safe!
User A request:
├─> Inject A's keys
├─> await generate_content() ← Async, but single-threaded
└─> Cleanup A's keys
User B request (after A):
├─> Inject B's keys
├─> await generate_content()
└─> Cleanup B's keys
```
**BUT:** If your code uses threading or multiprocessing, this approach WON'T work safely.
---
## 🎛️ Modes Compared
### **Local Mode (DEPLOY_ENV=local):**
```
Request arrives
↓
Middleware detects DEPLOY_ENV=local
↓
SKIP injection (keys already in .env)
↓
os.getenv('GEMINI_API_KEY') → reads from .env file
↓
Works! ✅
```
### **Production Mode (DEPLOY_ENV=production):**
```
Request arrives with user_id=user_123
↓
Middleware detects DEPLOY_ENV=production
↓
Fetch user_123's keys from database
↓
Inject into os.environ (temporarily)
↓
os.getenv('GEMINI_API_KEY') → gets user_123's key
↓
Process request
↓
Clean up os.environ
↓
Works! ✅
```
---
## 🚨 Important Caveats
### **1. Async-Only Safety**
This approach is safe ONLY because FastAPI uses asyncio (single-threaded event loop).
**If you use:**
- `concurrent.futures.ThreadPoolExecutor`
- `multiprocessing.Pool`
- `threading.Thread`
Then environment injection is **NOT SAFE** and will cause race conditions!
### **2. Better Long-Term Approach**
For critical services, refactor to pass `user_id` explicitly:
```python
# Instead of:
def generate(prompt: str):
key = os.getenv('GEMINI_API_KEY') # Fragile!
# Do this:
def generate(user_id: str, prompt: str):
with user_api_keys(user_id) as keys:
key = keys['gemini'] # Explicit and safe!
```
---
## 📝 Summary
### **The Magic:**
1. **Request arrives** → Middleware extracts `user_id`
2. **Fetch from DB** → Get user's keys
3. **Inject temporarily** → `os.environ['GEMINI_API_KEY'] = user_key`
4. **Process request** → All `os.getenv()` calls get user's key
5. **Cleanup** → Remove from `os.environ`
6. **Next request** → Different user, different keys
### **Why It Works:**
- ✅ FastAPI is async + single-threaded
- ✅ Injection is request-scoped
- ✅ Cleanup is guaranteed (finally block)
- ✅ Existing code works without changes
- ✅ Each user gets their own keys
### **Limitations:**
- ⚠️ Not safe with threading/multiprocessing
- ⚠️ Slightly slower (DB query per request)
- ⚠️ Better to refactor critical services
### **Bottom Line:**
> **It works!** Your existing code that uses `os.getenv()` will get user-specific keys in production, with zero code changes. The middleware handles everything automatically.
---
## 🔄 Migration Path
### **Phase 1: Now (Compatibility Layer)**
- ✅ Middleware injects keys for ALL services
- ✅ No code changes needed
- ✅ Works immediately
### **Phase 2: Later (Gradual Refactor)**
- Refactor critical services to use `UserAPIKeyContext` directly
- Remove dependency on `os.getenv()`
- More explicit, safer
### **Phase 3: Future (Full Migration)**
- All services use `user_api_keys(user_id)`
- Remove injection middleware
- Clean, explicit architecture
**For now:** Middleware lets you deploy immediately without touching 100+ files! 🎉

349
docs/API_KEY_MANAGEMENT_ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,349 @@
# API Key Management Architecture
## Overview
ALwrity supports two deployment modes with different API key management strategies:
1. **Local Development**: API keys stored in `.env` files for convenience
2. **Production (Vercel + Render)**: User-specific API keys stored in database with full user isolation
## Architecture
### 🏠 **Local Development Mode**
**Detection:**
- `DEBUG=true` in environment variables, OR
- `DEPLOY_ENV` is not set
**API Key Storage:**
- **Backend**: `backend/.env` file
- **Frontend**: `frontend/.env` file
- **Database**: Also saved for consistency
**Flow:**
```
User completes onboarding
↓
API keys saved to database (user-isolated)
↓
API keys ALSO saved to .env files (for convenience)
↓
Backend services read from .env file
↓
Single developer, single set of keys
```
**Advantages:**
- ✅ Quick setup for developers
- ✅ No need to configure environment for every user
- ✅ Keys persist across server restarts
---
### 🌐 **Production Mode (Vercel + Render)**
**Detection:**
- `DEBUG=false` or not set, AND
- `DEPLOY_ENV` is set (e.g., `DEPLOY_ENV=render`)
**API Key Storage:**
- **Backend**: PostgreSQL database (user-isolated)
- **Frontend**: `localStorage` (runtime only)
- **NOT in .env files**
**Flow:**
```
Alpha Tester A completes onboarding
↓
API keys saved to database with user_id_A
↓
Backend services fetch keys from database when user_id_A makes requests
↓
Multiple users, each with their own keys
↓
Alpha Tester B completes onboarding
↓
API keys saved to database with user_id_B
↓
Backend services fetch keys from database when user_id_B makes requests
```
**Advantages:**
- ✅ **Complete user isolation** - User A's keys never conflict with User B's keys
- ✅ **Zero cost for you** - Each alpha tester uses their own API keys
- ✅ **Secure** - Keys stored encrypted in database
- ✅ **Scalable** - Unlimited alpha testers, each with their own keys
---
## Implementation
### **1. Backend: User API Key Context**
The `UserAPIKeyContext` class provides user-specific API keys to backend services:
```python
from services.user_api_key_context import user_api_keys
# In your backend service
async def generate_content(user_id: str, prompt: str):
# Get user-specific API keys
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
exa_key = keys.get('exa')
# Use keys for this specific user
response = await call_gemini_api(gemini_key, prompt)
return response
```
**How it works:**
- **Development**: Reads from `backend/.env`
- **Production**: Fetches from database for the specific `user_id`
### **2. Frontend: CopilotKit Key Management**
```typescript
// Frontend automatically handles this:
// 1. Saves to localStorage (for runtime use)
// 2. In dev: Also saves to frontend/.env
// 3. In prod: Only uses localStorage
const copilotApiKey = localStorage.getItem('copilotkit_api_key');
```
### **3. Environment Variable Detection**
**Backend (`backend/.env`):**
```bash
# Development
DEBUG=true
# Production
DEBUG=false
DEPLOY_ENV=render # or 'railway', 'heroku', etc.
```
**Render Dashboard:**
```
DEBUG=false
DEPLOY_ENV=render
```
**Vercel Dashboard:**
```
REACT_APP_API_URL=https://alwrity.onrender.com
REACT_APP_BACKEND_URL=https://alwrity.onrender.com
```
---
## Use Cases
### **Use Case 1: You (Developer) - Local Development**
**Setup:**
```bash
# backend/.env
DEBUG=true
GEMINI_API_KEY=your_personal_key
EXA_API_KEY=your_personal_key
COPILOTKIT_API_KEY=your_personal_key
```
**Behavior:**
- You complete onboarding once
- Keys saved to both database AND `.env` files
- All your local testing uses these keys
- No need to re-enter keys
---
### **Use Case 2: Alpha Tester A - Production**
**Setup:**
- Alpha Tester A visits `https://alwrity-ai.vercel.app`
- Goes through onboarding
- Enters their own API keys:
- `GEMINI_API_KEY=tester_a_gemini_key`
- `EXA_API_KEY=tester_a_exa_key`
- `COPILOTKIT_API_KEY=tester_a_copilot_key`
**Behavior:**
- Keys saved to database with `user_id=tester_a_clerk_id`
- When Tester A generates content:
- Backend fetches `tester_a_gemini_key` from database
- Uses Tester A's Gemini quota
- All costs charged to Tester A's Gemini account
---
### **Use Case 3: Alpha Tester B - Production (Same Time)**
**Setup:**
- Alpha Tester B visits `https://alwrity-ai.vercel.app`
- Goes through onboarding
- Enters their own API keys:
- `GEMINI_API_KEY=tester_b_gemini_key`
- `EXA_API_KEY=tester_b_exa_key`
- `COPILOTKIT_API_KEY=tester_b_copilot_key`
**Behavior:**
- Keys saved to database with `user_id=tester_b_clerk_id`
- When Tester B generates content:
- Backend fetches `tester_b_gemini_key` from database
- Uses Tester B's Gemini quota
- All costs charged to Tester B's Gemini account
- **Tester A and Tester B completely isolated** ✅
---
## Database Schema
```sql
-- OnboardingSession: One per user
CREATE TABLE onboarding_sessions (
id SERIAL PRIMARY KEY,
user_id VARCHAR(255) UNIQUE NOT NULL, -- Clerk user ID
current_step INTEGER DEFAULT 1,
progress FLOAT DEFAULT 0.0,
started_at TIMESTAMP DEFAULT NOW(),
completed_at TIMESTAMP
);
-- APIKey: Multiple per user (one per provider)
CREATE TABLE api_keys (
id SERIAL PRIMARY KEY,
session_id INTEGER REFERENCES onboarding_sessions(id),
provider VARCHAR(50) NOT NULL, -- 'gemini', 'exa', 'copilotkit'
key TEXT NOT NULL, -- Encrypted in production
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW(),
UNIQUE(session_id, provider) -- One key per provider per user
);
```
**Isolation:**
- Each user has their own `onboarding_session`
- Each session has its own set of `api_keys`
- Query: `SELECT key FROM api_keys WHERE session_id = (SELECT id FROM onboarding_sessions WHERE user_id = ?)`
---
## Migration Path
### **Current State:**
- ❌ All users' keys overwrite the same `.env` file
- ❌ Last user's keys are used for all users
### **New State:**
- ✅ Development: `.env` file for convenience
- ✅ Production: Database per user
- ✅ Complete user isolation
### **Code Changes Required:**
**Before (BAD - uses global .env):**
```python
import os
def generate_content(prompt: str):
gemini_key = os.getenv('GEMINI_API_KEY') # Same for all users!
response = call_gemini_api(gemini_key, prompt)
return response
```
**After (GOOD - uses user-specific keys):**
```python
from services.user_api_key_context import user_api_keys
def generate_content(user_id: str, prompt: str):
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini') # User-specific key!
response = call_gemini_api(gemini_key, prompt)
return response
```
---
## Testing
### **Test Local Development:**
1. Set `DEBUG=true` in `backend/.env`
2. Complete onboarding with test keys
3. Check `backend/.env` - should contain keys ✅
4. Generate content - should use keys from `.env` ✅
### **Test Production:**
1. Set `DEBUG=false` and `DEPLOY_ENV=render` on Render
2. User A completes onboarding with keys A
3. User B completes onboarding with keys B
4. User A generates content - uses keys A ✅
5. User B generates content - uses keys B ✅
6. Check database:
```sql
SELECT user_id, provider, key FROM api_keys
JOIN onboarding_sessions ON api_keys.session_id = onboarding_sessions.id;
```
Should show separate keys for User A and User B ✅
---
## Security Considerations
### **Production Enhancements (Future):**
1. **Encrypt API keys** in database using application secret
2. **Rate limiting** per user to prevent abuse
3. **Key validation** before saving
4. **Audit logging** of API key usage
5. **Key rotation** support
### **Current Implementation:**
- ✅ Keys stored in database (not in code)
- ✅ User isolation via `user_id`
- ✅ HTTPS encryption in transit
- ⚠️ Keys not encrypted at rest (TODO)
---
## Troubleshooting
### **Issue: "No API key found"**
- **Development**: Check `backend/.env` file exists and has keys
- **Production**: Check database has keys for this user:
```sql
SELECT * FROM api_keys
WHERE session_id = (SELECT id FROM onboarding_sessions WHERE user_id = 'user_xxx');
```
### **Issue: "Wrong user's keys being used"**
- **Cause**: Service not using `UserAPIKeyContext`
- **Fix**: Update service to use `user_api_keys(user_id)` context manager
### **Issue: "Keys not saving to .env in development"**
- **Cause**: `DEBUG` not set to `true`
- **Fix**: Set `DEBUG=true` in `backend/.env`
---
## Summary
| Feature | Local Development | Production |
|---------|------------------|------------|
| **Key Storage** | `.env` files + Database | Database only |
| **User Isolation** | Not needed (single user) | Full isolation |
| **Cost** | Your API keys | Each user's API keys |
| **Convenience** | High (keys persist) | Medium (enter once) |
| **Scalability** | 1 developer | Unlimited users |
| **Detection** | `DEBUG=true` | `DEPLOY_ENV` set |
**Bottom Line:**
- 🏠 **Local**: Quick setup, your keys, `.env` convenience
- 🌐 **Production**: User isolation, their keys, zero cost for you
This architecture ensures:
1. ✅ You can develop locally with convenience
2. ✅ Alpha testers use their own keys (no cost to you)
3. ✅ Complete user isolation in production
4. ✅ Seamless transition between environments

299
docs/API_KEY_QUICK_REFERENCE.md Normal file
View File

@@ -0,0 +1,299 @@
# API Key Management - Quick Reference
## 🎯 The Big Picture
**Problem:** You want to develop locally with convenience, but alpha testers should use their own API keys (so you don't pay for their usage).
**Solution:**
- **Local Dev**: API keys saved to `.env` files (convenient)
- **Production**: API keys saved to database per user (isolated, zero cost to you)
---
## 🚀 How It Works
### **1. Local Development (You)**
```bash
# backend/.env
DEBUG=true
GEMINI_API_KEY=your_key_here
EXA_API_KEY=your_exa_key
COPILOTKIT_API_KEY=your_copilot_key
```
**Behavior:**
- ✅ Complete onboarding once
- ✅ Keys saved to `.env` AND database
- ✅ All services use keys from `.env`
- ✅ Convenient, keys persist
**You pay for:** Your own API usage
---
### **2. Production (Alpha Testers)**
```bash
# Render environment variables
DEBUG=false
DEPLOY_ENV=render
DATABASE_URL=postgresql://...
```
**Behavior:**
- ✅ Each tester completes onboarding with their keys
- ✅ Keys saved to database (user-specific rows)
- ✅ Services fetch keys from database per user
- ✅ Complete user isolation
**You pay for:** $0-$7/month (infrastructure only)
**Testers pay for:** Their own API usage
---
## 📝 Code Examples
### **Using User API Keys in Services**
```python
from services.user_api_key_context import user_api_keys
import google.generativeai as genai
def generate_blog(user_id: str, topic: str):
# Get user-specific API keys
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
# Configure Gemini with THIS user's key
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
# Generate content (charges THIS user's Gemini account)
response = model.generate_content(f"Write a blog about {topic}")
return response.text
```
**What this does:**
- **Dev mode** (`user_id=None` or `DEBUG=true`): Uses `.env` file
- **Prod mode** (`DEPLOY_ENV=render`): Fetches from database for this `user_id`
---
## 🔄 Migration Checklist
### **Step 1: Update Environment Variables**
**Local (backend/.env):**
```bash
DEBUG=true
# Your development API keys (stay as-is)
GEMINI_API_KEY=...
EXA_API_KEY=...
```
**Render Dashboard:**
```bash
DEBUG=false
DEPLOY_ENV=render
DATABASE_URL=postgresql://...
# Remove GEMINI_API_KEY, EXA_API_KEY from here!
# Users will provide their own via onboarding
```
### **Step 2: Update Services to Use user_api_keys**
**Before:**
```python
import os
gemini_key = os.getenv('GEMINI_API_KEY') # ❌ Same for all users!
```
**After:**
```python
from services.user_api_key_context import user_api_keys
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini') # ✅ User-specific!
```
### **Step 3: Update FastAPI Endpoints**
**Add user_id parameter:**
```python
@router.post("/api/generate")
async def generate(
prompt: str,
current_user: dict = Depends(get_current_user) # Get authenticated user
):
user_id = current_user.get('user_id') # Extract user_id
# Pass user_id to service
result = await my_service.generate(user_id, prompt)
return result
```
### **Step 4: Test**
**Local:**
1. Complete onboarding
2. Check `backend/.env` has your keys ✅
3. Generate content - should work ✅
**Production:**
1. Deploy to Render with `DEPLOY_ENV=render`
2. User A: Complete onboarding with keys A
3. User B: Complete onboarding with keys B
4. User A generates content → Uses keys A ✅
5. User B generates content → Uses keys B ✅
---
## 🔍 Troubleshooting
### **"No API key found" error**
**In development:**
```bash
# Check backend/.env exists and has:
DEBUG=true
GEMINI_API_KEY=your_key_here
```
**In production:**
```sql
-- Check database has keys for this user:
SELECT s.user_id, k.provider, k.key
FROM api_keys k
JOIN onboarding_sessions s ON k.session_id = s.id
WHERE s.user_id = 'user_xxx';
```
### **Wrong user's keys being used**
**Cause:** Service not using `user_api_keys(user_id)`
**Fix:**
```python
# OLD (wrong):
gemini_key = os.getenv('GEMINI_API_KEY')
# NEW (correct):
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
```
### **Keys not saving to .env in development**
**Cause:** `DEBUG` not set to `true`
**Fix:**
```bash
# backend/.env
DEBUG=true # Must be explicitly true
```
---
## 📊 Cost Breakdown
### **Your Monthly Costs**
| Item | Dev | Production |
|------|-----|------------|
| **Infrastructure** | $0 | $0-7/month |
| **Database** | Free | Free (Render) |
| **API Usage (Gemini, Exa, etc.)** | Your usage | $0 (users pay!) |
| **Total** | Your API usage | $0-7/month |
### **Alpha Tester Costs**
| Item | Cost |
|------|------|
| **ALwrity Subscription** | Free (alpha) |
| **Their Gemini API** | Their usage |
| **Their Exa API** | Their usage |
| **Total** | Their API usage |
---
## 🎓 Key Concepts
### **Environment Detection**
```python
is_development = (
os.getenv('DEBUG', 'false').lower() == 'true' or
os.getenv('DEPLOY_ENV') is None
)
if is_development:
# Use .env file (convenience)
keys = load_from_env()
else:
# Use database (user isolation)
keys = load_from_database(user_id)
```
### **User Isolation**
```
Database guarantees:
┌──────────────────┬─────────────┬──────────────────┐
│ user_id │ provider │ key │
├──────────────────┼─────────────┼──────────────────┤
│ user_tester_a │ gemini │ tester_a_key │ ← Isolated
│ user_tester_b │ gemini │ tester_b_key │ ← Isolated
└──────────────────┴─────────────┴──────────────────┘
Query for Tester A: WHERE user_id = 'user_tester_a'
Query for Tester B: WHERE user_id = 'user_tester_b'
No overlap, no conflicts!
```
---
## 🚀 Quick Start
### **For Local Development:**
1. Clone repo
2. Set `DEBUG=true` in `backend/.env`
3. Add your API keys to `backend/.env`
4. Run backend: `python start_alwrity_backend.py --dev`
5. Complete onboarding (keys auto-save to `.env`)
6. Done! ✅
### **For Production Deployment:**
1. Deploy backend to Render
2. Set environment variables:
- `DEBUG=false`
- `DEPLOY_ENV=render`
- `DATABASE_URL=postgresql://...`
3. Deploy frontend to Vercel
4. Alpha testers complete onboarding with their keys
5. Done! Each tester uses their own keys ✅
---
## 📚 Further Reading
- [Complete Architecture Guide](./API_KEY_MANAGEMENT_ARCHITECTURE.md)
- [Usage Examples](./EXAMPLES_USER_API_KEYS.md)
- [Flow Diagrams](./API_KEY_FLOW_DIAGRAM.md)
---
## ✅ Summary
**The magic:**
- Same codebase works in both dev and prod
- Dev: Convenience of `.env` files
- Prod: Isolation via database
- Zero cost: Testers use their own API keys
- Automatic: Just set `DEBUG` and `DEPLOY_ENV`
**Bottom line:**
> Write code once, works everywhere. Development is convenient, production is isolated. You focus on building, testers pay for their usage. Win-win! 🎉

264
docs/CRITICAL_ONBOARDING_DATABASE_MIGRATION.md Normal file
View File

@@ -0,0 +1,264 @@
# 🚨 CRITICAL: Onboarding Data Must Use Database
## Issue Summary
**Severity:** 🔴 CRITICAL
**Impact:** User isolation, data persistence, security
**Status:** ⚠️ NEEDS IMMEDIATE FIX AFTER DEPLOYMENT STABILIZES
## Problem Description
The onboarding system currently saves all user data to a JSON file (`.onboarding_progress.json`) instead of using the database. This causes multiple critical issues:
### 1. **No User Isolation** 🔴
- All users share the same JSON file
- User data can be overwritten by other users
- Privacy violation - users can see each other's data
- **Line:** `backend/services/api_key_manager.py:45`
- **Code:** `self.progress_file = progress_file or ".onboarding_progress.json"`
### 2. **Data Loss on Deployment** 🔴
- Render uses ephemeral filesystem
- File is deleted on every deployment/restart
- Users lose all onboarding progress
- Have to restart onboarding after each deployment
### 3. **No Scalability** 🔴
- Won't work with multiple backend instances
- File locking issues
- Race conditions
- Performance bottleneck
### 4. **Security Risk** 🔴
- API keys stored in plain text JSON file
- No encryption
- File accessible with filesystem access
- Should be in database with proper security
## Current Architecture
```
User completes step → OnboardingProgress.mark_step_completed()
→ save_progress() (line 214)
→ json.dump(progress_data, ".onboarding_progress.json")
```
**File Location:** `backend/.onboarding_progress.json`
**Affected Code:**
- `backend/services/api_key_manager.py` (OnboardingProgress class)
- `backend/api/onboarding_utils/endpoints_core.py`
- `backend/api/onboarding_utils/step_management_service.py`
## Database Models Available
✅ **Good News:** Proper database models already exist!
**File:** `backend/models/onboarding.py`
```python
- OnboardingSession (user_id, current_step, progress, started_at, updated_at)
- APIKey (session_id, provider, key, created_at, updated_at)
- WebsiteAnalysis (session_id, website_url, analysis_date, writing_style, etc.)
- ResearchPreferences (session_id, research_depth, content_types, etc.)
```
**Database Schema:**
- ✅ User isolation via `user_id` and `session_id`
- ✅ Proper relationships and foreign keys
- ✅ Timestamps for audit trail
- ✅ JSON fields for complex data
- ✅ Cascade deletes for cleanup
## Required Changes
### Phase 1: Database Layer (Priority 1)
**File:** `backend/services/onboarding_database_service.py` (NEW)
```python
class OnboardingDatabaseService:
"""Database-backed onboarding service replacing JSON file storage."""
def get_or_create_session(self, user_id: str) -> OnboardingSession:
"""Get existing session or create new one."""
def get_progress(self, user_id: str) -> OnboardingProgress:
"""Load progress from database."""
def save_step_data(self, user_id: str, step_number: int, data: Dict):
"""Save step data to database."""
def mark_step_completed(self, user_id: str, step_number: int):
"""Mark step as completed in database."""
def get_step_data(self, user_id: str, step_number: int) -> Dict:
"""Retrieve step data from database."""
```
### Phase 2: Refactor API Key Manager (Priority 1)
**File:** `backend/services/api_key_manager.py`
**Changes:**
1. Remove JSON file operations (lines 214-242)
2. Add database dependency injection
3. Replace `save_progress()` with database calls
4. Replace `load_progress()` with database queries
5. Add user_id parameter to all methods
**Before:**
```python
def mark_step_completed(self, step_number: int, data: Optional[Dict] = None):
# ... update in-memory state ...
self.save_progress() # Saves to JSON file
```
**After:**
```python
def mark_step_completed(self, user_id: str, step_number: int, data: Optional[Dict] = None):
# ... update database ...
db_service.save_step_data(user_id, step_number, data)
db_service.mark_step_completed(user_id, step_number)
```
### Phase 3: Update Endpoints (Priority 2)
**Files to Update:**
- `backend/api/onboarding_utils/endpoints_core.py`
- `backend/api/onboarding_utils/step_management_service.py`
- `backend/api/onboarding_utils/step3_routes.py`
- `backend/api/onboarding_utils/step4_persona_routes.py`
**Changes:**
1. Pass `user_id` from `get_current_user` to all service calls
2. Remove file-based caching
3. Use database queries for progress retrieval
### Phase 4: Migration Script (Priority 3)
**File:** `backend/scripts/migrate_onboarding_to_database.py` (NEW)
```python
def migrate_json_to_database():
"""
Migrate existing .onboarding_progress.json to database.
Only needed if production has existing data in JSON files.
"""
# Read JSON file
# Create database records
# Backup JSON file
# Delete JSON file
```
## Implementation Plan
### Step 1: Create Database Service (1-2 hours)
- [ ] Create `onboarding_database_service.py`
- [ ] Implement CRUD operations
- [ ] Add user isolation checks
- [ ] Write unit tests
### Step 2: Refactor API Key Manager (2-3 hours)
- [ ] Remove JSON file operations
- [ ] Add database calls
- [ ] Update method signatures with user_id
- [ ] Test with database
### Step 3: Update Endpoints (1-2 hours)
- [ ] Pass user_id to service calls
- [ ] Remove file-based logic
- [ ] Test each endpoint
### Step 4: Testing (2-3 hours)
- [ ] Test user isolation
- [ ] Test data persistence across deployments
- [ ] Test concurrent users
- [ ] Test error handling
### Step 5: Deployment (1 hour)
- [ ] Deploy to staging
- [ ] Run migration script if needed
- [ ] Deploy to production
- [ ] Monitor for issues
**Total Estimated Time:** 8-12 hours
## Temporary Mitigation
Until this is fixed, we must:
1. ✅ Add `.onboarding_progress.json` to `.gitignore`
2. ✅ Document that onboarding data will be lost on deployment
3. ⚠️ Warn users that onboarding must be completed in one session
4. ⚠️ Consider using Render's persistent disk (expensive workaround)
## Testing Checklist
After migration:
- [ ] User A completes onboarding
- [ ] User B completes onboarding
- [ ] Verify User A and User B data are separate
- [ ] Redeploy backend
- [ ] Verify both users' data persists
- [ ] User C starts onboarding
- [ ] Verify User C doesn't see User A or B data
- [ ] Test concurrent onboarding (multiple users at once)
- [ ] Verify API keys are stored securely
- [ ] Test onboarding restart (partial completion)
## Security Considerations
### Current (Insecure):
```json
{
"steps": [
{
"step_number": 1,
"data": {
"api_keys": {
"gemini": "ACTUAL_API_KEY_HERE",
"exa": "ACTUAL_API_KEY_HERE"
}
}
}
]
}
```
### After Migration (Secure):
- API keys in database with user isolation
- Encrypted at rest (if database supports it)
- Access controlled by user_id
- Audit trail via timestamps
## References
- Database Models: `backend/models/onboarding.py`
- Current Implementation: `backend/services/api_key_manager.py`
- Endpoints: `backend/api/onboarding_utils/`
- Issue tracking: GitHub Issue #XXX (to be created)
## Priority
**This must be fixed before:**
- ❌ Going to production with real users
- ❌ Accepting paying customers
- ❌ Handling sensitive data
- ❌ Scaling to multiple instances
**Acceptable to delay if:**
- ✅ Still in alpha/beta with limited users
- ✅ Users aware of data loss on deployment
- ✅ Not handling production workloads yet
## Conclusion
This is a critical architectural flaw that violates basic principles:
- User data isolation
- Data persistence
- Security best practices
- Scalability
**Must be fixed immediately after current deployment stabilizes.**

489
docs/EXAMPLES_USER_API_KEYS.md Normal file
View File

@@ -0,0 +1,489 @@
# User API Key Context - Usage Examples
This document shows how to use the `UserAPIKeyContext` in your backend services to ensure user-specific API keys are used.
## Quick Start
### **1. Basic Usage in FastAPI Endpoint**
```python
from fastapi import APIRouter, Depends
from middleware.auth_middleware import get_current_user
from services.user_api_key_context import user_api_keys
import google.generativeai as genai
router = APIRouter()
@router.post("/api/generate-content")
async def generate_content(
prompt: str,
current_user: dict = Depends(get_current_user)
):
user_id = current_user.get('user_id')
# Get user-specific API keys
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
if not gemini_key:
raise HTTPException(status_code=400, detail="Gemini API key not configured")
# Configure Gemini with user's key
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
# Generate content using this user's quota
response = model.generate_content(prompt)
return {
"content": response.text,
"user_id": user_id # For debugging
}
```
---
## Examples by Use Case
### **Example 1: Blog Writer Service**
**File: `backend/services/blog_writer_service.py`**
```python
from services.user_api_key_context import user_api_keys, get_gemini_key
import google.generativeai as genai
class BlogWriterService:
"""
Service for generating blog content using user-specific API keys.
"""
def __init__(self, user_id: str):
self.user_id = user_id
async def generate_blog_outline(self, topic: str) -> dict:
"""Generate blog outline using user's Gemini API key."""
# Method 1: Using context manager (recommended)
with user_api_keys(self.user_id) as keys:
gemini_key = keys.get('gemini')
if not gemini_key:
raise ValueError(f"No Gemini API key found for user {self.user_id}")
# Configure Gemini with user's key
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
prompt = f"Create a detailed blog outline for: {topic}"
response = model.generate_content(prompt)
return {
"outline": response.text,
"topic": topic,
"user_id": self.user_id
}
async def generate_blog_section(self, section_heading: str, context: str) -> str:
"""Generate blog section using user's Gemini API key."""
# Method 2: Using convenience function
gemini_key = get_gemini_key(self.user_id)
if not gemini_key:
raise ValueError(f"No Gemini API key found for user {self.user_id}")
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
prompt = f"Write a blog section for '{section_heading}'\n\nContext: {context}"
response = model.generate_content(prompt)
return response.text
```
**Usage in FastAPI:**
```python
from fastapi import APIRouter, Depends
from middleware.auth_middleware import get_current_user
from services.blog_writer_service import BlogWriterService
router = APIRouter()
@router.post("/api/blog/outline")
async def create_blog_outline(
topic: str,
current_user: dict = Depends(get_current_user)
):
user_id = current_user.get('user_id')
# Create service instance with user_id
blog_service = BlogWriterService(user_id)
# Service automatically uses this user's API keys
outline = await blog_service.generate_blog_outline(topic)
return outline
```
---
### **Example 2: Research Service with Multiple APIs**
**File: `backend/services/research_service.py`**
```python
from services.user_api_key_context import user_api_keys
from exa_py import Exa
import google.generativeai as genai
class ResearchService:
"""
Service for conducting research using user-specific API keys.
"""
def __init__(self, user_id: str):
self.user_id = user_id
async def conduct_research(self, query: str) -> dict:
"""
Conduct research using both Exa (search) and Gemini (analysis).
Uses user-specific API keys for both services.
"""
with user_api_keys(self.user_id) as keys:
exa_key = keys.get('exa')
gemini_key = keys.get('gemini')
if not exa_key or not gemini_key:
raise ValueError(f"Missing required API keys for user {self.user_id}")
# 1. Search using user's Exa API key
exa = Exa(api_key=exa_key)
search_results = exa.search_and_contents(
query,
num_results=5,
text=True
)
# 2. Analyze results using user's Gemini API key
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
# Prepare context from search results
context = "\n\n".join([
f"Source: {r.url}\n{r.text[:500]}..."
for r in search_results.results
])
prompt = f"""
Analyze the following research results for query: "{query}"
{context}
Provide a comprehensive summary and key insights.
"""
analysis = model.generate_content(prompt)
return {
"query": query,
"sources": [r.url for r in search_results.results],
"analysis": analysis.text,
"user_id": self.user_id # For debugging
}
```
---
### **Example 3: Persona Generation Service**
**File: `backend/services/persona/core_persona_service.py`**
```python
from services.user_api_key_context import user_api_keys, get_gemini_key
import google.generativeai as genai
from typing import Optional
class CorePersonaService:
"""
Service for generating AI writing personas.
"""
def generate_core_persona(
self,
onboarding_data: dict,
user_id: Optional[str] = None
) -> dict:
"""
Generate core persona using user's Gemini API key.
Args:
onboarding_data: User's onboarding information
user_id: User ID (optional - uses .env in dev mode if None)
"""
# Get user-specific Gemini key
# In dev mode (user_id=None), this uses .env
# In prod mode, this fetches from database
gemini_key = get_gemini_key(user_id)
if not gemini_key:
if user_id:
raise ValueError(f"No Gemini API key found for user {user_id}")
else:
raise ValueError("No Gemini API key found in .env file")
# Configure Gemini
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
# Extract user's business info
business_data = onboarding_data.get('businessData', {})
website_analysis = onboarding_data.get('websiteAnalysis', {})
prompt = f"""
Generate an AI writing persona based on:
Business: {business_data.get('name')}
Industry: {business_data.get('industry')}
Tone: {website_analysis.get('tone')}
Create a detailed writing persona including voice, style, and personality.
"""
response = model.generate_content(prompt)
return {
"persona": response.text,
"user_id": user_id,
"source": "dev_env" if user_id is None else "user_database"
}
```
---
### **Example 4: Background Task with User Keys**
**File: `backend/services/async_content_generator.py`**
```python
from fastapi import BackgroundTasks
from services.user_api_key_context import user_api_keys
import google.generativeai as genai
async def generate_content_background(
user_id: str,
task_id: str,
prompt: str,
callback_url: str = None
):
"""
Background task that generates content using user's API keys.
This runs asynchronously and doesn't block the API response.
"""
try:
# Get user-specific API keys
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
if not gemini_key:
# Log error and notify user
logger.error(f"No Gemini API key for user {user_id} in task {task_id}")
return
# Configure Gemini
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
# Generate content (this may take a while)
response = model.generate_content(prompt)
# Save to database or send callback
if callback_url:
# Notify user that content is ready
await send_callback(callback_url, {
"task_id": task_id,
"content": response.text,
"status": "completed"
})
logger.info(f"Task {task_id} completed for user {user_id}")
except Exception as e:
logger.error(f"Task {task_id} failed for user {user_id}: {e}")
# Usage in FastAPI endpoint
@router.post("/api/generate-async")
async def generate_async(
prompt: str,
background_tasks: BackgroundTasks,
current_user: dict = Depends(get_current_user)
):
user_id = current_user.get('user_id')
task_id = str(uuid.uuid4())
# Queue background task
background_tasks.add_task(
generate_content_background,
user_id=user_id,
task_id=task_id,
prompt=prompt
)
return {
"task_id": task_id,
"status": "queued",
"message": "Content generation started"
}
```
---
### **Example 5: Migrating Existing Service**
**Before (WRONG - uses global .env):**
```python
import os
import google.generativeai as genai
class OldBlogService:
def generate_content(self, prompt: str):
# BAD: Uses same API key for all users!
gemini_key = os.getenv('GEMINI_API_KEY')
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
response = model.generate_content(prompt)
return response.text
```
**After (CORRECT - uses user-specific keys):**
```python
from services.user_api_key_context import user_api_keys
import google.generativeai as genai
class NewBlogService:
def __init__(self, user_id: str):
self.user_id = user_id
def generate_content(self, prompt: str):
# GOOD: Uses user-specific API key!
with user_api_keys(self.user_id) as keys:
gemini_key = keys.get('gemini')
if not gemini_key:
raise ValueError(f"No Gemini API key for user {self.user_id}")
genai.configure(api_key=gemini_key)
model = genai.GenerativeModel('gemini-pro')
response = model.generate_content(prompt)
return response.text
```
---
## Best Practices
### ✅ **DO:**
1. **Always pass `user_id` to services:**
```python
service = BlogWriterService(user_id=current_user.get('user_id'))
```
2. **Use context manager for multiple keys:**
```python
with user_api_keys(user_id) as keys:
gemini_key = keys.get('gemini')
exa_key = keys.get('exa')
```
3. **Check for missing keys:**
```python
if not gemini_key:
raise HTTPException(status_code=400, detail="Please configure your Gemini API key")
```
4. **Log which user's keys are being used:**
```python
logger.info(f"Generating content for user {user_id} with their API keys")
```
### ❌ **DON'T:**
1. **Don't use `os.getenv()` directly:**
```python
# WRONG - same key for all users!
gemini_key = os.getenv('GEMINI_API_KEY')
```
2. **Don't forget to pass `user_id`:**
```python
# WRONG - will use .env even in production!
with user_api_keys() as keys: # Missing user_id!
```
3. **Don't hardcode API keys:**
```python
# WRONG - security risk!
genai.configure(api_key="AIzaSy...")
```
---
## Testing
### **Test in Development:**
```python
# Set DEBUG=true in backend/.env
# Then test:
def test_dev_mode():
# user_id=None should use .env file
with user_api_keys(user_id=None) as keys:
assert keys.get('gemini') == os.getenv('GEMINI_API_KEY')
```
### **Test in Production:**
```python
# Set DEBUG=false and DEPLOY_ENV=render
# Then test:
def test_prod_mode():
# Should fetch from database
user_id = "user_12345"
with user_api_keys(user_id) as keys:
# Keys should come from database, not .env
assert keys.get('gemini') != os.getenv('GEMINI_API_KEY')
```
---
## Summary
| Method | Use Case | Example |
|--------|----------|---------|
| `user_api_keys(user_id)` | Multiple keys needed | Research service (Exa + Gemini) |
| `get_gemini_key(user_id)` | Single key needed | Blog writer (only Gemini) |
| `get_exa_key(user_id)` | Single key needed | Search service (only Exa) |
| `get_user_api_keys(user_id)` | FastAPI dependency | Endpoint that needs all keys |
**Key Principle:**
> Always pass `user_id` to get user-specific API keys. In development (`user_id=None`), it uses `.env` for convenience.
This ensures:
- ✅ **Local dev**: Your keys from `.env`
- ✅ **Production**: Each user's keys from database
- ✅ **Zero cost**: Alpha testers use their own API keys
- ✅ **User isolation**: No conflicts between users

215
docs/PERSONA_DATA_MIGRATION_GUIDE.md Normal file
View File

@@ -0,0 +1,215 @@
# Persona Data Table Migration Guide
## Overview
This guide explains how to create the `persona_data` table for storing Step 4 (Persona Generation) data from the onboarding flow.
## Background
The `persona_data` table was missing from the database schema, causing Step 4 onboarding data to only be saved to JSON files instead of the database. This migration adds the required table with proper user isolation.
## Migration Methods
### Method 1: Automatic Migration (Recommended)
The easiest way is to restart your backend server. The table will be created automatically when the application starts.
```bash
# Stop the backend if running (Ctrl+C)
# Then restart it:
python backend/start_alwrity_backend.py --dev
```
**How it works:**
- The `init_database()` function in `backend/services/database.py` (line 69) calls `OnboardingBase.metadata.create_all(bind=engine)`
- This automatically creates all missing tables defined in the `OnboardingBase` models
- Since we added the `PersonaData` model, it will be created on startup
### Method 2: Manual Migration Script
If you prefer to run the migration manually without restarting the backend:
```bash
# From the project root directory:
python backend/scripts/create_persona_data_table.py
```
**What this script does:**
1. Checks if the `persona_data` table already exists
2. Creates the table if it doesn't exist
3. Verifies the table was created successfully
4. Shows the table structure (columns and types)
5. Lists all onboarding-related tables and their status
### Method 3: SQL Migration (Production/Manual)
For production environments or manual database management:
```bash
# Connect to your PostgreSQL database and run:
psql -U your_username -d your_database -f backend/database/migrations/add_persona_data_table.sql
```
**Or using psql command:**
```sql
-- Connect to your database
\c your_database
-- Run the migration
\i backend/database/migrations/add_persona_data_table.sql
-- Verify the table was created
\dt persona_data
\d persona_data
```
## Table Structure
The `persona_data` table includes:
| Column | Type | Description |
|--------|------|-------------|
| `id` | SERIAL | Primary key |
| `session_id` | INTEGER | Foreign key to `onboarding_sessions.id` |
| `core_persona` | JSONB | Core persona data (demographics, psychographics, etc.) |
| `platform_personas` | JSONB | Platform-specific personas (LinkedIn, Twitter, etc.) |
| `quality_metrics` | JSONB | Quality assessment metrics |
| `selected_platforms` | JSONB | Array of selected platforms |
| `created_at` | TIMESTAMP | When the record was created |
| `updated_at` | TIMESTAMP | When the record was last updated |
**Indexes:**
- `idx_persona_data_session_id` - For efficient session lookups
- `idx_persona_data_created_at` - For time-based queries
**Constraints:**
- Foreign key to `onboarding_sessions.id` with `ON DELETE CASCADE`
## Verification
After running the migration, verify it was successful:
### Using Python:
```python
from services.database import engine
from sqlalchemy import inspect
inspector = inspect(engine)
tables = inspector.get_table_names()
if 'persona_data' in tables:
print("✅ persona_data table exists")
columns = inspector.get_columns('persona_data')
for col in columns:
print(f" - {col['name']}: {col['type']}")
else:
print("❌ persona_data table not found")
```
### Using SQL:
```sql
-- Check if table exists
SELECT EXISTS (
SELECT FROM information_schema.tables
WHERE table_name = 'persona_data'
);
-- Show table structure
\d persona_data
```
### Using the Backend Logs:
After restarting the backend, look for this log message:
```
Database initialized successfully with all models including subscription system and business info
```
Then, when a user completes Step 4, you should see:
```
✅ DATABASE: Persona data saved to database for user user_xxxxx
```
## Expected Behavior After Migration
Once the table is created and the backend is running with the updated code:
1. **Step 4 Completion:**
- Persona data (corePersona, platformPersonas, qualityMetrics, selectedPlatforms) is saved to the database
- Database logs confirm: `✅ DATABASE: Persona data saved to database for user {user_id}`
2. **User Isolation:**
- Each user's persona data is stored separately using their `user_id`
- Data is linked to the user's onboarding session
3. **Data Persistence:**
- Persona data is no longer lost when JSON files are deleted
- Data survives backend restarts
- Data is accessible across different sessions
## Troubleshooting
### Table Already Exists Error
If you see "table already exists" errors:
- This is normal! It means the table was already created
- The migration scripts use `CREATE TABLE IF NOT EXISTS` to handle this
- No action needed
### Permission Denied
If you get permission errors:
```
ERROR: permission denied for schema public
```
**Solution:** Ensure your database user has CREATE TABLE permissions:
```sql
GRANT CREATE ON SCHEMA public TO your_database_user;
```
### Foreign Key Constraint Fails
If the `onboarding_sessions` table doesn't exist:
1. Run the full database initialization first:
```python
from services.database import init_database
init_database()
```
2. Then create the `persona_data` table
### Missing Database Connection
If you see "database connection" errors:
1. Check your `DATABASE_URL` environment variable
2. Ensure PostgreSQL/SQLite is running
3. Verify database credentials
## Rollback (If Needed)
To remove the `persona_data` table:
```sql
DROP TABLE IF EXISTS persona_data CASCADE;
```
**Warning:** This will delete all persona data. Use with caution!
## Related Files
- **Model:** `backend/models/onboarding.py` - `PersonaData` class (lines 149-183)
- **Service:** `backend/services/onboarding_database_service.py` - `save_persona_data()` method (lines 298-338)
- **Migration:** `backend/database/migrations/add_persona_data_table.sql`
- **Script:** `backend/scripts/create_persona_data_table.py`
- **Database Init:** `backend/services/database.py` - `init_database()` function (line 63-80)
## Summary
**Recommended approach for local development:**
```bash
# Just restart the backend - the table will be created automatically!
python backend/start_alwrity_backend.py --dev
```
**For production deployment:**
- The table will be created automatically on first deployment
- Or run the SQL migration manually before deployment
- No downtime required - the migration is additive only
## Questions?
If you encounter issues:
1. Check the backend logs for detailed error messages
2. Verify all onboarding tables exist using the verification script
3. Ensure your database user has proper permissions
4. Check that the `PersonaData` model is imported correctly in `backend/services/onboarding_database_service.py`