Base code

2026-01-08 22:39:53 +07:00
parent 697115c61a
commit c35fa52117
2169 changed files with 626670 additions and 0 deletions
--- a/docs/sse_migration_strategy.md
+++ b/docs/sse_migration_strategy.md
@@ -0,0 +1,343 @@
+# SSE Migration Strategy & Implementation Plan
+
+## 🚨 **Current Implementation Problems**
+
+### **Backend Issues**
+- **Complex SSE Manager**: The `SSEAIServiceManager` with lambda functions is overly complex
+- **Async Generator Problems**: The `sse_yield` function using `__anext__()` is fragile
+- **Message Format Inconsistency**: Backend sends different message formats that frontend struggles to parse
+- **Tight Coupling**: AI service manager is tightly coupled to SSE implementation
+- **Error Propagation**: Errors in one component cascade to others
+- **Debugging Difficulty**: Complex async flows make debugging hard
+
+### **Frontend Issues**
+- **EventSource Limitations**: No built-in reconnection, poor error handling
+- **Message Parsing Complexity**: Too many message types to handle
+- **Timeout Handling**: Frontend timeouts don't align with backend processing
+- **Connection State Management**: Poor handling of connection states
+- **Progress Tracking**: Inconsistent progress calculation and display
+
+### **Architecture Problems**
+- **Tight Coupling**: Frontend and backend are tightly coupled to specific message formats
+- **No Reusability**: SSE implementation is specific to strategy generation
+- **Error Handling**: Inconsistent error handling across components
+- **Testing Difficulty**: Complex async flows make testing challenging
+
+## 🎯 **Proposed Solution: Clean SSE with sse-starlette**
+
+### **Phase 1: MVP Polling Solution (1-2 hours)**
+**Goal**: Get strategy generation working immediately with simple polling
+
+**Implementation**:
+- Replace complex SSE with simple polling mechanism
+- Poll strategy status every 10 seconds
+- Show progress modal with educational content
+- Handle timeouts gracefully
+- Remove all SSE-related complexity
+
+**Benefits**:
+- ✅ Immediate working solution
+- ✅ Simple to implement and debug
+- ✅ Reliable and predictable
+- ✅ Easy to test
+
+### **Phase 2: Proper SSE Implementation (1-2 days)**
+**Goal**: Implement clean, reusable SSE infrastructure
+
+**Implementation**:
+- Use `sse-starlette` for backend SSE
+- Create reusable SSE client for frontend
+- Standardize message format
+- Add proper error handling and reconnection
+- Make SSE infrastructure reusable for other features
+
+**Benefits**:
+- ✅ Real-time updates
+- ✅ Better user experience
+- ✅ Reusable infrastructure
+- ✅ Proper error handling
+
+## 🏗️ **Technical Architecture**
+
+### **Backend: sse-starlette Implementation**
+
+#### **Core SSE Module** (`backend/services/sse/`)
+```
+backend/services/sse/
+├── __init__.py
+├── sse_manager.py          # Core SSE management
+├── message_formatter.py    # Standardized message formatting
+├── connection_manager.py   # Connection lifecycle management
+├── error_handler.py        # SSE error handling
+└── types.py               # SSE message types and schemas
+```
+
+#### **SSE Manager Features**
+- **Connection Management**: Handle multiple SSE connections
+- **Message Broadcasting**: Send messages to specific clients
+- **Error Handling**: Graceful error handling and recovery
+- **Message Formatting**: Consistent message format across all features
+- **Connection Monitoring**: Track connection health and status
+
+#### **Message Format Standardization**
+```python
+# Standard SSE message format
+{
+    "event": "progress|complete|error|educational",
+    "data": {
+        "step": 1,
+        "progress": 10,
+        "message": "Processing...",
+        "educational_content": {...},
+        "timestamp": "2024-01-01T00:00:00Z"
+    }
+}
+```
+
+### **Frontend: Reusable SSE Client**
+
+#### **Core SSE Module** (`frontend/src/services/sse/`)
+```
+frontend/src/services/sse/
+├── index.ts
+├── SSEConnection.ts        # Core SSE connection management
+├── SSEEventManager.ts      # Event handling and message parsing
+├── SSEReconnection.ts      # Automatic reconnection logic
+├── SSEMessageTypes.ts      # TypeScript types for messages
+└── SSEUtils.ts            # Utility functions
+```
+
+#### **SSE Client Features**
+- **Automatic Reconnection**: Handle connection drops gracefully
+- **Message Parsing**: Parse standardized message format
+- **Event Handling**: Handle different event types
+- **Error Recovery**: Recover from errors automatically
+- **Connection Monitoring**: Monitor connection health
+
+#### **React Hook** (`frontend/src/hooks/useSSE.ts`)
+```typescript
+const useSSE = (url: string, options?: SSEOptions) => {
+  // Returns: { data, error, isConnected, reconnect }
+}
+```
+
+## 📋 **Implementation Phases**
+
+### **Phase 1: MVP Polling (Immediate - 1-2 hours)**
+
+#### **Backend Changes**
+1. **Remove SSE complexity** from `ai_generation_endpoints.py`
+2. **Simplify AI generation** to return immediately after starting
+3. **Add status endpoint** to check generation progress
+4. **Remove SSEAIServiceManager** and related complexity
+
+#### **Frontend Changes**
+1. **Replace SSE with polling** in `ContentStrategyBuilder.tsx`
+2. **Implement simple progress modal** with educational content
+3. **Add polling mechanism** (every 10 seconds)
+4. **Handle timeouts gracefully** (5-minute timeout)
+5. **Remove all SSE-related code**
+
+#### **Files to Modify**
+- `backend/api/content_planning/api/content_strategy/endpoints/ai_generation_endpoints.py`
+- `frontend/src/components/ContentPlanningDashboard/components/ContentStrategyBuilder.tsx`
+- `frontend/src/services/contentPlanningApi.ts`
+
+### **Phase 2: Clean SSE Infrastructure (1-2 days)**
+
+#### **Backend Implementation**
+1. **Create SSE infrastructure** (`backend/services/sse/`)
+2. **Implement sse-starlette endpoints** for strategy generation
+3. **Standardize message format** across all SSE endpoints
+4. **Add connection management** and error handling
+5. **Create reusable SSE utilities**
+
+#### **Frontend Implementation**
+1. **Create SSE client infrastructure** (`frontend/src/services/sse/`)
+2. **Implement React hook** for SSE connections
+3. **Add automatic reconnection** logic
+4. **Standardize message parsing** and event handling
+5. **Create reusable SSE components**
+
+#### **New Files to Create**
+```
+Backend:
+- backend/services/sse/__init__.py
+- backend/services/sse/sse_manager.py
+- backend/services/sse/message_formatter.py
+- backend/services/sse/connection_manager.py
+- backend/services/sse/error_handler.py
+- backend/services/sse/types.py
+
+Frontend:
+- frontend/src/services/sse/index.ts
+- frontend/src/services/sse/SSEConnection.ts
+- frontend/src/services/sse/SSEEventManager.ts
+- frontend/src/services/sse/SSEReconnection.ts
+- frontend/src/services/sse/SSEMessageTypes.ts
+- frontend/src/services/sse/SSEUtils.ts
+- frontend/src/hooks/useSSE.ts
+```
+
+### **Phase 3: Migration & Testing (1 day)**
+
+#### **Migration Steps**
+1. **Migrate strategy generation** to new SSE infrastructure
+2. **Test end-to-end functionality** with new SSE
+3. **Add comprehensive error handling** and recovery
+4. **Implement educational content** streaming
+5. **Add monitoring and logging** for SSE connections
+
+#### **Testing Strategy**
+1. **Unit tests** for SSE infrastructure
+2. **Integration tests** for SSE endpoints
+3. **End-to-end tests** for strategy generation
+4. **Error scenario testing** (network drops, timeouts)
+5. **Performance testing** (multiple concurrent connections)
+
+## 🔧 **Technical Specifications**
+
+### **Backend SSE Manager Interface**
+```python
+class SSEManager:
+    async def create_connection(self, client_id: str) -> SSEConnection
+    async def send_message(self, client_id: str, message: SSEMessage)
+    async def broadcast_message(self, message: SSEMessage, filter_func=None)
+    async def close_connection(self, client_id: str)
+    async def get_connection_status(self, client_id: str) -> ConnectionStatus
+```
+
+### **Frontend SSE Client Interface**
+```typescript
+interface SSEConnection {
+  connect(): Promise<void>
+  disconnect(): void
+  send(message: SSEMessage): void
+  on(event: string, handler: EventHandler): void
+  off(event: string, handler: EventHandler): void
+  isConnected(): boolean
+  reconnect(): Promise<void>
+}
+```
+
+### **Message Format Specification**
+```typescript
+interface SSEMessage {
+  event: 'progress' | 'complete' | 'error' | 'educational' | 'status'
+  data: {
+    step?: number
+    progress?: number
+    message?: string
+    educational_content?: EducationalContent
+    error?: string
+    timestamp: string
+    [key: string]: any
+  }
+}
+```
+
+## 🎯 **Success Criteria**
+
+### **Phase 1 Success Criteria**
+- ✅ Strategy generation works reliably
+- ✅ No more "Request timed out" errors
+- ✅ Users can see progress and educational content
+- ✅ Simple, debuggable implementation
+- ✅ Strategy creation completes successfully
+
+### **Phase 2 Success Criteria**
+- ✅ Real-time progress updates via SSE
+- ✅ Automatic reconnection on network issues
+- ✅ Standardized message format across features
+- ✅ Reusable SSE infrastructure
+- ✅ Proper error handling and recovery
+- ✅ Educational content streaming
+
+### **Phase 3 Success Criteria**
+- ✅ All features migrated to new SSE infrastructure
+- ✅ Comprehensive testing coverage
+- ✅ Performance meets requirements
+- ✅ Error scenarios handled gracefully
+- ✅ Monitoring and logging in place
+
+## 🚀 **Migration Benefits**
+
+### **Immediate Benefits (Phase 1)**
+- **Reliability**: No more timeout errors
+- **Simplicity**: Easy to debug and maintain
+- **User Experience**: Clear progress feedback
+- **Stability**: Predictable behavior
+
+### **Long-term Benefits (Phase 2+)**
+- **Reusability**: SSE infrastructure for other features
+- **Real-time Updates**: Better user experience
+- **Scalability**: Handle multiple concurrent connections
+- **Maintainability**: Clean, modular architecture
+- **Extensibility**: Easy to add new SSE features
+
+## 📝 **Implementation Notes**
+
+### **Dependencies**
+- **Backend**: `sse-starlette` package
+- **Frontend**: No additional dependencies (uses native EventSource)
+
+### **Configuration**
+- **SSE Timeout**: 5 minutes for long-running operations
+- **Reconnection**: Exponential backoff (1s, 2s, 4s, 8s, max 30s)
+- **Message Format**: JSON with standardized structure
+- **Error Handling**: Graceful degradation with fallback options
+
+### **Monitoring & Logging**
+- **Connection Status**: Track active connections
+- **Message Flow**: Log message types and frequencies
+- **Error Tracking**: Monitor and alert on SSE errors
+- **Performance Metrics**: Track response times and throughput
+
+### **Security Considerations**
+- **Authentication**: Validate client connections
+- **Rate Limiting**: Prevent abuse of SSE endpoints
+- **Message Validation**: Validate all incoming messages
+- **Connection Limits**: Limit concurrent connections per user
+
+## 🔄 **Rollback Plan**
+
+### **If Phase 1 Fails**
+- Revert to current SSE implementation
+- Keep polling as fallback option
+- Document issues for future reference
+
+### **If Phase 2 Fails**
+- Keep Phase 1 polling implementation
+- Identify specific issues with sse-starlette
+- Consider alternative SSE libraries or WebSocket implementation
+
+### **If Phase 3 Fails**
+- Rollback to Phase 2 implementation
+- Fix specific issues identified during testing
+- Re-run migration with fixes
+
+## 📚 **References & Resources**
+
+### **Documentation**
+- [sse-starlette Documentation](https://github.com/sysid/sse-starlette)
+- [Server-Sent Events MDN](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events)
+- [EventSource API](https://developer.mozilla.org/en-US/docs/Web/API/EventSource)
+
+### **Best Practices**
+- [SSE Best Practices](https://html.spec.whatwg.org/multipage/server-sent-events.html)
+- [Real-time Web Applications](https://web.dev/real-time-web-applications/)
+- [Error Handling in SSE](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Error_handling)
+
+### **Examples & Templates**
+- [sse-starlette Examples](https://github.com/sysid/sse-starlette/tree/main/examples)
+- [React SSE Hook Examples](https://github.com/facebook/react/tree/main/packages/react-dom/src/events)
+- [FastAPI SSE Examples](https://fastapi.tiangolo.com/advanced/websockets/)
+
+---
+
+**Next Steps**: 
+1. Commit current code
+2. Refresh session
+3. Start Phase 1 implementation (MVP polling)
+4. Test strategy generation works
+5. Proceed to Phase 2 (clean SSE infrastructure)