moreminimore-vibe/README-CUSTOM-INTEGRATION.md

# Dyad Custom Features Integration Guide

This guide explains how to use the custom features integration script to maintain your remove-limit modifications when updating the main Dyad codebase.

## Overview

The integration script (`scripts/integrate-custom-features.sh`) helps you merge your custom remove-limit features with upstream updates from the original Dyad repository. It automatically:

- Creates backups before making changes
- Detects custom modifications in key files
- Creates missing custom files
- Validates the integration
- Provides restore functionality

## Features Integrated

### Remove Limit Features
- **Smart Context Management**: Advanced context handling with rolling summaries
- **Rate Limit Simulation**: Configurable rate limiting for testing
- **Enhanced Chat Streaming**: Improved message handling with custom payloads
- **Context Snippet Management**: Intelligent snippet organization and retrieval

### Smart Context System
- **Context Caching**: Efficient caching with TTL
- **Rolling Summaries**: Automatic summarization when context exceeds thresholds
- **Snippet Importance Scoring**: Intelligent ranking of context snippets
- **Size Management**: Automatic trimming and optimization

## Usage

### Basic Integration

```bash
# Integrate custom features (creates backup automatically)
./scripts/integrate-custom-features.sh integrate

# Validate current integration
./scripts/integrate-custom-features.sh validate

# Show help
./scripts/integrate-custom-features.sh help
```

### Restore from Backup

```bash
# List available backups
ls -la backups/

# Restore from a specific backup
./scripts/integrate-custom-features.sh restore backup-20231201-120000
```

## Files Managed

### Core Custom Files
- `src/components/HelpDialog.tsx` - Enhanced help dialog with custom features
- `src/components/chat/PromoMessage.tsx` - Modified promo messages
- `src/ipc/handlers/chat_stream_handlers.ts` - Enhanced chat streaming with remove-limit
- `src/ipc/ipc_client.ts` - Extended IPC client with smart context methods
- `src/ipc/ipc_host.ts` - Updated IPC host with new handlers
- `src/ipc/ipc_types.ts` - Extended type definitions
- `src/preload.ts` - Updated preload script with new channels
- `testing/fake-llm-server/chatCompletionHandler.ts` - Enhanced testing server

### New Custom Files
- `src/ipc/handlers/smart_context_handlers.ts` - Smart context IPC handlers
- `src/ipc/utils/smart_context_store.ts` - Context storage and management
- `src/hooks/useSmartContext.ts` - React hooks for smart context

## Complete Step-by-Step Integration Guide

This section provides detailed, copy-paste ready commands for every step of the integration process. Follow these commands exactly as written.

### 📋 Pre-Update Checklist (5 minutes)

**Step 1: Verify Current Working Directory**
```bash
# Make sure you're in the correct directory
pwd
# Expected output: /Users/kunthawatgreethong/Gitea/moreminimore-vibe

# If not, navigate to the correct directory
cd /Users/kunthawatgreethong/Gitea/moreminimore-vibe
```

**Step 2: Check Git Status**
```bash
# Check if you have any uncommitted changes
git status
# Expected: "working tree clean" or list your changes

# If you have changes, commit them first
git add .
git commit -m "Save current state before upstream update"
```

**Step 3: Verify Custom Features Are Working**
```bash
# Validate current integration
./scripts/integrate-custom-features.sh validate
# Expected: [SUCCESS] Integration validation passed

# If validation fails, fix issues before proceeding
./scripts/integrate-custom-features.sh help
```

**Step 4: Check Current Git Branch**
```bash
# Make sure you're on your main branch
git branch
# Expected: * main or * master (with * indicating current branch)

# If not on main branch, switch to it
git checkout main
```

### 🔄 Upstream Update Process (10-15 minutes)

**Step 5: Fetch Latest Changes from Upstream**
```bash
# Fetch all changes from upstream repository
git fetch upstream
# Expected: No output (or shows what was fetched)

# Check what new commits are available
git log main..upstream/main --oneline
# Expected: List of new commits from upstream
```

**Step 6: Stash Your Local Changes (Optional but Recommended)**
```bash
# If you have any local changes not committed
git stash push -m "Temporary stash before upstream merge"
# Expected: Saved working directory and index state

# To see your stashes
git stash list
```

**Step 7: Merge Upstream Changes**
```bash
# Merge upstream changes into your main branch
git merge upstream/main
# Expected: Shows merge summary or conflicts

# If there are conflicts, you'll see:
# CONFLICT (content): Merge conflict in file-name
```

**Step 8: Handle Merge Conflicts (If Any)**
```bash
# If conflicts exist, check which files have conflicts
git status
# Look for files marked as "both modified"

# Open conflicted files and resolve them manually
# Look for these markers in files:
# <<<<<<< HEAD
# Your changes
# =======
# Upstream changes
# >>>>>>> upstream/main

# After resolving conflicts in each file:
git add path/to/resolved/file.ts

# Continue with merge when all conflicts resolved
git merge --continue
```

**Step 9: Restore Stashed Changes (If You Stashed)**
```bash
# Restore your stashed changes
git stash pop
# Expected: Shows what was restored

# If there are new conflicts, resolve them
git status
git add . (resolve conflicts first)
git commit -m "Merge upstream changes and resolve conflicts"
```

### 🔧 Integration Process (5-10 minutes)

**Step 10: Run the Integration Script**
```bash
# Execute the integration script
./scripts/integrate-custom-features.sh integrate
# Expected output:
# [2025-12-18 HH:MM:SS] Starting custom features integration...
# [2025-12-18 HH:MM:SS] Creating backup: backup-YYYYMMDD-HHMMSS
# [SUCCESS] Backup created at: /Users/kunthawatgreethong/Gitea/moreminimore-vibe/backups/backup-YYYYMMDD-HHMMSS
# [2025-12-18 HH:MM:SS] Creating missing custom files...
# [SUCCESS] Missing custom files created
# [2025-12-18 HH:MM:SS] Validating integration...
# [SUCCESS] Integration validation passed
# [SUCCESS] Custom features integration completed successfully!
```

**Step 11: Verify Integration Success**
```bash
# Check that all expected files exist
ls -la src/ipc/handlers/smart_context_handlers.ts
ls -la src/ipc/utils/smart_context_store.ts
ls -la src/hooks/useSmartContext.ts

# Expected: All three files should exist and show file info
```

### ✅ Validation and Testing (10-15 minutes)

**Step 12: Run Validation Script**
```bash
# Validate the integration
./scripts/integrate-custom-features.sh validate
# Expected:
# [2025-12-18 HH:MM:SS] Validating integration...
# [SUCCESS] Integration validation passed

# If you see warnings, note them but continue
# Warnings about missing custom modifications are usually OK
```

**Step 13: Check TypeScript Compilation**
```bash
# Check for TypeScript errors (may have some MCP-related warnings)
npm run ts
# Expected: May show some MCP-related errors but should complete
# If many new errors appear, investigate before proceeding
```

**Step 14: Test Application Startup**
```bash
# Install any new dependencies if needed
npm install

# Start the application in development mode
npm start
# Expected: Application should start without critical errors
# Let it run for 30-60 seconds to ensure stability
# Press Ctrl+C to stop when confirmed working
```

**Step 15: Test Custom Features Manually**
```bash
# Start the application again
npm start

# Test these features in the UI:
# 1. Create a new chat
# 2. Send a message
# 3. Check if smart context is working (no obvious errors)
# 4. Verify help dialog opens correctly
# 5. Check that promo messages display properly

# If everything works, integration is successful!
# If issues occur, proceed to troubleshooting section
```

### 🚨 If Something Goes Wrong

**Step 16: Identify the Problem**
```bash
# Check what went wrong
./scripts/integrate-custom-features.sh validate
# Note any errors or warnings

# Check application logs if it won't start
npm start 2>&1 | tee startup-errors.log
# Review startup-errors.log for clues
```

**Step 17: Restore from Backup (If Needed)**
```bash
# List available backups
ls -la backups/
# Expected: List of backup directories with timestamps

# Find your most recent backup (look for the latest timestamp)
# Example: backup-20231218-154512

# Restore from the most recent working backup
./scripts/integrate-custom-features.sh restore backup-20231218-154512
# Expected:
# [2025-12-18 HH:MM:SS] Restoring from backup: backup-20231218-154512
# [SUCCESS] Restore completed
```

**Step 18: Verify Restoration**
```bash
# Check that custom features are back
./scripts/integrate-custom-features.sh validate
# Expected: [SUCCESS] Integration validation passed

# Test application startup
npm start
# Should work as it did before the update
```

**Step 19: Manual Investigation (If Restoration Doesn't Work)**
```bash
# Check Git state
git status
git log --oneline -10

# Check specific files for issues
git diff HEAD~1 src/ipc/ipc_types.ts
git diff HEAD~1 src/ipc/ipc_client.ts

# Look for specific error patterns
grep -r "SmartContext" src/ --include="*.ts" --include="*.tsx"
```

### 🔍 Troubleshooting Commands

**Check Integration Script Health**
```bash
# Make script executable (if permission issues)
chmod +x scripts/integrate-custom-features.sh

# Test script help
./scripts/integrate-custom-features.sh help
# Should show usage instructions
```

**Verify File Permissions**
```bash
# Check script permissions
ls -la scripts/integrate-custom-features.sh
# Should show -rwxr-xr-x (executable)

# Fix permissions if needed
chmod 755 scripts/integrate-custom-features.sh
```

**Check Node.js and npm**
```bash
# Verify Node.js version
node --version
# Expected: v18.x.x or higher

# Verify npm version
npm --version
# Expected: 9.x.x or higher

# Clear npm cache if needed
npm cache clean --force
```

**Check Git Configuration**
```bash
# Verify upstream remote is configured
git remote -v
# Should show upstream remote pointing to dyad-sh/dyad

# If upstream is missing, add it
git remote add upstream https://github.com/dyad-sh/dyad.git
```

### 📝 Quick Reference Commands

**Emergency Restore (One-liner)**
```bash
# Restore latest backup (replace with actual backup name)
./scripts/integrate-custom-features.sh restore $(ls -t backups/ | head -1)
```

**Check All Backups**
```bash
# List all backups with details
ls -la backups/ | grep backup
```

**Force Clean Integration**
```bash
# For problematic integrations, start fresh
git clean -fd
git reset --hard HEAD
./scripts/integrate-custom-features.sh integrate
```

**Verify All Custom Files**
```bash
# Quick check that all custom files exist
files=("src/ipc/handlers/smart_context_handlers.ts" "src/ipc/utils/smart_context_store.ts" "src/hooks/useSmartContext.ts")
for file in "${files[@]}"; do
  if [[ -f "$file" ]]; then
    echo "✅ $file exists"
  else
    echo "❌ $file missing"
  fi
done
```

### ⏱️ Time Estimates

- **Pre-Update Checklist**: 5 minutes
- **Upstream Update**: 10-15 minutes (longer if many conflicts)
- **Integration Process**: 5-10 minutes
- **Validation and Testing**: 10-15 minutes
- **Troubleshooting**: 15-30 minutes (if needed)

**Total Time**: 30-60 minutes for smooth integration, up to 2 hours if problems occur.

### 🎯 Success Indicators

You'll know the integration was successful when:

1. ✅ Integration script runs without errors
2. ✅ Validation script shows "[SUCCESS] Integration validation passed"
3. ✅ Application starts with `npm start`
4. ✅ No critical errors in console
5. ✅ Custom features work in the UI
6. ✅ TypeScript compilation completes (may have warnings)

If all these are true, your custom features have been successfully integrated with the latest upstream changes!

---

## Integration Workflow Summary

### Before Upstream Update
1. **Current State**: Your custom features are integrated
2. **Backup**: Script automatically creates backup
3. **Validation**: Ensure everything is working

### After Upstream Update
1. **Update Code**: Pull latest changes from upstream
2. **Run Integration**: `./scripts/integrate-custom-features.sh integrate`
3. **Validate**: Check that integration succeeded
4. **Test**: Verify custom features work correctly

### If Something Goes Wrong
1. **Restore**: `./scripts/integrate-custom-features.sh restore <backup-name>`
2. **Investigate**: Check what conflicts occurred
3. **Manual Merge**: Resolve conflicts manually if needed
4. **Retry**: Run integration again

## Custom Features Explained

### Smart Context System

The smart context system provides intelligent management of chat context to handle longer conversations without hitting token limits.

#### Key Components

1. **SmartContextStore**: Manages context caching, summarization, and retrieval
2. **Context Snippets**: Individual pieces of context with importance scoring
3. **Rolling Summaries**: Automatically generated summaries of older messages
4. **Size Management**: Intelligent trimming based on importance and recency

#### Usage in Components

```typescript
import { useSmartContext, useUpdateSmartContext } from '@/hooks/useSmartContext';

// Get smart context for a chat
const { data: context, isLoading } = useSmartContext({
  chatId: 123,
  maxTokens: 8000,
  includeSummaries: true
});

// Update context with new content
const updateContext = useUpdateSmartContext();
updateContext.mutate({
  chatId: 123,
  content: "New message content",
  type: "message",
  importance: 1.0
});
```

### Rate Limit Simulation

The system includes configurable rate limiting for testing purposes:

```typescript
// In chat_stream_handlers.ts
const rateLimitConfig = {
  enabled: true,
  requestsPerMinute: 60,
  burstLimit: 10
};
```

### Enhanced Chat Streaming

Improved message handling with:
- Custom payload processing
- Better error handling
- Enhanced metadata support
- Streaming optimizations

## Configuration

### Smart Context Settings

You can configure the smart context behavior by modifying the `SmartContextStore`:

```typescript
// In src/ipc/utils/smart_context_store.ts
private maxContextSize = 100000; // 100k characters
private maxSnippets = 50;
private summaryThreshold = 20000; // Summarize when context exceeds this
```

### Rate Limit Settings

Configure rate limiting in the chat stream handlers:

```typescript
// Adjust these values based on your needs
const RATE_LIMIT_CONFIG = {
  requestsPerMinute: 60,
  burstLimit: 10,
  enabled: process.env.NODE_ENV === 'development'
};
```

## Troubleshooting

### Common Issues

1. **TypeScript Errors**: The script skips TypeScript compilation due to existing MCP issues. Focus on functionality first.
2. **Missing Custom Modifications**: The script warns if files don't contain expected custom patterns.
3. **Backup Restoration**: Always restore from the most recent working backup.

### Validation Warnings

If you see warnings about missing custom modifications:

1. Check if the file actually needs custom changes
2. Verify the custom patterns are being detected correctly
3. Manually add custom modifications if needed

### Manual Intervention

Sometimes you may need to manually merge changes:

1. **Conflicts**: Resolve Git conflicts in key files
2. **New Features**: Add custom features to new upstream files
3. **API Changes**: Update custom code to match new APIs

## Best Practices

1. **Regular Backups**: Always create backups before major changes
2. **Test Thoroughly**: Verify all custom features work after integration
3. **Document Changes**: Keep notes about manual modifications
4. **Version Control**: Commit working states regularly
5. **Gradual Updates**: Update upstream changes incrementally

## Development

### Adding New Custom Features

1. **Update Script**: Add new files to the `CUSTOM_FILES` or `NEW_CUSTOM_FILES` arrays
2. **Create Handlers**: Add IPC handlers for new functionality
3. **Update Types**: Extend type definitions as needed
4. **Test Integration**: Ensure new features integrate properly

### Modifying Existing Features

1. **Update Patterns**: Add new detection patterns to `has_custom_modifications()`
2. **Test Validation**: Ensure the script detects your changes
3. **Document Updates**: Update this documentation

## Support

If you encounter issues:

1. **Check Logs**: Look for error messages in the script output
2. **Restore Backup**: Use a recent backup to restore working state
3. **Manual Merge**: Resolve conflicts manually if needed
4. **Test Incrementally**: Test changes one at a time

## Future Enhancements

Planned improvements to the integration script:

- **Automatic Conflict Resolution**: Smarter merging of conflicting changes
- **Enhanced Validation**: More comprehensive testing
- **GUI Interface**: Visual tool for managing integrations
- **Cloud Backups**: Optional cloud backup storage
- **Rollback System**: More granular rollback capabilities

---

**Note**: This integration script is designed to work with the specific remove-limit features. If you have additional custom modifications, you may need to extend the script accordingly.