Release Candidate: Production Release with Multi-Tenant & Onboarding Enhancements
This commit is contained in:
ajaysi
314
docs/Billing_Subscription/stripe-dev-guide.md
Normal file
314
docs/Billing_Subscription/stripe-dev-guide.md
Normal file
@@ -0,0 +1,314 @@
|
||||
# Stripe Billing & Subscriptions – Developer Guide
|
||||
|
||||
This document explains how Stripe is integrated into ALwrity for subscriptions, billing, disputes, and fraud handling. It is aimed at developers working on the backend and frontend.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
- **Backend**
|
||||
- Core service: `StripeService`
|
||||
- File: `backend/services/subscription/stripe_service.py`
|
||||
- Subscription/payment API routes:
|
||||
- `backend/api/subscription/routes/payment.py`
|
||||
- `backend/api/subscription/routes/disputes.py`
|
||||
- `backend/api/subscription/routes/fraud_warnings.py`
|
||||
- Models:
|
||||
- `UserSubscription`, `SubscriptionPlan`, `BillingCycle`, `UsageStatus`, `FraudWarning`
|
||||
- File: `backend/models/subscription_models.py`
|
||||
- **Frontend**
|
||||
- Pricing and checkout UI:
|
||||
- `frontend/src/components/Pricing/PricingPage.tsx`
|
||||
- Internal admin dashboards:
|
||||
- `frontend/src/pages/StripeDisputesDashboard.tsx`
|
||||
- Routing:
|
||||
- `frontend/src/App.tsx` (route at `/stripe-disputes`)
|
||||
|
||||
Data flows:
|
||||
|
||||
- Public users:
|
||||
- Browse pricing → select plan → start Stripe Checkout → complete subscription.
|
||||
- Admin/internal users:
|
||||
- Use `/stripe-disputes` dashboard to manage disputes and early fraud warnings.
|
||||
|
||||
---
|
||||
|
||||
## 2. Configuration & Environment
|
||||
|
||||
Required environment variables (backend):
|
||||
|
||||
- `STRIPE_SECRET_KEY`
|
||||
- Stripe API key (test or live).
|
||||
- `STRIPE_WEBHOOK_SECRET`
|
||||
- Webhook signing secret for subscription webhooks.
|
||||
- `ADMIN_EMAILS` (optional)
|
||||
- Comma-separated list of admin emails allowed to access dispute/fraud endpoints.
|
||||
- `ADMIN_EMAIL_DOMAIN` (optional)
|
||||
- Domain considered admin (e.g. `example.com`).
|
||||
- `DISABLE_AUTH` (optional)
|
||||
- If `"true"`, bypasses admin checks for local/testing use only.
|
||||
|
||||
Stripe configuration:
|
||||
|
||||
- Price IDs are mapped in code (see below) and must exist in the configured Stripe account.
|
||||
- Webhook endpoint must be configured in Stripe Dashboard:
|
||||
- Path: `/api/subscription/webhook`
|
||||
- Events: `checkout.session.completed`, `invoice.payment_succeeded`, `invoice.payment_failed`, `customer.subscription.updated`, `customer.subscription.deleted`, `radar.early_fraud_warning.created` (and optionally `radar.early_fraud_warning.updated`).
|
||||
|
||||
---
|
||||
|
||||
## 3. Plans, Prices and Mapping
|
||||
|
||||
Stripe price mapping lives in `StripeService`:
|
||||
|
||||
- File: `backend/services/subscription/stripe_service.py`
|
||||
|
||||
Key structures:
|
||||
|
||||
- `STRIPE_PLAN_PRICE_MAPPING`
|
||||
- Maps `(SubscriptionTier, BillingCycle)` → Stripe `price_id`.
|
||||
- `STRIPE_PRICE_TO_PLAN`
|
||||
- Reverse map: `price_id` → `{ tier, billing_cycle }`.
|
||||
|
||||
Helper methods:
|
||||
|
||||
- `_get_price_id_for_plan(tier, billing_cycle) -> str`
|
||||
- Used when creating Checkout sessions.
|
||||
- `_get_plan_for_price_id(price_id) -> (SubscriptionPlan, BillingCycle)`
|
||||
- Used when mapping Stripe subscription items back into our internal `SubscriptionPlan`.
|
||||
|
||||
### Adding or updating plans
|
||||
|
||||
1. Create prices in Stripe (with correct recurring configuration).
|
||||
2. Update `STRIPE_PLAN_PRICE_MAPPING` with new price IDs.
|
||||
3. Ensure a `SubscriptionPlan` row exists in the DB for the tier being mapped.
|
||||
4. Redeploy backend with updated mapping.
|
||||
|
||||
---
|
||||
|
||||
## 4. Checkout and Subscription Lifecycle
|
||||
|
||||
### 4.1 Create Checkout Session
|
||||
|
||||
Endpoint:
|
||||
|
||||
- `POST /api/subscription/create-checkout-session`
|
||||
- File: `backend/api/subscription/routes/payment.py`
|
||||
|
||||
Request body:
|
||||
|
||||
- `tier: SubscriptionTier` (e.g. `"basic"`, `"pro"`)
|
||||
- `billing_cycle: BillingCycle` (e.g. `"monthly"`)
|
||||
- `success_url: str`
|
||||
- `cancel_url: str`
|
||||
|
||||
Flow:
|
||||
|
||||
1. Auth middleware resolves `current_user` and `user_id`.
|
||||
2. `StripeService.create_checkout_session`:
|
||||
- Fetches `price_id` via `_get_price_id_for_plan`.
|
||||
- Finds or creates Stripe Customer (with `user_id` in metadata).
|
||||
- Creates a Stripe Checkout Session:
|
||||
- Mode: `subscription`.
|
||||
- Metadata: includes `user_id` and `price_id`.
|
||||
3. Returns `checkout_session.url` to the frontend.
|
||||
|
||||
Special handling:
|
||||
|
||||
- Metered prices:
|
||||
- For metered prices, `quantity` is omitted to comply with Stripe rules.
|
||||
- For non-metered prices, `quantity` is set to `1`.
|
||||
|
||||
### 4.2 Customer Portal Session
|
||||
|
||||
Endpoint:
|
||||
|
||||
- `POST /api/subscription/create-portal-session`
|
||||
|
||||
Flow:
|
||||
|
||||
1. Lookup `UserSubscription` and `stripe_customer_id`.
|
||||
2. If missing, search Stripe by `metadata['user_id']`.
|
||||
3. Create Stripe Billing Portal session and return URL.
|
||||
|
||||
### 4.3 Webhook Handling
|
||||
|
||||
Endpoint:
|
||||
|
||||
- `POST /api/subscription/webhook`
|
||||
- File: `backend/api/subscription/routes/payment.py`
|
||||
- Delegates to `StripeService.handle_webhook`.
|
||||
|
||||
Verification:
|
||||
|
||||
- `stripe.Webhook.construct_event(payload, sig_header, STRIPE_WEBHOOK_SECRET)` is used to validate signatures.
|
||||
|
||||
Handled events:
|
||||
|
||||
- `checkout.session.completed`
|
||||
- Retrieves subscription and price.
|
||||
- Updates `UserSubscription` to active and stores `stripe_customer_id` and `stripe_subscription_id`.
|
||||
- `invoice.payment_succeeded`
|
||||
- Sets `UserSubscription.status` to `ACTIVE`.
|
||||
- Updates `current_period_end` from invoice period.
|
||||
- `invoice.payment_failed`
|
||||
- Sets status to `PAST_DUE`, `is_active` false.
|
||||
- `customer.subscription.updated`
|
||||
- Syncs status and `auto_renew`.
|
||||
- `customer.subscription.deleted`
|
||||
- Marks subscription as cancelled and disables auto renew.
|
||||
|
||||
Helper:
|
||||
|
||||
- `_update_user_subscription` centralizes updating/creating `UserSubscription` records based on Stripe data.
|
||||
|
||||
---
|
||||
|
||||
## 5. Disputes Integration
|
||||
|
||||
Backend routes:
|
||||
|
||||
- File: `backend/api/subscription/routes/disputes.py`
|
||||
|
||||
Endpoints:
|
||||
|
||||
- `GET /api/subscription/disputes`
|
||||
- Proxies `stripe.Dispute.list`.
|
||||
- `GET /api/subscription/disputes/{dispute_id}`
|
||||
- Proxies `stripe.Dispute.retrieve`.
|
||||
- `POST /api/subscription/disputes/{dispute_id}`
|
||||
- Proxies `stripe.Dispute.modify` with `evidence`.
|
||||
- `POST /api/subscription/disputes/{dispute_id}/close`
|
||||
- Proxies `stripe.Dispute.close`.
|
||||
|
||||
Admin guard:
|
||||
|
||||
- `_ensure_admin(current_user)` ensures:
|
||||
- Admin by email, domain, or role `"admin"`.
|
||||
- Can be bypassed only when `DISABLE_AUTH=true` (local use).
|
||||
|
||||
Frontend UI:
|
||||
|
||||
- File: `frontend/src/pages/StripeDisputesDashboard.tsx`
|
||||
- Route: `/stripe-disputes`
|
||||
- Disputes tab:
|
||||
- Lists disputes and allows:
|
||||
- Viewing details.
|
||||
- Submitting evidence fields:
|
||||
- `customer_email_address`, `customer_name`, `customer_purchase_ip`, `access_activity_log`, `uncategorized_text`.
|
||||
- Tagging a high-level fraud type, which is encoded into `uncategorized_text`.
|
||||
- Closing the dispute.
|
||||
|
||||
---
|
||||
|
||||
## 6. Early Fraud Warnings (EFW) and Proactive Refunds
|
||||
|
||||
### 6.1 Ingestion
|
||||
|
||||
Model:
|
||||
|
||||
- `FraudWarning` in `backend/models/subscription_models.py`
|
||||
- Columns: `id`, `charge_id`, `payment_intent_id`, `user_id`, `amount`, `currency`, `status`, `action`, `action_at`, `reason_notes`, `metadata`, `created_at`.
|
||||
|
||||
Ingestion logic:
|
||||
|
||||
- `StripeService._handle_early_fraud_warning`:
|
||||
- Triggered for event types starting with `radar.early_fraud_warning.`.
|
||||
- Retrieves the associated `Charge` to populate amount, currency, and metadata.
|
||||
- Infers `user_id` from `charge.metadata.user_id` when available.
|
||||
- Upserts a `FraudWarning` row with status `"open"` and action `"none"`.
|
||||
- Stores raw EFW and Charge data in `metadata`.
|
||||
|
||||
### 6.2 Fraud Warnings API
|
||||
|
||||
File: `backend/api/subscription/routes/fraud_warnings.py`
|
||||
|
||||
Endpoints:
|
||||
|
||||
- `GET /api/subscription/fraud-warnings`
|
||||
- Query params:
|
||||
- `status` (default `"open"`)
|
||||
- `limit`, `offset`
|
||||
- Returns a list of warnings with core fields.
|
||||
- `GET /api/subscription/fraud-warnings/{id}`
|
||||
- Returns full details including `metadata`.
|
||||
- `POST /api/subscription/fraud-warnings/{id}/refund`
|
||||
- Performs a **full refund** via `stripe.Refund.create(charge=...)`.
|
||||
- Updates `status="refunded"`, `action="refund_full"`, `action_at` and `reason_notes`.
|
||||
- `POST /api/subscription/fraud-warnings/{id}/ignore`
|
||||
- Sets `status="ignored"`, `action="ignored"`, updates notes.
|
||||
|
||||
All endpoints apply the same admin guard used for disputes.
|
||||
|
||||
### 6.3 Frontend Fraud Warnings Tab
|
||||
|
||||
- File: `frontend/src/pages/StripeDisputesDashboard.tsx`
|
||||
|
||||
Behavior:
|
||||
|
||||
- Adds a tabbed view:
|
||||
- Tab 1: Disputes.
|
||||
- Tab 2: Fraud Warnings.
|
||||
- Fraud Warnings tab:
|
||||
- Lists EFWs (from `/fraud-warnings`).
|
||||
- Shows details including:
|
||||
- Stripe EFW `fraud_type`, `actionable` flag.
|
||||
- Amount, created time, internal status/action.
|
||||
- Allows:
|
||||
- Proactive full refund (calls `/fraud-warnings/{id}/refund`).
|
||||
- Mark as ignored (calls `/fraud-warnings/{id}/ignore`).
|
||||
- Add/update internal notes.
|
||||
|
||||
---
|
||||
|
||||
## 7. Rate Limiting for Checkout
|
||||
|
||||
Endpoint: `POST /api/subscription/create-checkout-session`
|
||||
|
||||
File: `backend/api/subscription/routes/payment.py`
|
||||
|
||||
Logic:
|
||||
|
||||
- Per-user in-memory rate limiting:
|
||||
- Window: 60 seconds.
|
||||
- Max requests: 10 within the window.
|
||||
- On exceed:
|
||||
- Logs a warning with `user_id`, IP, attempts count.
|
||||
- Returns HTTP 429 with a friendly error message.
|
||||
|
||||
Purpose:
|
||||
|
||||
- Protects against card testing and abuse by limiting how often a user can create Checkout sessions.
|
||||
|
||||
Considerations:
|
||||
|
||||
- For multi-instance deployments, a shared store (e.g. Redis) is recommended to make rate limiting consistent across instances.
|
||||
|
||||
---
|
||||
|
||||
## 8. Extending and Maintaining the Integration
|
||||
|
||||
### Adding new subscription tiers or prices
|
||||
|
||||
1. Create or update prices in Stripe.
|
||||
2. Update `STRIPE_PLAN_PRICE_MAPPING` in `StripeService`.
|
||||
3. Ensure corresponding rows in `SubscriptionPlan`.
|
||||
4. Add any needed frontend logic (e.g. additional tiers in pricing UI).
|
||||
|
||||
### Supporting additional Stripe events
|
||||
|
||||
- Extend `StripeService.handle_webhook` with new event types.
|
||||
- Implement corresponding handlers (`_handle_*`) that:
|
||||
- Parse event data.
|
||||
- Update your DB models.
|
||||
- Log with enough context.
|
||||
|
||||
### Making the system more robust
|
||||
|
||||
- Reintroduce idempotency keys for write operations (Checkout creation, refunds) using stable dedupe keys.
|
||||
- Replace in-memory rate limiting with shared store-based limiting when scaling horizontally.
|
||||
- Add more detailed logs/metrics around:
|
||||
- New subscriptions.
|
||||
- Failed payments.
|
||||
- Disputes and early fraud warnings.
|
||||
|
||||
202
docs/Billing_Subscription/stripe-go-live-checklist.md
Normal file
202
docs/Billing_Subscription/stripe-go-live-checklist.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# Stripe Go-Live Checklist
|
||||
|
||||
This checklist is for preparing ALwrity’s Stripe integration for production. Use it before switching to live keys or onboarding real customers.
|
||||
|
||||
Tick each item as you complete it.
|
||||
|
||||
---
|
||||
|
||||
## 1. Configuration & Environment
|
||||
|
||||
- [ ] **Separate environments set up**
|
||||
- [ ] Test mode Stripe account configured.
|
||||
- [ ] Live mode Stripe account configured.
|
||||
- [ ] **Environment variables configured for production**
|
||||
- [ ] `STRIPE_SECRET_KEY` set to **live** secret key.
|
||||
- [ ] `STRIPE_WEBHOOK_SECRET` set to **live** webhook signing secret.
|
||||
- [ ] `ADMIN_EMAILS` configured with correct admin emails (comma-separated).
|
||||
- [ ] `ADMIN_EMAIL_DOMAIN` configured if using domain-based admin access.
|
||||
- [ ] `DISABLE_AUTH` is **not** set to `"true"` in production.
|
||||
- [ ] **Secrets handling**
|
||||
- [ ] No Stripe keys are committed to the repo.
|
||||
- [ ] Secrets are stored only in your deployment platform / secret manager.
|
||||
|
||||
---
|
||||
|
||||
## 2. Prices, Plans and Mapping
|
||||
|
||||
- [ ] **All required prices exist in Stripe (live)**
|
||||
- [ ] BASIC monthly price created.
|
||||
- [ ] PRO monthly price created (if used).
|
||||
- [ ] Yearly prices created if you plan to sell yearly plans.
|
||||
- [ ] **Price mapping in backend updated**
|
||||
- [ ] `STRIPE_PLAN_PRICE_MAPPING` uses **live** price IDs (not test IDs).
|
||||
- [ ] Mapping covers all tiers and billing cycles you intend to offer.
|
||||
- [ ] **SubscriptionPlan data is consistent**
|
||||
- [ ] DB has `SubscriptionPlan` rows for each tier (BASIC/PRO/etc.).
|
||||
- [ ] `is_active` is set to true for sellable plans.
|
||||
|
||||
---
|
||||
|
||||
## 3. Database & Migrations
|
||||
|
||||
- [ ] **Model changes applied in production DB**
|
||||
- [ ] Tables related to subscriptions exist:
|
||||
- [ ] `subscription_plans`
|
||||
- [ ] `user_subscriptions`
|
||||
- [ ] Usage/billing tables exist if used (`api_usage_logs`, `usage_summaries`, etc.).
|
||||
- [ ] `fraud_warnings` table exists for early fraud warnings:
|
||||
- [ ] Checked via DB console or migration logs.
|
||||
- [ ] **Migration strategy verified**
|
||||
- [ ] Any migration scripts run successfully on staging.
|
||||
- [ ] Same process is planned for production.
|
||||
|
||||
---
|
||||
|
||||
## 4. Webhook Setup
|
||||
|
||||
- [ ] **Production webhook endpoint configured in Stripe Dashboard**
|
||||
- [ ] URL points to your production backend:
|
||||
- e.g. `https://your-domain.com/api/subscription/webhook`
|
||||
- [ ] Uses HTTPS.
|
||||
- [ ] **Subscribed events include at least**
|
||||
- [ ] `checkout.session.completed`
|
||||
- [ ] `invoice.payment_succeeded`
|
||||
- [ ] `invoice.payment_failed`
|
||||
- [ ] `customer.subscription.updated`
|
||||
- [ ] `customer.subscription.deleted`
|
||||
- [ ] `radar.early_fraud_warning.created`
|
||||
- [ ] (Optional) `radar.early_fraud_warning.updated`
|
||||
- [ ] **Webhook secret set correctly**
|
||||
- [ ] Copy live webhook signing secret from Stripe into `STRIPE_WEBHOOK_SECRET`.
|
||||
- [ ] Confirm no test webhook secret is used in production.
|
||||
- [ ] **Webhook endpoint health check**
|
||||
- [ ] Trigger a test event from Stripe Dashboard (in a safe environment).
|
||||
- [ ] Verify the backend logs show successful verification and handling.
|
||||
|
||||
---
|
||||
|
||||
## 5. Internal Admin Tools (Ops Readiness)
|
||||
|
||||
- [ ] **Admin roles/permissions**
|
||||
- [ ] Confirm at least one admin user can access `/stripe-disputes`.
|
||||
- [ ] Non-admin users cannot access sensitive endpoints (disputes, fraud warnings).
|
||||
- [ ] **Disputes dashboard**
|
||||
- [ ] `/stripe-disputes` loads without error.
|
||||
- [ ] Disputes tab can:
|
||||
- [ ] List disputes.
|
||||
- [ ] Show dispute details.
|
||||
- [ ] Submit evidence.
|
||||
- [ ] Close a dispute.
|
||||
- [ ] **Fraud Warnings tab**
|
||||
- [ ] Fraud Warnings tab loads without error.
|
||||
- [ ] List of early fraud warnings is visible when test EFWs exist.
|
||||
- [ ] Details dialog shows:
|
||||
- [ ] Issuer fraud type.
|
||||
- [ ] Actionable flag.
|
||||
- [ ] Internal status / actions.
|
||||
- [ ] Buttons:
|
||||
- [ ] “Refund Full Amount” works (in test/staging).
|
||||
- [ ] “Mark as Ignored” works (updates status).
|
||||
- [ ] **Ops team trained**
|
||||
- [ ] Ops have read the Ops Guide.
|
||||
- [ ] They understand:
|
||||
- [ ] How to respond to disputes.
|
||||
- [ ] When to proactively refund EFWs.
|
||||
- [ ] When to escalate to engineering.
|
||||
|
||||
---
|
||||
|
||||
## 6. Manual Test Flows (Before Real Customers)
|
||||
|
||||
Perform these in **test** environment first, then in live with small amounts.
|
||||
|
||||
### 6.1 New Subscription Flow
|
||||
|
||||
- [ ] As a test user:
|
||||
- [ ] Go to Pricing page.
|
||||
- [ ] Select BASIC monthly (or equivalent).
|
||||
- [ ] Start Stripe Checkout and complete payment with test card.
|
||||
- [ ] You are redirected back to the success URL.
|
||||
- [ ] Backend:
|
||||
- [ ] Webhook logs show `checkout.session.completed` processed.
|
||||
- [ ] `UserSubscription` updated with `stripe_customer_id` and `stripe_subscription_id`.
|
||||
- [ ] Subscription status is `ACTIVE`.
|
||||
|
||||
### 6.2 Billing Portal
|
||||
|
||||
- [ ] From the app, open the billing portal (Customer Portal).
|
||||
- [ ] `/api/subscription/create-portal-session` returns a URL.
|
||||
- [ ] You can:
|
||||
- [ ] View invoices.
|
||||
- [ ] Update card details.
|
||||
- [ ] Cancel a subscription.
|
||||
- [ ] After cancellation:
|
||||
- [ ] Webhook logs show `customer.subscription.deleted`.
|
||||
- [ ] `UserSubscription` is updated to cancelled and not active.
|
||||
|
||||
### 6.3 Failed Payment (Test Mode)
|
||||
|
||||
- [ ] Use a known failing test card.
|
||||
- [ ] Trigger a failed invoice.
|
||||
- [ ] Verify:
|
||||
- [ ] `invoice.payment_failed` processed.
|
||||
- [ ] `UserSubscription` status is set to `PAST_DUE` and `is_active` is false.
|
||||
|
||||
### 6.4 Dispute (Test Mode)
|
||||
|
||||
- [ ] Create a test dispute in Stripe’s test mode.
|
||||
- [ ] Confirm:
|
||||
- [ ] Dispute appears in the Disputes tab.
|
||||
- [ ] You can open details and submit evidence.
|
||||
|
||||
### 6.5 Early Fraud Warning (Test Mode)
|
||||
|
||||
- [ ] Create a test Early Fraud Warning (if supported in test mode or via Stripe tools).
|
||||
- [ ] Confirm:
|
||||
- [ ] EFW is ingested and appears in Fraud Warnings tab.
|
||||
- [ ] Details dialog shows issuer `fraud_type` and `actionable` flag.
|
||||
- [ ] “Refund Full Amount” works in test (Stripe shows charge refunded).
|
||||
|
||||
---
|
||||
|
||||
## 7. Rate Limiting and Abuse Protection
|
||||
|
||||
- [ ] **Checkout endpoint rate limiting**
|
||||
- [ ] Confirm `create-checkout-session` applies per-user rate limits.
|
||||
- [ ] Hitting the endpoint rapidly produces HTTP 429 and a log entry.
|
||||
- [ ] **Monitoring for card testing**
|
||||
- [ ] Logs for rate-limited events are visible in your logging system.
|
||||
- [ ] You have a plan to investigate suspicious spikes (many 429s or many failed payments).
|
||||
|
||||
---
|
||||
|
||||
## 8. Monitoring & Alerts
|
||||
|
||||
- [ ] **Logging**
|
||||
- [ ] Backend logs are centralized (e.g. in a logging service).
|
||||
- [ ] Key Stripe flows (webhooks, disputes, fraud warnings) log useful context.
|
||||
- [ ] **Basic alerting**
|
||||
- [ ] At minimum, you can detect:
|
||||
- [ ] Webhook failures.
|
||||
- [ ] Unusually high dispute volume.
|
||||
- [ ] Frequent early fraud warnings.
|
||||
|
||||
---
|
||||
|
||||
## 9. Final Production Switch
|
||||
|
||||
- [ ] **Keys double-checked**
|
||||
- [ ] Production environment uses live Stripe keys and webhook secret.
|
||||
- [ ] No references to test keys remain in production configs.
|
||||
- [ ] **Test charge in live mode**
|
||||
- [ ] Complete a small real transaction in live mode.
|
||||
- [ ] Verify:
|
||||
- [ ] Subscription is active.
|
||||
- [ ] Internal dashboard reflects the subscription correctly.
|
||||
- [ ] Refund/portal flows work as expected.
|
||||
- [ ] **Ops sign-off**
|
||||
- [ ] Ops team confirms they can use Disputes and Fraud Warnings tools comfortably.
|
||||
|
||||
Once all items are checked, you can consider the Stripe integration ready for production traffic.
|
||||
|
||||
242
docs/Billing_Subscription/stripe-ops-guide.md
Normal file
242
docs/Billing_Subscription/stripe-ops-guide.md
Normal file
@@ -0,0 +1,242 @@
|
||||
# Stripe Billing & Subscriptions – Ops Team Guide
|
||||
|
||||
This guide is for non-technical operations and support staff. It explains how to use ALwrity’s internal Stripe tools to review payments, handle disputes, and respond to early fraud warnings.
|
||||
|
||||
You do **not** need to use the Stripe Dashboard for day-to-day work; use the internal tools described here.
|
||||
|
||||
---
|
||||
|
||||
## 1. Where to go in the app
|
||||
|
||||
- Sign in to ALwrity with your admin account.
|
||||
- Open the internal Stripe dashboard:
|
||||
- URL: `/stripe-disputes`
|
||||
- You will see two tabs:
|
||||
- **Disputes** – for chargebacks / disputes raised by card issuers.
|
||||
- **Fraud Warnings** – for early fraud warnings (EFWs) where issuers suspect fraud before a dispute is filed.
|
||||
|
||||
If you cannot access this page:
|
||||
|
||||
- Your account might not be whitelisted as an admin. Contact the engineering team to check your email and role.
|
||||
|
||||
---
|
||||
|
||||
## 2. Disputes Tab – Handling Chargebacks
|
||||
|
||||
When a customer disputes a payment with their bank, Stripe creates a **Dispute**. The Disputes tab helps you:
|
||||
|
||||
- See all disputes.
|
||||
- Review details (amount, reason, charge ID).
|
||||
- Submit evidence.
|
||||
- Close disputes when needed.
|
||||
|
||||
### 2.1 Disputes List
|
||||
|
||||
The table shows:
|
||||
|
||||
- **ID** – Stripe’s dispute ID (useful if support needs to talk to Stripe).
|
||||
- **Amount** – Disputed amount.
|
||||
- **Status** – Current status (e.g. `needs_response`, `under_review`, `won`, `lost`).
|
||||
- **Reason** – Bank’s reason (e.g. `fraudulent`, `product_not_received`).
|
||||
- **Charge** – The related Stripe charge ID.
|
||||
- **Created** – When the dispute was created.
|
||||
|
||||
Actions:
|
||||
|
||||
- **Refresh disputes** – Reloads the list from Stripe.
|
||||
- **Details** – Opens the dispute details dialog.
|
||||
- **Close** – Shortcut to close the dispute (same as “Close Dispute” inside the dialog).
|
||||
|
||||
### 2.2 Dispute Details & Evidence
|
||||
|
||||
When you click **Details**, you see:
|
||||
|
||||
- **ID / Amount / Status / Reason / Charge / Created** – Basic information summarizing the case.
|
||||
- **Fraud Type** – A dropdown where you classify the dispute:
|
||||
- `Card testing` – many small rapid attempts, usually bots testing cards.
|
||||
- `Stolen card` – customer’s card was used without permission.
|
||||
- `Overpayment fraud` – customer overpays and asks for a refund via another method.
|
||||
- `Alternative refund` – customer tries to get a payout via cash/crypto/bank transfer instead of back to card.
|
||||
- `Other` – anything else.
|
||||
- **Customer Email / Name / IP** – Fields to record known customer details.
|
||||
- **Access Activity Log** – Summary of account activity:
|
||||
- Example:
|
||||
- `"User logged in from IP 1.2.3.4, created 3 projects, downloaded 2 reports."`
|
||||
- **Fraud Indicators / Notes** – A free text area where you:
|
||||
- Summarize what looks suspicious (or legitimate).
|
||||
- Mention patterns like:
|
||||
- Many failed attempts before one success.
|
||||
- Overpayment + request for alternate refund.
|
||||
- Different billing and login locations.
|
||||
|
||||
Buttons:
|
||||
|
||||
- **Submit Evidence**
|
||||
- Sends your evidence to Stripe for this dispute.
|
||||
- Use this when you want to **contest** the dispute and show that the charge is valid.
|
||||
- **Close Dispute**
|
||||
- Tells Stripe you are not going to submit more evidence.
|
||||
- Use this if:
|
||||
- The dispute is clearly correct (e.g. genuine mistake).
|
||||
- The amount is lower than the dispute fee and not worth contesting.
|
||||
|
||||
Tips:
|
||||
|
||||
- Be specific and factual in evidence:
|
||||
- “User logged in and used the product for 3 days” is better than “Looks fine”.
|
||||
- Use the **Fraud Type** dropdown to tag cases consistently; it helps the team see patterns.
|
||||
|
||||
---
|
||||
|
||||
## 3. Fraud Warnings Tab – Early Fraud Warning (EFW)
|
||||
|
||||
An **Early Fraud Warning** is a signal from the card issuer that a charge may be fraudulent, before a dispute is created.
|
||||
|
||||
The Fraud Warnings tab helps you:
|
||||
|
||||
- See EFWs for our charges.
|
||||
- Decide whether to proactively refund to avoid a later dispute.
|
||||
- Record decisions and notes.
|
||||
|
||||
### 3.1 Fraud Warnings List
|
||||
|
||||
Columns:
|
||||
|
||||
- **ID** – The Early Fraud Warning ID from Stripe.
|
||||
- **Charge** – Related Stripe charge ID.
|
||||
- **Amount** – Charge amount.
|
||||
- **Status** – Our internal status:
|
||||
- `open` – Needs review.
|
||||
- `refunded` – We proactively refunded the card.
|
||||
- `ignored` – We reviewed and decided not to refund.
|
||||
- **Action** – The latest action taken (`none`, `refund_full`, `ignored`).
|
||||
- **Created** – When the warning was created.
|
||||
|
||||
Actions:
|
||||
|
||||
- **Refresh warnings** – Reloads current open warnings.
|
||||
- **Details** – Opens the warning details dialog.
|
||||
|
||||
### 3.2 Fraud Warning Details and Actions
|
||||
|
||||
Inside the details dialog you see:
|
||||
|
||||
- **ID / Charge / Amount** – Basic reference info.
|
||||
- **Status / Action** – Current state and last action taken.
|
||||
- **Created / Last Action At** – Timeline.
|
||||
- **Issuer Fraud Type** – What the bank believes is happening (e.g. `made_with_stolen_card`).
|
||||
- **Actionable** – Indicates whether Stripe considers this warning still actionable:
|
||||
- “Yes” – No full refund yet and no dispute; you can still act.
|
||||
- “No” – It has either been refunded or disputed already.
|
||||
- **Action Notes** – Free text for internal reasoning.
|
||||
|
||||
Buttons:
|
||||
|
||||
- **Refund Full Amount**
|
||||
- Sends a full refund for the underlying charge via Stripe.
|
||||
- Sets status to `refunded` and action to `refund_full`.
|
||||
- Use this when:
|
||||
- The charge amount is relatively small (similar to or less than your dispute fee).
|
||||
- The warning and behavior strongly suggest fraud (e.g. stolen card, clear card testing).
|
||||
- **Mark as Ignored**
|
||||
- Marks the warning as `ignored` without refund.
|
||||
- Use this when:
|
||||
- You believe the charge is legitimate.
|
||||
- The user has confirmed the purchase, or your internal logs show normal behavior.
|
||||
- **Close**
|
||||
- Closes the dialog only (no changes to Stripe or status).
|
||||
|
||||
Notes:
|
||||
|
||||
- You can add or update **Action Notes** before clicking Refund or Mark as Ignored:
|
||||
- Example:
|
||||
- `"Customer confirmed via support email that they made this purchase."`
|
||||
- `"High risk: many failed attempts, unusual IP, amount small – refunding to avoid dispute."`
|
||||
|
||||
---
|
||||
|
||||
## 4. How to Decide: Refund vs Ignore
|
||||
|
||||
These are general guidelines; when in doubt, coordinate with product/engineering.
|
||||
|
||||
### 4.1 When to Consider Proactive Refund
|
||||
|
||||
- The amount is **small**, roughly in the range of the expected dispute fee.
|
||||
- The pattern clearly matches fraud:
|
||||
- Many rapid attempts with different cards or card numbers.
|
||||
- Charge is from a suspicious IP/country inconsistent with user profile.
|
||||
- Issuer fraud type suggests stolen or counterfeit card.
|
||||
- The user is not reachable or does not respond to your messages.
|
||||
|
||||
In these cases:
|
||||
|
||||
- Use **Fraud Warnings → Details → Refund Full Amount**.
|
||||
- Add a short note explaining why:
|
||||
- `"EFW flagged as made_with_stolen_card; small charge; refunding proactively."`
|
||||
|
||||
### 4.2 When to Ignore (No Proactive Refund)
|
||||
|
||||
- The customer confirms they made the purchase.
|
||||
- Your logs show normal use of the product:
|
||||
- Regular logins, content creation, downloads.
|
||||
- Amount is large and there is no strong sign of fraud:
|
||||
- In this case you typically wait and, if a dispute occurs, respond with strong evidence.
|
||||
|
||||
In these cases:
|
||||
|
||||
- Use **Fraud Warnings → Details → Mark as Ignored**.
|
||||
- Add notes:
|
||||
- `"Customer confirmed via email; usage patterns normal; ignoring EFW."`
|
||||
|
||||
---
|
||||
|
||||
## 5. Things You Should Not Do
|
||||
|
||||
- Do **not** send refunds via:
|
||||
- Bank transfer
|
||||
- Cash
|
||||
- Crypto
|
||||
- Any method outside Stripe
|
||||
|
||||
Always refund via Stripe so:
|
||||
|
||||
- The cardholder is repaid correctly.
|
||||
- Issuers see the refund related to the original charge.
|
||||
|
||||
If someone asks for a different refund method, treat it as a potential **overpayment** or **alternative refund** scam and escalate to the team.
|
||||
|
||||
---
|
||||
|
||||
## 6. When to Escalate to Engineering
|
||||
|
||||
Contact engineering when:
|
||||
|
||||
- You see a sudden **spike in disputes** or fraud warnings.
|
||||
- The internal dashboard shows errors when:
|
||||
- Loading disputes/fraud warnings.
|
||||
- Submitting evidence.
|
||||
- Refunding/ignoring warnings.
|
||||
- You need a new flow:
|
||||
- Example: new product or plan changes that alter how subscriptions work.
|
||||
|
||||
Provide:
|
||||
|
||||
- Screenshot of the issue.
|
||||
- Dispute ID or Fraud Warning ID.
|
||||
- A short description of what you were trying to do.
|
||||
|
||||
---
|
||||
|
||||
## 7. Quick Reference
|
||||
|
||||
- **Disputes Tab**
|
||||
- Use to respond to formal disputes.
|
||||
- Add evidence and close disputes when appropriate.
|
||||
- **Fraud Warnings Tab**
|
||||
- Use to review early fraud warnings.
|
||||
- Decide whether to refund or ignore.
|
||||
- **Action Notes**
|
||||
- Always record a short reason when you refund or ignore.
|
||||
|
||||
If you follow this guide, you will help protect the business from fraud while treating legitimate customers fairly.
|
||||
|
||||
605
docs/linkedin_factual_google_grounded_url_content.md
Normal file
605
docs/linkedin_factual_google_grounded_url_content.md
Normal file
@@ -0,0 +1,605 @@
|
||||
# LinkedIn Factual Google Grounded URL Content Enhancement Plan
|
||||
|
||||
## 📋 **Executive Summary**
|
||||
|
||||
This document outlines ALwrity's comprehensive plan to enhance LinkedIn content quality from basic AI generation to enterprise-grade, factually grounded content using Google AI's advanced capabilities. The implementation will integrate Google Search grounding and URL context tools to provide LinkedIn professionals with credible, current, and industry-relevant content.
|
||||
|
||||
**🟢 IMPLEMENTATION STATUS: Phase 1 Native Grounding Completed**
|
||||
|
||||
## 🎯 **Problem Statement**
|
||||
|
||||
### **Current State Issues**
|
||||
- **Generic AI Content**: Produces bland, non-specific content lacking industry relevance
|
||||
- **No Source Verification**: Content claims lack factual backing or citations
|
||||
- **Outdated Information**: AI knowledge cutoff limits current industry insights
|
||||
- **Low Professional Credibility**: Content doesn't meet enterprise LinkedIn standards
|
||||
- **No Industry Context**: Fails to leverage current trends, reports, or expert insights
|
||||
- **Mock Research System**: Current `_conduct_research` method returns simulated data
|
||||
- **Limited Grounding**: Content not factually verified or source-attributed
|
||||
|
||||
### **Business Impact**
|
||||
- **User Dissatisfaction**: Professional users expect higher quality content
|
||||
- **Competitive Disadvantage**: Other tools may offer better content quality
|
||||
- **Trust Issues**: Unverified content damages brand credibility
|
||||
- **Limited Adoption**: Enterprise users won't adopt low-quality content tools
|
||||
|
||||
## 🚀 **Solution Overview**
|
||||
|
||||
### **Google AI Integration Strategy**
|
||||
1. **Google Search Grounding**: Real-time web search for current industry information
|
||||
2. **URL Context Integration**: Specific source grounding from authoritative URLs
|
||||
3. **Citation System**: Inline source attribution for all factual claims
|
||||
4. **Quality Assurance**: Automated fact-checking and source validation
|
||||
5. **Enhanced Gemini Provider**: Grounded content generation with source integration
|
||||
|
||||
### **Expected Outcomes**
|
||||
- **Enterprise-Grade Content**: Professional quality suitable for LinkedIn professionals
|
||||
- **Factual Accuracy**: All claims backed by current, verifiable sources
|
||||
- **Industry Relevance**: Content grounded in latest trends and insights
|
||||
- **Trust Building**: Verifiable sources increase user confidence and adoption
|
||||
|
||||
## 🏗️ **Technical Architecture**
|
||||
|
||||
### **Core Components**
|
||||
|
||||
#### **1. Enhanced Gemini Provider Module** ✅ **IMPLEMENTED**
|
||||
- **Grounded Content Generation**: AI content generation with source integration
|
||||
- **Citation Engine**: Automatic inline citation generation and management
|
||||
- **Source Integration**: Seamless incorporation of research data into content
|
||||
- **Quality Validation**: Content quality assessment and scoring
|
||||
- **Fallback Systems**: Graceful degradation when grounding fails
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/services/llm_providers/gemini_grounded_provider.py`
|
||||
- **Class**: `GeminiGroundedProvider`
|
||||
- **Key Methods**:
|
||||
- `generate_grounded_content()` - Main content generation with sources
|
||||
- `_build_grounded_prompt()` - Source-integrated prompt building
|
||||
- `_add_citations()` - Automatic citation insertion
|
||||
- `_assess_content_quality()` - Quality scoring and validation
|
||||
|
||||
#### **2. Real Research Service** ✅ **IMPLEMENTED**
|
||||
- **Google Custom Search API**: Industry-specific search with credibility scoring
|
||||
- **Source Ranking Algorithm**: Prioritize sources by credibility, recency, and relevance
|
||||
- **Domain Authority Assessment**: Evaluate source reliability and expertise
|
||||
- **Content Extraction**: Extract relevant insights and statistics from sources
|
||||
- **Real-time Updates**: Current information from the last month
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/services/research/google_search_service.py`
|
||||
- **Class**: `GoogleSearchService`
|
||||
- **Key Methods**:
|
||||
- `search_industry_trends()` - Main search functionality
|
||||
- `_build_search_query()` - Intelligent query construction
|
||||
- `_perform_search()` - API call management with retry logic
|
||||
- `_process_search_results()` - Result processing and scoring
|
||||
- `_calculate_relevance_score()` - Relevance scoring algorithm
|
||||
- `_calculate_credibility_score()` - Source credibility assessment
|
||||
|
||||
#### **3. Citation Management System** ✅ **IMPLEMENTED**
|
||||
- **Inline Citation Formatting**: [Source 1], [Source 2] style citations
|
||||
- **Citation Validation**: Ensure all claims have proper source attribution
|
||||
- **Source List Generation**: Comprehensive list of sources with links
|
||||
- **Citation Coverage Analysis**: Track percentage of claims with citations
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/services/citation/citation_manager.py`
|
||||
- **Class**: `CitationManager`
|
||||
- **Key Methods**:
|
||||
- `add_citations()` - Insert citations into content
|
||||
- `validate_citations()` - Verify citation completeness
|
||||
- `generate_source_list()` - Create formatted source references
|
||||
- `extract_citations()` - Parse existing citations from content
|
||||
- `_identify_citation_patterns()` - Pattern recognition for citations
|
||||
|
||||
#### **4. Content Quality Analyzer** ✅ **IMPLEMENTED**
|
||||
- **Factual Accuracy Scoring**: Assess content against source verification
|
||||
- **Professional Tone Analysis**: Evaluate enterprise-appropriate language
|
||||
- **Industry Relevance Metrics**: Measure topic-specific content alignment
|
||||
- **Overall Quality Scoring**: Composite score for content assessment
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/services/quality/content_analyzer.py`
|
||||
- **Class**: `ContentQualityAnalyzer`
|
||||
- **Key Methods**:
|
||||
- `analyze_content_quality()` - Main quality assessment
|
||||
- `_assess_factual_accuracy()` - Source verification scoring
|
||||
- `_assess_professional_tone()` - Language appropriateness analysis
|
||||
- `_assess_industry_relevance()` - Topic alignment scoring
|
||||
- `_calculate_overall_score()` - Composite quality calculation
|
||||
|
||||
#### **5. Enhanced LinkedIn Service** ✅ **IMPLEMENTED**
|
||||
- **Integrated Grounding**: Seamless integration of all grounding services
|
||||
- **Content Generation**: Enhanced methods for all LinkedIn content types
|
||||
- **Research Integration**: Real research with fallback to mock data
|
||||
- **Quality Metrics**: Comprehensive content quality reporting
|
||||
- **Grounding Status**: Detailed grounding operation tracking
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/services/linkedin_service.py`
|
||||
- **Class**: `LinkedInService` (renamed from `LinkedInContentService`)
|
||||
- **Key Methods**:
|
||||
- `generate_linkedin_post()` - Enhanced post generation with grounding
|
||||
- `generate_linkedin_article()` - Research-backed article creation
|
||||
- `generate_linkedin_carousel()` - Grounded carousel generation
|
||||
- `generate_linkedin_video_script()` - Script generation with sources
|
||||
- `_conduct_research()` - Real Google search with fallback
|
||||
- `_generate_grounded_*_content()` - Grounded content generation methods
|
||||
|
||||
#### **6. Enhanced Data Models** ✅ **IMPLEMENTED**
|
||||
- **Grounding Support**: New fields for sources, citations, and quality metrics
|
||||
- **Enhanced Responses**: Comprehensive response models with grounding data
|
||||
- **Quality Metrics**: Detailed content quality assessment models
|
||||
- **Citation Models**: Structured citation and source management
|
||||
|
||||
**Implementation Details:**
|
||||
- **File**: `backend/models/linkedin_models.py`
|
||||
- **New Models**:
|
||||
- `GroundingLevel` - Enum for grounding levels (none, basic, enhanced, enterprise)
|
||||
- `ContentQualityMetrics` - Comprehensive quality scoring
|
||||
- `Citation` - Inline citation structure
|
||||
- Enhanced `ResearchSource` with credibility and domain authority
|
||||
- Enhanced response models with grounding status and quality metrics
|
||||
|
||||
### **Data Flow Architecture**
|
||||
```
|
||||
User Request → Content Type + Industry + Preferences
|
||||
↓
|
||||
Real Google Search → Industry-Relevant Current Sources
|
||||
↓
|
||||
Source Analysis → Identify Most Credible and Recent Sources
|
||||
↓
|
||||
Grounded Content Generation → AI Content with Source Integration
|
||||
↓
|
||||
Citation Addition → Automatic Inline Source Attribution
|
||||
↓
|
||||
Quality Validation → Ensure All Claims Are Properly Sourced
|
||||
↓
|
||||
Output Delivery → Professional Content with Inline Citations
|
||||
```
|
||||
|
||||
## 🔧 **Implementation Phases**
|
||||
|
||||
### **Phase 1: Native Google Search Grounding** ✅ **COMPLETED**
|
||||
|
||||
#### **Objectives** ✅ **ACHIEVED**
|
||||
- ✅ Implement native Google Search grounding functionality via Gemini API
|
||||
- ✅ Establish automatic citation system from grounding metadata
|
||||
- ✅ Enable automatic industry-relevant searches with no manual intervention
|
||||
- ✅ Build source verification and credibility ranking from grounding chunks
|
||||
|
||||
#### **Key Features** ✅ **IMPLEMENTED**
|
||||
- ✅ **Native Search Integration**: Gemini API automatically handles search queries and processing
|
||||
- ✅ **Automatic Source Extraction**: Sources extracted from `groundingMetadata.groundingChunks`
|
||||
- ✅ **Citation Generation**: Automatic inline citations from `groundingMetadata.groundingSupports`
|
||||
- ✅ **Quality Validation**: Content quality assessment with source coverage metrics
|
||||
- ✅ **Real-time Information**: Current data from the last month via native Google Search
|
||||
|
||||
#### **Technical Requirements** ✅ **COMPLETED**
|
||||
- ✅ Google GenAI library integration (`google-genai>=0.3.0`)
|
||||
- ✅ Native `google_search` tool configuration in Gemini API
|
||||
- ✅ Grounding metadata processing and source extraction
|
||||
- ✅ Citation formatting and link management from grounding data
|
||||
- ✅ Enhanced Gemini provider with native grounding capabilities
|
||||
|
||||
#### **Files Created/Modified** ✅ **COMPLETED**
|
||||
- ✅ `backend/services/llm_providers/gemini_grounded_provider.py` - Native grounding provider
|
||||
- ✅ `backend/services/linkedin_service.py` - Updated for native grounding
|
||||
- ✅ `backend/requirements.txt` - Updated Google GenAI dependencies
|
||||
- ✅ `backend/test_native_grounding.py` - Native grounding test script
|
||||
- ✅ **Architecture Simplified**: Removed custom Google Search service dependency
|
||||
- ✅ **Native Integration**: Direct Gemini API grounding tool usage
|
||||
- ✅ **Automatic Workflow**: Model handles search, processing, and citation automatically
|
||||
|
||||
### **Phase 2: URL Context Integration** 🔄 **PLANNED**
|
||||
|
||||
#### **Objectives**
|
||||
- Enable specific source grounding from user-provided URLs
|
||||
- Integrate curated industry report library
|
||||
- Implement competitor analysis capabilities
|
||||
- Build source management and organization system
|
||||
|
||||
#### **Key Features**
|
||||
- **URL Input System**: Allow users to provide relevant source URLs
|
||||
- **Industry Report Library**: Curated collection of authoritative sources
|
||||
- **Competitor Analysis**: Industry benchmarking and insights
|
||||
- **Source Categorization**: Organize sources by industry, type, and credibility
|
||||
- **Content Extraction**: Pull relevant information from specific URLs
|
||||
|
||||
#### **Technical Requirements**
|
||||
- Google AI API integration with `url_context` tool
|
||||
- URL validation and content extraction
|
||||
- Source categorization and tagging system
|
||||
- Content grounding in specific sources
|
||||
|
||||
### **Phase 3: Advanced Features** 📋 **PLANNED**
|
||||
|
||||
#### **Objectives**
|
||||
- Implement advanced analytics and performance tracking
|
||||
- Build AI-powered source credibility scoring
|
||||
- Enable multi-language industry insights
|
||||
- Create custom source integration capabilities
|
||||
|
||||
#### **Key Features**
|
||||
- **Performance Analytics**: Track content quality and user satisfaction
|
||||
- **Advanced Source Scoring**: AI-powered credibility assessment
|
||||
- **Multi-language Support**: International industry insights
|
||||
- **Custom Source Integration**: User-defined source libraries
|
||||
- **Quality Metrics Dashboard**: Real-time content quality monitoring
|
||||
|
||||
## 📊 **Content Quality Improvements**
|
||||
|
||||
### **Before vs. After Comparison**
|
||||
|
||||
| Aspect | Current State | Enhanced State |
|
||||
|--------|---------------|----------------|
|
||||
| **Factual Accuracy** | Generic AI claims | All claims backed by current sources |
|
||||
| **Industry Relevance** | Generic content | Grounded in latest industry trends |
|
||||
| **Source Verification** | No sources | Inline citations with clickable links |
|
||||
| **Information Recency** | Knowledge cutoff limited | Real-time current information |
|
||||
| **Professional Credibility** | Basic AI quality | Enterprise-grade content |
|
||||
| **User Trust** | Low (unverified content) | High (verifiable sources) |
|
||||
| **Research Quality** | Mock/simulated data | Real Google search results |
|
||||
| **Citation Coverage** | 0% | 95%+ of claims cited |
|
||||
|
||||
### **Specific LinkedIn Content Enhancements**
|
||||
|
||||
#### **Posts & Articles**
|
||||
- **Trending Topics**: Current industry discussions and hashtags
|
||||
- **Expert Insights**: Quotes and insights from industry leaders
|
||||
- **Data-Driven Content**: Statistics and research findings
|
||||
- **Competitive Analysis**: Industry benchmarking and insights
|
||||
- **Source Attribution**: Every claim backed by verifiable sources
|
||||
|
||||
#### **Carousels & Presentations**
|
||||
- **Visual Data**: Charts and graphs from industry reports
|
||||
- **Trend Analysis**: Current market movements and predictions
|
||||
- **Case Studies**: Real examples from industry leaders
|
||||
- **Best Practices**: Current industry standards and recommendations
|
||||
- **Citation Integration**: Source references for all data points
|
||||
|
||||
## 🎯 **Implementation Priorities**
|
||||
|
||||
### **High Priority (Phase 1)** ✅ **COMPLETED**
|
||||
1. ✅ **Google Search Integration**: Core grounding functionality
|
||||
2. ✅ **Citation System**: Inline source attribution
|
||||
3. ✅ **Enhanced Actions**: Search-enabled content generation
|
||||
4. ✅ **Quality Validation**: Source verification and fact-checking
|
||||
5. ✅ **Enhanced Gemini Provider**: Grounded content generation
|
||||
|
||||
### **Medium Priority (Phase 2)** 🔄 **NEXT**
|
||||
1. **URL Context Integration**: Specific source grounding
|
||||
2. **Industry Report Integration**: Curated source library
|
||||
3. **Competitor Analysis**: Industry benchmarking tools
|
||||
4. **Trend Monitoring**: Real-time industry insights
|
||||
5. **Source Management**: User control over source selection
|
||||
|
||||
### **Low Priority (Phase 3)** 📋 **PLANNED**
|
||||
1. **Advanced Analytics**: Content performance tracking
|
||||
2. **Source Ranking**: AI-powered source credibility scoring
|
||||
3. **Multi-language Support**: International industry insights
|
||||
4. **Custom Source Integration**: User-defined source libraries
|
||||
5. **Quality Dashboard**: Real-time content quality monitoring
|
||||
|
||||
## 💰 **Business Impact & ROI**
|
||||
|
||||
### **User Experience Improvements**
|
||||
- **Professional Credibility**: Enterprise-level content quality
|
||||
- **Time Savings**: Research-backed content in minutes vs. hours
|
||||
- **Trust Building**: Verifiable sources increase user confidence
|
||||
- **Industry Relevance**: Always current and relevant content
|
||||
- **Source Transparency**: Users can verify all claims
|
||||
|
||||
### **Competitive Advantages**
|
||||
- **Unique Positioning**: First LinkedIn tool with grounded AI content
|
||||
- **Quality Differentiation**: Professional-grade vs. generic AI content
|
||||
- **Trust Leadership**: Source verification builds user loyalty
|
||||
- **Industry Expertise**: Deep industry knowledge and insights
|
||||
- **Enterprise Appeal**: Suitable for professional and corporate use
|
||||
|
||||
### **Revenue Impact**
|
||||
- **Premium Pricing**: Enterprise-grade features justify higher pricing
|
||||
- **User Retention**: Higher quality content increases user loyalty
|
||||
- **Market Expansion**: Appeal to enterprise and professional users
|
||||
- **Partnership Opportunities**: Industry report providers and publishers
|
||||
- **Subscription Upgrades**: Premium grounding features drive upgrades
|
||||
|
||||
## 🔒 **Technical Requirements & Dependencies**
|
||||
|
||||
### **Google AI API Requirements** ✅ **IMPLEMENTED**
|
||||
- ✅ **API Access**: Google AI API with grounding capabilities
|
||||
- ✅ **Search API**: Google Custom Search API for industry research
|
||||
- ✅ **Authentication**: Proper API key management and security
|
||||
- ✅ **Rate Limits**: Understanding and managing API usage limits
|
||||
- ✅ **Cost Management**: Monitoring and optimizing API costs
|
||||
|
||||
### **Infrastructure Requirements** ✅ **COMPLETED**
|
||||
- ✅ **Backend Services**: Enhanced content generation pipeline
|
||||
- ✅ **Database**: Source management and citation storage
|
||||
- ✅ **Caching**: Search result caching for performance
|
||||
- ✅ **Monitoring**: API usage and content quality monitoring
|
||||
- ✅ **Fallback Systems**: Graceful degradation when APIs fail
|
||||
|
||||
### **Security & Compliance**
|
||||
- **Data Privacy**: Secure handling of user content and sources
|
||||
- **Source Validation**: Ensuring sources are safe and appropriate
|
||||
- **Content Moderation**: Filtering inappropriate or unreliable sources
|
||||
- **Compliance**: Meeting industry and regulatory requirements
|
||||
- **API Security**: Secure API key management and usage
|
||||
|
||||
## 📈 **Success Metrics & KPIs**
|
||||
|
||||
### **Content Quality Metrics**
|
||||
- **Source Verification Rate**: Percentage of claims with citations
|
||||
- **Source Credibility Score**: Average credibility of used sources
|
||||
- **Content Freshness**: Age of information used in content
|
||||
- **User Satisfaction**: Content quality ratings and feedback
|
||||
- **Citation Coverage**: Percentage of factual claims properly cited
|
||||
|
||||
### **Business Metrics**
|
||||
- **User Adoption**: Increase in enterprise user adoption
|
||||
- **Content Usage**: Higher engagement with generated content
|
||||
- **User Retention**: Improved user loyalty and retention
|
||||
- **Revenue Growth**: Increased pricing and subscription rates
|
||||
- **Premium Feature Usage**: Adoption of grounding features
|
||||
|
||||
### **Technical Metrics**
|
||||
- **API Performance**: Response times and reliability
|
||||
- **Search Accuracy**: Relevance of search results
|
||||
- **Citation Accuracy**: Proper source attribution
|
||||
- **System Uptime**: Overall system reliability
|
||||
- **Fallback Success Rate**: Successful degradation when needed
|
||||
|
||||
## 🚧 **Risk Assessment & Mitigation**
|
||||
|
||||
### **Technical Risks**
|
||||
- **API Dependencies**: Google AI API availability and changes
|
||||
- **Performance Issues**: Search integration impact on response times
|
||||
- **Cost Overruns**: Uncontrolled API usage and costs
|
||||
- **Integration Complexity**: Technical challenges in implementation
|
||||
|
||||
### **Mitigation Strategies** ✅ **IMPLEMENTED**
|
||||
- ✅ **API Redundancy**: Backup content generation methods
|
||||
- ✅ **Performance Optimization**: Efficient search and caching strategies
|
||||
- ✅ **Cost Controls**: Usage monitoring and optimization
|
||||
- ✅ **Phased Implementation**: Gradual rollout to manage complexity
|
||||
- ✅ **Fallback Systems**: Graceful degradation to existing methods
|
||||
|
||||
### **Business Risks**
|
||||
- **User Adoption**: Resistance to new features or workflows
|
||||
- **Quality Expectations**: Meeting high enterprise standards
|
||||
- **Competitive Response**: Other tools implementing similar features
|
||||
- **Market Changes**: Shifts in user needs or preferences
|
||||
|
||||
### **Mitigation Strategies**
|
||||
- **User Education**: Clear communication of benefits and value
|
||||
- **Quality Assurance**: Rigorous testing and validation
|
||||
- **Continuous Innovation**: Staying ahead of competition
|
||||
- **User Feedback**: Regular input and iteration
|
||||
- **Beta Testing**: Gradual rollout with user feedback
|
||||
|
||||
## 🔄 **Migration Strategy**
|
||||
|
||||
### **Current System Analysis** ✅ **COMPLETED**
|
||||
- ✅ **LinkedIn Service**: Well-structured with research capabilities
|
||||
- ✅ **Gemini Provider**: Google AI integration already in place
|
||||
- ✅ **Mock Research**: Current `_conduct_research` method
|
||||
- ✅ **CopilotKit Actions**: Frontend actions for content generation
|
||||
|
||||
### **Migration Approach** ✅ **IMPLEMENTED**
|
||||
- ✅ **Incremental Enhancement**: Build on existing infrastructure
|
||||
- ✅ **Feature Flags**: Enable/disable grounding features
|
||||
- ✅ **Backward Compatibility**: Maintain existing functionality
|
||||
- ✅ **User Choice**: Allow users to opt-in to grounding features
|
||||
- ✅ **Performance Monitoring**: Track impact on existing systems
|
||||
|
||||
### **Rollout Plan** 🔄 **IN PROGRESS**
|
||||
- ✅ **Phase 1**: Core grounding for posts and articles
|
||||
- 🔄 **Phase 2**: Enhanced source management and URL context
|
||||
- 📋 **Phase 3**: Advanced analytics and quality monitoring
|
||||
- 🔄 **User Groups**: Start with power users, expand gradually
|
||||
- 🔄 **Feedback Integration**: Continuous improvement based on usage
|
||||
|
||||
## 🔧 **Recent Fixes Applied**
|
||||
|
||||
### **Service Refactoring & Code Organization** ✅ **COMPLETED**
|
||||
- ✅ **LinkedIn Service Refactoring**: Extracted quality metrics handling to separate `QualityHandler` module
|
||||
- ✅ **Content Generation Extraction**: Moved large post and article generation methods to `ContentGenerator` module
|
||||
- ✅ **Research Logic Extraction**: Extracted research handling logic to `ResearchHandler` module
|
||||
- ✅ **Code Organization**: Created `backend/services/linkedin/` package for better code structure
|
||||
- ✅ **Quality Metrics Extraction**: Moved complex quality metrics creation logic to dedicated handler
|
||||
- ✅ **Maintainability Improvement**: Significantly reduced `linkedin_service.py` complexity and improved readability
|
||||
- ✅ **Function Size Reduction**: Broke down large functions into focused, manageable modules
|
||||
|
||||
### **Critical Bug Fixes** ✅ **COMPLETED**
|
||||
- ✅ **Citation Processing Fixed**: Updated `CitationManager` to handle both Dict and ResearchSource Pydantic models
|
||||
- ✅ **Quality Analysis Fixed**: Updated `ContentQualityAnalyzer` to work with ResearchSource objects
|
||||
- ✅ **Data Type Compatibility**: Resolved `.get()` method calls on Pydantic model objects
|
||||
- ✅ **Service Integration**: All citation and quality services now work correctly with native grounding
|
||||
|
||||
### **Grounding Debugging & Error Handling** ✅ **COMPLETED**
|
||||
- ✅ **Removed Mock Data Fallbacks**: Eliminated all fallback mock sources that were masking real issues
|
||||
- ✅ **Enhanced Error Logging**: Added detailed logging of API response structure and grounding metadata
|
||||
- ✅ **Fail-Fast Approach**: Services now fail immediately instead of silently falling back to mock data
|
||||
- ✅ **Debug Information**: Added comprehensive logging of response attributes, types, and values
|
||||
- ✅ **Critical Error Detection**: Clear error messages when grounding chunks, supports, or metadata are missing
|
||||
|
||||
### **Frontend Grounding Data Display** ✅ **COMPLETED**
|
||||
- ✅ **GroundingDataDisplay Component**: Created comprehensive component to show research sources, citations, and quality metrics
|
||||
- ✅ **Enhanced Interfaces**: Updated TypeScript interfaces to include grounding data fields (citations, quality_metrics, grounding_enabled)
|
||||
- ✅ **Real-time Updates**: Frontend now listens for grounding data updates from CopilotKit actions
|
||||
- ✅ **Rich Data Visualization**: Displays quality scores, source credibility, citation coverage, and research source details
|
||||
- ✅ **Professional UI**: Clean, enterprise-grade interface showing AI-generated content with factual grounding
|
||||
|
||||
### **Import Error Resolution** ✅ **COMPLETED**
|
||||
- ✅ **Fixed Relative Import Errors**: Changed all relative imports to absolute imports
|
||||
- ✅ **Updated Service Import Paths**: Fixed `__init__.py` files to use correct import paths
|
||||
- ✅ **Router Import Fix**: Fixed LinkedIn router to import `LinkedInService` class and create instance
|
||||
- ✅ **Function Name Corrections**: Updated to use correct Gemini provider function names
|
||||
- ✅ **Graceful Service Initialization**: Added try-catch blocks for missing dependencies
|
||||
|
||||
### **Files Modified**
|
||||
- `backend/services/linkedin_service.py` - Fixed imports, added error handling, and **SIGNIFICANTLY REFACTORED** for maintainability
|
||||
- `backend/routers/linkedin.py` - Fixed service import, initialization, and method calls
|
||||
- `backend/services/research/__init__.py` - Fixed import paths
|
||||
- `backend/services/citation/__init__.py` - Fixed import paths
|
||||
- `backend/services/quality/__init__.py` - Fixed import paths
|
||||
- `backend/services/llm_providers/__init__.py` - Fixed import paths and function names
|
||||
- `backend/services/linkedin/quality_handler.py` - **NEW**: Extracted quality metrics handling to separate module
|
||||
- `backend/services/linkedin/content_generator.py` - **NEW**: Extracted large content generation methods (posts & articles)
|
||||
- `backend/services/linkedin/research_handler.py` - **NEW**: Extracted research logic and timing handling
|
||||
- `backend/services/linkedin/__init__.py` - **NEW**: Package initialization for linkedin services
|
||||
- `backend/services/citation/citation_manager.py` - **FIXED**: Updated to handle ResearchSource Pydantic models
|
||||
- `backend/services/quality/content_analyzer.py` - **FIXED**: Updated to work with ResearchSource objects
|
||||
- `backend/services/llm_providers/gemini_grounded_provider.py` - **FIXED**: Removed mock data fallbacks, enhanced error handling and debugging
|
||||
- `frontend/src/services/linkedInWriterApi.ts` - **ENHANCED**: Added grounding data interfaces (citations, quality_metrics, grounding_enabled)
|
||||
- `frontend/src/components/LinkedInWriter/components/GroundingDataDisplay.tsx` - **NEW**: Component to display research sources, citations, and quality metrics
|
||||
- `frontend/src/components/LinkedInWriter/components/ContentEditor.tsx` - **ENHANCED**: Integrated grounding data display
|
||||
- `frontend/src/components/LinkedInWriter/hooks/useLinkedInWriter.ts` - **ENHANCED**: Added grounding data state management
|
||||
- `frontend/src/components/LinkedInWriter/RegisterLinkedInActions.tsx` - **ENHANCED**: Updated to extract and pass grounding data
|
||||
- `backend/test_imports.py` - Created comprehensive import test script
|
||||
- `backend/test_linkedin_service.py` - Created service functionality test script
|
||||
- `backend/test_request_validation.py` - Created request validation test script
|
||||
- `frontend/src/services/linkedInWriterApi.ts` - Added missing grounding fields to request interfaces
|
||||
- `frontend/src/components/LinkedInWriter/RegisterLinkedInActions.tsx` - Updated actions to send required grounding fields
|
||||
|
||||
## 🧪 **Testing & Validation**
|
||||
|
||||
### **Integration Testing** ✅ **COMPLETED**
|
||||
- ✅ **Test Script**: `backend/test_grounding_integration.py`
|
||||
- ✅ **Service Initialization**: All new services initialize correctly
|
||||
- ✅ **Content Generation**: Grounded content generation works
|
||||
- ✅ **Citation System**: Citations are properly generated and formatted
|
||||
- ✅ **Quality Analysis**: Content quality metrics are calculated
|
||||
- ✅ **Fallback Systems**: Graceful degradation when grounding fails
|
||||
|
||||
### **Test Coverage**
|
||||
- ✅ **Individual Services**: Each service component tested independently
|
||||
- ✅ **Integration Flow**: Complete content generation pipeline tested
|
||||
- ✅ **Error Handling**: Fallback mechanisms and error scenarios tested
|
||||
- ✅ **Performance**: Response times and resource usage monitored
|
||||
- ✅ **API Integration**: Google Search and Gemini API integration tested
|
||||
|
||||
### **Next Testing Steps**
|
||||
- ✅ **Import Issues Resolved**: All import errors fixed and services working
|
||||
- ✅ **Service Initialization**: All services initialize successfully with graceful fallbacks
|
||||
- ✅ **Basic Functionality**: LinkedIn post generation working correctly
|
||||
- ✅ **Core Grounding Components**: Provider initialization, prompt building, and content processing verified
|
||||
- ✅ **Router Method Calls Fixed**: All LinkedIn service method calls corrected
|
||||
- ✅ **Backend Startup**: Backend imports and starts successfully
|
||||
- ✅ **Service Integration**: LinkedIn service integration working correctly
|
||||
- ✅ **Request Validation Fixed**: Frontend now sends required grounding fields
|
||||
- ✅ **Pydantic Model Validation**: Request validation working correctly
|
||||
- 🔄 **API Integration Testing**: Test with different API keys and rate limits
|
||||
- 🔄 **Content Generation Testing**: Verify actual content generation with grounding
|
||||
- 🔄 **User Acceptance Testing**: Real user scenarios and feedback
|
||||
- 🔄 **Performance Testing**: Load testing and optimization
|
||||
- 🔄 **Security Testing**: API key management and data security
|
||||
- 🔄 **Compliance Testing**: Industry standards and regulations
|
||||
- 🔄 **End-to-End Testing**: Complete user workflow validation
|
||||
|
||||
## 🚀 **Next Implementation Steps**
|
||||
|
||||
### **Week 1: API Integration & Testing** 🔄 **IMMEDIATE PRIORITY**
|
||||
|
||||
#### **1. API Key Management & Testing**
|
||||
- **Test with different API keys**: Verify grounding works with various API configurations
|
||||
- **Rate limit handling**: Implement proper retry logic and rate limit management
|
||||
- **API quota monitoring**: Track usage and implement cost controls
|
||||
- **Fallback mechanisms**: Ensure graceful degradation when API is unavailable
|
||||
|
||||
#### **2. Content Generation Verification**
|
||||
- **Test actual content generation**: Verify that grounded content is being generated
|
||||
- **Source extraction testing**: Ensure sources are properly extracted from grounding metadata
|
||||
- **Citation generation**: Test inline citation formatting and source attribution
|
||||
- **Quality metrics**: Verify content quality assessment is working
|
||||
|
||||
#### **3. Integration Testing**
|
||||
- **End-to-end workflow**: Test complete LinkedIn content generation pipeline
|
||||
- **Error handling**: Verify all error scenarios are handled gracefully
|
||||
- **Performance testing**: Measure response times and optimize where needed
|
||||
- **User acceptance testing**: Test with real user scenarios
|
||||
|
||||
### **Week 2: Phase 2 - URL Context Integration** 📋 **NEXT PHASE**
|
||||
|
||||
#### **1. URL Context Service Implementation**
|
||||
- **Create URL context service**: `backend/services/url_context/url_context_service.py`
|
||||
- **Google AI URL context tool**: Integrate with `url_context` tool from Google AI
|
||||
- **URL validation**: Implement proper URL validation and content extraction
|
||||
- **Source categorization**: Build system to categorize and tag sources
|
||||
|
||||
#### **2. Enhanced Source Management**
|
||||
- **Industry report library**: Curated collection of authoritative sources
|
||||
- **Competitor analysis**: Industry benchmarking and insights
|
||||
- **Source credibility scoring**: AI-powered source assessment
|
||||
- **User source input**: Allow users to provide custom URLs
|
||||
|
||||
#### **3. Advanced Features**
|
||||
- **Multi-language support**: International industry insights
|
||||
- **Custom source integration**: User-defined source libraries
|
||||
- **Quality dashboard**: Real-time content quality monitoring
|
||||
- **Performance analytics**: Track content quality and user satisfaction
|
||||
|
||||
### **Week 3: Production Deployment** 📋 **FUTURE PHASE**
|
||||
|
||||
#### **1. Production Readiness**
|
||||
- **Security hardening**: API key management and data security
|
||||
- **Performance optimization**: Caching, rate limiting, and response optimization
|
||||
- **Monitoring & alerting**: Real-time system monitoring and error tracking
|
||||
- **Documentation**: Complete API documentation and user guides
|
||||
|
||||
#### **2. User Experience**
|
||||
- **UI/UX improvements**: Enhanced grounding level selection interface
|
||||
- **Source preview**: Allow users to preview sources before generation
|
||||
- **Citation management**: User-friendly citation editing and management
|
||||
- **Quality feedback**: User feedback integration for continuous improvement
|
||||
|
||||
#### **3. Business Integration**
|
||||
- **Premium features**: Enterprise-grade grounding features
|
||||
- **Analytics dashboard**: Business metrics and usage analytics
|
||||
- **Customer support**: Support tools and documentation
|
||||
- **Marketing materials**: Case studies and success stories
|
||||
|
||||
## 📚 **References & Resources**
|
||||
|
||||
### **Google AI Documentation**
|
||||
- [Google Search Grounding](https://ai.google.dev/gemini-api/docs/google-search)
|
||||
- [URL Context Integration](https://ai.google.dev/gemini-api/docs/url-context)
|
||||
- [Gemini API Reference](https://ai.google.dev/gemini-api/docs/api-reference)
|
||||
- [Google Custom Search API](https://developers.google.com/custom-search)
|
||||
|
||||
### **Industry Standards**
|
||||
- LinkedIn Content Best Practices
|
||||
- Enterprise Content Quality Standards
|
||||
- Professional Citation Guidelines
|
||||
- Industry Research Methodologies
|
||||
- Source Credibility Assessment
|
||||
|
||||
### **Technical Resources**
|
||||
- CopilotKit Integration Guides
|
||||
- Google AI API Best Practices
|
||||
- Content Quality Assessment Tools
|
||||
- Performance Optimization Techniques
|
||||
- API Rate Limiting Strategies
|
||||
|
||||
### **Implementation Resources** ✅ **CREATED**
|
||||
- ✅ **Service Documentation**: Comprehensive service implementations
|
||||
- ✅ **Test Scripts**: Integration testing and validation
|
||||
- ✅ **Code Examples**: Working implementations for all components
|
||||
- ✅ **Dependency Management**: Updated requirements and dependencies
|
||||
- ✅ **Error Handling**: Robust fallback and error management
|
||||
|
||||
---
|
||||
|
||||
## 📝 **Document Information**
|
||||
|
||||
- **Document Version**: 3.0
|
||||
- **Last Updated**: January 2025
|
||||
- **Author**: ALwrity Development Team
|
||||
- **Review Cycle**: Quarterly
|
||||
- **Next Review**: April 2025
|
||||
- **Implementation Status**: Phase 1 Completed, Phase 2 Planning
|
||||
|
||||
---
|
||||
|
||||
*This document serves as the comprehensive guide for implementing LinkedIn factual Google grounded URL content enhancement in ALwrity. Phase 1 core services have been completed and are ready for testing and deployment. All implementation decisions should reference this document for consistency and alignment with the overall strategy.*
|
||||
431
docs/story writer/ANIME_STORY_BIBLE_IMPLEMENTATION.md
Normal file
431
docs/story writer/ANIME_STORY_BIBLE_IMPLEMENTATION.md
Normal file
@@ -0,0 +1,431 @@
|
||||
# Anime Story Bible – Design & Implementation (Story Writer SSOT)
|
||||
|
||||
This document is the single source of truth for the **Anime Story Bible** in Story Writer: what it is, where it lives in the codebase, how it is generated, and how it is used across outline, story text, images, and motion/animation.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Concepts
|
||||
|
||||
- **Anime Story Bible**: Structured description of characters, world, and visual style for anime-style stories. It is designed to be:
|
||||
- Stable across the whole story (single bible per story)
|
||||
- Machine-readable (Pydantic/TypeScript models)
|
||||
- Reusable for text, image, and video prompts
|
||||
|
||||
- **Design Goals**
|
||||
- Maintain **character consistency** across scenes and media
|
||||
- Maintain **world rules** (tech/magic, constraints) throughout the narrative
|
||||
- Maintain a **coherent anime visual style** across images and motion clips
|
||||
- Allow future reuse for other story templates and media pipelines
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Model (Backend & Frontend)
|
||||
|
||||
### 2.1 Backend Models
|
||||
|
||||
File: [story_models.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/models/story_models.py)
|
||||
|
||||
Key classes:
|
||||
|
||||
- `AnimeCharacter`
|
||||
- `AnimeWorld`
|
||||
- `AnimeVisualStyle`
|
||||
- `AnimeStoryBible`
|
||||
|
||||
They are defined as:
|
||||
|
||||
- `AnimeCharacter`
|
||||
- `id`: stable snake_case identifier
|
||||
- `name`
|
||||
- `age_range`
|
||||
- `role` (protagonist, antagonist, mentor, etc.)
|
||||
- `look` (key visual details)
|
||||
- `outfit_palette`
|
||||
- `personality_tags: List[str]`
|
||||
|
||||
- `AnimeWorld`
|
||||
- `setting` (locations and general world description)
|
||||
- `era` (near-future, alt 1990s, etc.)
|
||||
- `tech_or_magic_level`
|
||||
- `core_rules: List[str]` (constraints and consistent rules)
|
||||
|
||||
- `AnimeVisualStyle`
|
||||
- `style_preset` (anime_manga, cinematic_anime, cozy_slice_of_life, etc.)
|
||||
- `camera_style`
|
||||
- `color_mood`
|
||||
- `lighting`
|
||||
- `line_style`
|
||||
- `extra_tags: List[str]`
|
||||
|
||||
- `AnimeStoryBible`
|
||||
- `story_id?: str`
|
||||
- `main_cast: List[AnimeCharacter]`
|
||||
- `world: AnimeWorld`
|
||||
- `visual_style: AnimeVisualStyle`
|
||||
|
||||
The bible is attached to:
|
||||
|
||||
- `StorySetupOption.anime_bible: Optional[AnimeStoryBible]`
|
||||
- `StoryOutlineResponse.anime_bible: Optional[AnimeStoryBible]`
|
||||
|
||||
Additionally, for downstream usage:
|
||||
|
||||
- `StoryGenerationRequest.anime_bible: Optional[Dict[str, Any]]`
|
||||
- `StoryContinueRequest.anime_bible: Optional[Dict[str, Any]]`
|
||||
|
||||
This allows story-start and continuation to receive a JSON-serializable bible blob without strict coupling to the `AnimeStoryBible` class.
|
||||
|
||||
### 2.2 Frontend Models
|
||||
|
||||
File: [storyWriterApi.ts](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/services/storyWriterApi.ts)
|
||||
|
||||
Types mirror the backend:
|
||||
|
||||
- `AnimeCharacter`
|
||||
- `AnimeWorld`
|
||||
- `AnimeVisualStyle`
|
||||
- `AnimeStoryBible`
|
||||
|
||||
The bible flows through:
|
||||
|
||||
- `StoryOutlineResponse.anime_bible?: AnimeStoryBible`
|
||||
- `StoryGenerationRequest.anime_bible?: AnimeStoryBible | null`
|
||||
- `StoryStartRequest` and `StoryContinueRequest` (via `StoryGenerationRequest`)
|
||||
|
||||
State layer:
|
||||
|
||||
File: [useStoryWriterState.ts](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/hooks/useStoryWriterState.ts)
|
||||
|
||||
- `StoryWriterState.animeBible: any | null`
|
||||
- `setAnimeBible` setter
|
||||
- Persisted and restored via `localStorage`:
|
||||
- Saved under `animeBible` key in the serialized state
|
||||
- Ensures the bible survives refreshes
|
||||
|
||||
---
|
||||
|
||||
## 3. Bible Lifecycle & Generation
|
||||
|
||||
### 3.1 Generation Source
|
||||
|
||||
The Anime Story Bible is generated in the **story setup / outline** pipeline on the backend:
|
||||
|
||||
- The story setup step produces a single `StorySetupOption` enriched with `anime_bible` when the selected template is anime-focused.
|
||||
- The outline generation step (`StoryOutlineResponse`) carries `anime_bible` so the UI can display it and store it in Story Writer state.
|
||||
|
||||
SSOT:
|
||||
|
||||
- Models: [story_models.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/models/story_models.py)
|
||||
- Outline response: `StoryOutlineResponse.anime_bible`
|
||||
|
||||
### 3.2 Frontend Storage and Access
|
||||
|
||||
The frontend receives `anime_bible` from `StoryOutlineResponse` and:
|
||||
|
||||
- Stores it in `StoryWriterState.animeBible`
|
||||
- Persists it in `localStorage` with the rest of the story writer state
|
||||
- Exposes it to:
|
||||
- Director chip / bible viewer UI
|
||||
- Story generation (start/continue)
|
||||
- Scene animation (via `story_context`)
|
||||
|
||||
Key locations:
|
||||
|
||||
- State hook: [useStoryWriterState.ts](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/hooks/useStoryWriterState.ts)
|
||||
- Outline phase UI: [StoryOutline.tsx](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/components/StoryWriter/Phases/StoryOutline.tsx)
|
||||
|
||||
---
|
||||
|
||||
## 4. Integration Points (Current Implementation)
|
||||
|
||||
This section documents how the Anime Story Bible is currently used across the Story Writer pipelines.
|
||||
|
||||
### 4.1 Story Text Generation (Start & Continue)
|
||||
|
||||
#### 4.1.1 Requests
|
||||
|
||||
Frontend:
|
||||
|
||||
- `StoryGenerationRequest` (base)
|
||||
- Now includes `anime_bible?: AnimeStoryBible | null`
|
||||
- `getRequest()` in `useStoryWriterState` adds `anime_bible` automatically:
|
||||
|
||||
- [useStoryWriterState.ts:getRequest](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/hooks/useStoryWriterState.ts#L420-L461)
|
||||
|
||||
Story start:
|
||||
|
||||
- `StoryWriting.handleGenerateStart`:
|
||||
- Builds `request = state.getRequest()`
|
||||
- Calls `storyWriterApi.generateStoryStart(premise, outline, request)`
|
||||
- [StoryWriting.tsx](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/components/StoryWriter/Phases/StoryWriting.tsx#L308-L328)
|
||||
|
||||
Story continue:
|
||||
|
||||
- `StoryWriting.handleContinue`:
|
||||
- `request = state.getRequest()`
|
||||
- Builds `continueRequest = { ...request, premise, outline, story_text }`
|
||||
- Calls `storyWriterApi.continueStory(continueRequest)`
|
||||
- [StoryWriting.tsx](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/components/StoryWriter/Phases/StoryWriting.tsx#L377-L388)
|
||||
|
||||
Backend:
|
||||
|
||||
- `StoryGenerationRequest` / `StoryContinueRequest` include `anime_bible: Optional[Dict[str, Any]]`
|
||||
- [story_models.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/models/story_models.py#L11-L43)
|
||||
- [story_models.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/models/story_models.py#L243-L259)
|
||||
|
||||
#### 4.1.2 Routing Layer
|
||||
|
||||
File: [api/story_writer/routes/story_content.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/api/story_writer/routes/story_content.py)
|
||||
|
||||
- `generate_story_start`:
|
||||
|
||||
```python
|
||||
story_start = story_service.generate_story_start(
|
||||
premise=request.premise,
|
||||
outline=outline_data,
|
||||
persona=request.persona,
|
||||
story_setting=request.story_setting,
|
||||
character_input=request.character_input,
|
||||
plot_elements=request.plot_elements,
|
||||
writing_style=request.writing_style,
|
||||
story_tone=request.story_tone,
|
||||
narrative_pov=request.narrative_pov,
|
||||
audience_age_group=request.audience_age_group,
|
||||
content_rating=request.content_rating,
|
||||
ending_preference=request.ending_preference,
|
||||
story_length=story_length,
|
||||
anime_bible=getattr(request, "anime_bible", None),
|
||||
user_id=user_id,
|
||||
)
|
||||
```
|
||||
|
||||
- `continue_story`:
|
||||
|
||||
```python
|
||||
continuation = story_service.continue_story(
|
||||
premise=request.premise,
|
||||
outline=outline_data,
|
||||
story_text=request.story_text,
|
||||
persona=request.persona,
|
||||
story_setting=request.story_setting,
|
||||
character_input=request.character_input,
|
||||
plot_elements=request.plot_elements,
|
||||
writing_style=request.writing_style,
|
||||
story_tone=request.story_tone,
|
||||
narrative_pov=request.narrative_pov,
|
||||
audience_age_group=request.audience_age_group,
|
||||
content_rating=request.content_rating,
|
||||
ending_preference=request.ending_preference,
|
||||
anime_bible=getattr(request, "anime_bible", None),
|
||||
story_length=story_length,
|
||||
user_id=user_id,
|
||||
)
|
||||
```
|
||||
|
||||
#### 4.1.3 Service Layer Prompts
|
||||
|
||||
File: [story_content.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/services/story_writer/service_components/story_content.py)
|
||||
|
||||
- `StoryContentMixin.generate_story_start(...)` now accepts `anime_bible: Optional[Dict[str, Any]]` and injects a serialized bible block right after the persona prompt:
|
||||
|
||||
```python
|
||||
anime_bible_context = ""
|
||||
if anime_bible:
|
||||
try:
|
||||
serialized_bible = json.dumps(anime_bible, ensure_ascii=False, indent=2)
|
||||
except Exception:
|
||||
serialized_bible = str(anime_bible)
|
||||
anime_bible_context = f"""
|
||||
|
||||
You also have a structured ANIME STORY BIBLE that defines the main cast, world rules, and visual style. Use it as a hard constraint for character consistency, worldbuilding, and visual storytelling:
|
||||
|
||||
{serialized_bible}
|
||||
"""
|
||||
```
|
||||
|
||||
The context is included for both short-story and longer-story prompts. This ensures:
|
||||
|
||||
- Character, world, and style constraints are explicitly visible to the LLM
|
||||
- The same bible is applied consistently for start and continuation
|
||||
|
||||
- `StoryContentMixin.continue_story(...)` similarly accepts `anime_bible` and injects the same `anime_bible_context` into the continuation prompt, directly after `persona_prompt`.
|
||||
|
||||
This means every text generation step (start and continue) is conditioned on the bible when present.
|
||||
|
||||
### 4.2 Scene Animation (Image-to-Video)
|
||||
|
||||
The current bible-aware integration is focused on **motion prompts** for Kling image-to-video.
|
||||
|
||||
#### 4.2.1 Frontend: story_context payload
|
||||
|
||||
File: [StoryOutline.tsx](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/frontend/src/components/StoryWriter/Phases/StoryOutline.tsx)
|
||||
|
||||
`createStoryContextPayload` includes `anime_bible`:
|
||||
|
||||
```ts
|
||||
const createStoryContextPayload = () => ({
|
||||
persona: state.persona,
|
||||
story_setting: state.storySetting,
|
||||
characters: state.characters,
|
||||
plot_elements: state.plotElements,
|
||||
writing_style: state.writingStyle,
|
||||
story_tone: state.storyTone,
|
||||
narrative_pov: state.narrativePOV,
|
||||
audience_age_group: state.audienceAgeGroup,
|
||||
content_rating: state.contentRating,
|
||||
story_length: state.storyLength,
|
||||
premise: state.premise,
|
||||
outline: state.outline,
|
||||
story_content: state.storyContent,
|
||||
anime_bible: state.animeBible,
|
||||
});
|
||||
```
|
||||
|
||||
This payload is passed to:
|
||||
|
||||
- `storyWriterApi.animateScene(...)`
|
||||
- `storyWriterApi.animateSceneVoiceover(...)`
|
||||
|
||||
#### 4.2.2 Backend: Kling animation service
|
||||
|
||||
File: [kling_animation.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/services/wavespeed/kling_animation.py)
|
||||
|
||||
- `animate_scene_image(...)` is unchanged in signature but passes `story_context` to `generate_animation_prompt(...)`.
|
||||
|
||||
- `_build_fallback_prompt(scene_data, story_context)`:
|
||||
- Reads `anime_bible = story_context.get("anime_bible")`
|
||||
- Extracts:
|
||||
- Visual style details (`style_preset`, `camera_style`, `color_mood`, `lighting`, `line_style`, `extra_tags`)
|
||||
- World `setting`
|
||||
- `main_cast` names
|
||||
- Appends a concise style text to the deterministic fallback prompt to preserve:
|
||||
- Visual style
|
||||
- World flavor
|
||||
- Character design consistency (names only)
|
||||
|
||||
- `generate_animation_prompt(scene_data, story_context, user_id)`:
|
||||
- Builds an `ANIME STORY BIBLE VISUAL GUIDANCE` block when `anime_bible` is present, e.g.:
|
||||
- Visual style preset and camera style
|
||||
- Color mood, lighting, line style
|
||||
- Extra style tags
|
||||
- Main cast names to keep visually consistent
|
||||
- World/setting context
|
||||
- Inserts this block between `Setting` and the “Focus on” bullet list in the LLM prompt.
|
||||
- Both structured JSON responses and fallback text generation flows see this block.
|
||||
|
||||
Result:
|
||||
|
||||
- Motion prompts for Kling image-to-video are constrained by the bible, making animations conform to:
|
||||
- The same anime visual style
|
||||
- The same character set
|
||||
- The same world tone
|
||||
|
||||
### 4.3 Images (Current State)
|
||||
|
||||
Image generation currently uses:
|
||||
|
||||
- `StoryScene.image_prompt` generated during outline generation
|
||||
- Image provider settings (width, height, model)
|
||||
|
||||
The anime bible is not yet used in a **second-pass image prompt rewriter**. However:
|
||||
|
||||
- The bible is already aligned with the same outline and template that produced `image_prompt`.
|
||||
- The bible is threaded into Story Writer’s state and can be used later to refine image prompts or add style constraints.
|
||||
|
||||
Planned enhancement (not yet implemented):
|
||||
|
||||
- Add a lightweight prompt refinement step that:
|
||||
- Takes `scene.image_prompt` + `AnimeStoryBible.visual_style`
|
||||
- Emits a style-constrained `final_image_prompt`
|
||||
- Passes that to the image generation service
|
||||
|
||||
This document should be updated when that enhancement is implemented.
|
||||
|
||||
---
|
||||
|
||||
## 5. Adapting the Bible to Other Story Types
|
||||
|
||||
Although the Anime Story Bible is currently wired for anime-focused stories, the architecture is intentionally reusable.
|
||||
|
||||
### 5.1 Reuse Strategy
|
||||
|
||||
- **Data model**:
|
||||
- `AnimeStoryBible` is anime-specific, but the concept of a structured “story bible” (cast + world + style) is general.
|
||||
- Future story types can introduce sibling models (e.g., `NovelStoryBible`, `ComicStoryBible`) reusing similar patterns.
|
||||
|
||||
- **Transport layer**:
|
||||
- Requests use a flexible `anime_bible: Optional[Dict[str, Any]]` at the story-generation level.
|
||||
- This can be generalized to a `story_bible` field if we want cross-template reuse.
|
||||
|
||||
- **Prompt patterns**:
|
||||
- Both text and motion pipelines use the same pattern:
|
||||
- Serialize the bible
|
||||
- Inject a dedicated paragraph or bullet block
|
||||
- Explicitly instruct the model to treat it as a hard constraint
|
||||
- This pattern is template-agnostic and can be reused for other story modes.
|
||||
|
||||
### 5.2 Extension Guidelines
|
||||
|
||||
When adapting the bible pattern to other story types:
|
||||
|
||||
1. **Define the bible model**
|
||||
- Add a dedicated Pydantic model under `story_models.py`.
|
||||
- Mirror it with a TypeScript interface in `storyWriterApi.ts`.
|
||||
|
||||
2. **Attach it to responses**
|
||||
- Extend the relevant response models (setup, outline, etc.) with an optional bible field.
|
||||
- Ensure the generating service populates it when the template supports it.
|
||||
|
||||
3. **Thread through state**
|
||||
- Add a field to `StoryWriterState`.
|
||||
- Persist it in `localStorage`.
|
||||
- Provide setter(s) and ensure it is included in `getRequest()` when relevant.
|
||||
|
||||
4. **Inject into prompts**
|
||||
- Text: add a serialized bible context block after the persona prompt for:
|
||||
- Story start
|
||||
- Story continuation
|
||||
- Media: add a structured guidance block to:
|
||||
- Image prompt generation (if using AI to build prompts)
|
||||
- Motion/animation prompts
|
||||
|
||||
5. **Document the flow**
|
||||
- Update this document (or a sibling doc) with:
|
||||
- Model definitions
|
||||
- Where the bible is generated
|
||||
- Where and how it is injected into prompts
|
||||
|
||||
---
|
||||
|
||||
## 6. Summary of Recent Changes (Bible Wiring)
|
||||
|
||||
This section summarizes the concrete changes made for the initial Anime Story Bible integration:
|
||||
|
||||
- **Backend models**
|
||||
- Added `anime_bible` to `StoryGenerationRequest` and `StoryContinueRequest` as `Optional[Dict[str, Any]]`.
|
||||
- Confirmed `AnimeStoryBible` and related classes in [story_models.py](file:///c:/Users/diksha%20rawat/Desktop/ALwrity/backend/models/story_models.py).
|
||||
|
||||
- **Frontend models & state**
|
||||
- Added `anime_bible?: AnimeStoryBible | null` to `StoryGenerationRequest`.
|
||||
- `useStoryWriterState.getRequest()` now includes `anime_bible: state.animeBible || null`.
|
||||
- `createStoryContextPayload` for outline/animation includes `anime_bible: state.animeBible`.
|
||||
|
||||
- **Story text prompts**
|
||||
- `generate_story_start` and `continue_story` accept `anime_bible` and inject a serialized “ANIME STORY BIBLE” context block directly after `persona_prompt`.
|
||||
- Routing layer passes `request.anime_bible` through from API to service.
|
||||
|
||||
- **Motion prompts (Kling image-to-video)**
|
||||
- `story_context.anime_bible` is used in:
|
||||
- `_build_fallback_prompt` to append style/world/cast hints to deterministic prompts.
|
||||
- `generate_animation_prompt` to add an explicit `ANIME STORY BIBLE VISUAL GUIDANCE` block for the LLM.
|
||||
|
||||
- **Not yet implemented**
|
||||
- Second-pass enrichment of per-scene `image_prompt` using the bible.
|
||||
- Generalization beyond anime templates (would require broader “story bible” abstraction).
|
||||
|
||||
This document should be kept up to date whenever the Anime Story Bible is:
|
||||
|
||||
- Extended to new story templates
|
||||
- Used for additional media types (e.g., storyboard export, trailers)
|
||||
- Modified in structure or prompt integration
|
||||
|
||||
337
docs/story writer/STORY_STUDIO_SSOT.md
Normal file
337
docs/story writer/STORY_STUDIO_SSOT.md
Normal file
@@ -0,0 +1,337 @@
|
||||
# Story Studio – Single Source of Truth (SSOT)
|
||||
|
||||
## 1. Vision and Positioning
|
||||
|
||||
- **Product name**: Story Studio (evolution of Story Writer).
|
||||
- **What it is**: A narrative engine plus lightweight movie studio that turns brand and product context into structured stories, scenes, and multimedia assets.
|
||||
- **Role in ALwrity**: One of several “surfaces” driven by the same contextual brain (onboarding data, personas, SEO, competitive analysis), alongside Blog Writer, YouTube Creator, Podcast Maker, and Video Studio.
|
||||
|
||||
### Core positioning
|
||||
|
||||
- **Campaign Story Engine** (primary):
|
||||
- Generates product stories, brand manifestos, founder narratives, and customer journey stories.
|
||||
- Uses onboarding + persona + SEO context so stories are on-brand and strategically aligned.
|
||||
- **Story Studio / Fiction mode** (secondary):
|
||||
- High-quality freeform stories (including anime / fantasy), primarily for GSC traffic and creative users.
|
||||
- Lives “one click deeper” in the UI so marketing-first use cases stay front and center.
|
||||
|
||||
Story Studio is not just “a novelist playground”; it is the narrative layer that can feed:
|
||||
- YouTube Creator (scripts and scenes),
|
||||
- Podcast Maker (episodic story arcs),
|
||||
- Video Studio (scene-wise video generation),
|
||||
- Blog and social content (campaign narratives).
|
||||
|
||||
---
|
||||
|
||||
## 2. Modes and Templates
|
||||
|
||||
### 2.1 Modes
|
||||
|
||||
- **Marketing Narratives (default)**:
|
||||
- Entry path for ALwrity users arriving from the main app.
|
||||
- Story types:
|
||||
- Product Story (feature-driven but narrative-first).
|
||||
- Brand Manifesto / “Why we exist”.
|
||||
- Founder Story / Mission Arc.
|
||||
- Customer Case Story (problem → journey → outcome).
|
||||
- All marketing modes must:
|
||||
- Pull from onboarding canonical_profile (brand, product, audience, pain points).
|
||||
- Respect persona tone and style.
|
||||
- Optionally weave in SEO pillars and competitor differentiation language.
|
||||
|
||||
- **Pure Story Mode (creative)**:
|
||||
- Entry path for GSC traffic and direct Story Studio users.
|
||||
- Story types:
|
||||
- Freeform fiction.
|
||||
- Anime / manga stories.
|
||||
- Experimental narrative formats.
|
||||
- Marketing alignment:
|
||||
- Gently surfaces the idea: “Turn this into a video / campaign” via export options.
|
||||
|
||||
### 2.2 Template system
|
||||
|
||||
For each template, define:
|
||||
- Required fields: goal, target audience, setting, length, POV, style, tone.
|
||||
- Optional fields: persona to anchor voice, product/context attachments.
|
||||
- Output shape:
|
||||
- Premise (1–2 sentences).
|
||||
- Outline (scenes with structured metadata).
|
||||
- Story text (segments).
|
||||
- Multimedia plan (per scene: image prompt, audio cue, video emphasis).
|
||||
|
||||
Initial templates:
|
||||
- **Product Story**:
|
||||
- Uses onboarding product data (features, benefits, differentiators).
|
||||
- Structure: origin → tension → solution → proof → future.
|
||||
- **Brand Manifesto**:
|
||||
- Uses canonical_profile (mission, values, audience).
|
||||
- Structure: problem with status quo → belief → promise → invitation.
|
||||
- **Founder Story**:
|
||||
- Uses founder/persona context if available, otherwise guided questionnaire.
|
||||
- Structure: “before” life → trigger moment → struggle → insight → creation of the product.
|
||||
- **Customer Case Story**:
|
||||
- Uses ICP + persona + pain-point data.
|
||||
- Structure: “Customer X had problem Y” → discovery of solution → implementation → quantified outcome.
|
||||
- **Anime / Storyverse**:
|
||||
- Style, visuals, and multimedia prompts oriented toward anime / stylized output.
|
||||
|
||||
---
|
||||
|
||||
## 3. Context Inputs (Data and Signals)
|
||||
|
||||
Story Studio should treat ALwrity’s context as first-class inputs:
|
||||
|
||||
- **Onboarding / canonical_profile**:
|
||||
- brand_pitch, positioning, audience, products/services, content pillars.
|
||||
- **Personas**:
|
||||
- core persona plus platform adaptations (tone, structure, constraints).
|
||||
- **SEO and competitors**:
|
||||
- core keywords, sitemap analysis, competitive positioning and gaps.
|
||||
- **User preferences**:
|
||||
- story length, style, tone, POV, “safe vs experimental”, multimedia intensity.
|
||||
|
||||
Implementation:
|
||||
- Extend Story Setup to optionally show:
|
||||
- “Use my brand & product data” toggle.
|
||||
- Dropdown/select for which persona to write from.
|
||||
- Pre-filled prompts using SEO pillars and competitor gaps.
|
||||
- State builder (`getRequest()` in useStoryWriterState) should:
|
||||
- Attach canonical_profile / persona IDs or snapshots when marketing templates are used.
|
||||
|
||||
---
|
||||
|
||||
## 4. Text Model Strategy
|
||||
|
||||
### 4.1 Roles
|
||||
|
||||
- **Structured outline + scenes**:
|
||||
- Use `llm_text_gen` with `json_struct` for:
|
||||
- Scene lists (scene_number, title, description).
|
||||
- Per-scene image_prompt, audio_narration, character_descriptions, key_events.
|
||||
- Requirements:
|
||||
- High reliability on JSON output.
|
||||
- Good adherence to scene progression instructions.
|
||||
|
||||
- **Narrative generation (story text)**:
|
||||
- Use `llm_text_gen` in text mode for:
|
||||
- Premise, story start, continuation until word-count targets.
|
||||
- Requirements:
|
||||
- Style controllability (tone, genre, age group).
|
||||
- Long-context coherence for medium/long stories.
|
||||
|
||||
### 4.2 Providers and models
|
||||
|
||||
Use the existing provider abstraction:
|
||||
|
||||
- **OSS/primary**: HF-backed models (Llama 3.1 8B, Mistral 7B, Qwen2.5) for cost-effective long-form and experimental modes.
|
||||
|
||||
Decisions:
|
||||
- Keep model selection internal for now (no UI dropdown).
|
||||
- Introduce a simple “Quality vs Speed vs Cost” preset internally mapped to:
|
||||
- High quality: Gemini or best OSS model.
|
||||
- Balanced: OSS instruct model (e.g., Llama 3.1 8B, Mistral 7B).
|
||||
- Experimental: more creative / higher-temperature models.
|
||||
|
||||
Story length control remains as defined in story-writer-architecture.mdc:
|
||||
- Short (~1k words): one-shot.
|
||||
- Medium (<5k): multi-step.
|
||||
- Long (~10k): multi-step with IAMDONE marker.
|
||||
|
||||
---
|
||||
|
||||
## 5. Multimedia Model Strategy
|
||||
|
||||
Multimedia generation must reuse existing studio infrastructure rather than invent new stacks.
|
||||
|
||||
### 5.1 Images
|
||||
|
||||
- Use StoryImageGenerationService as primary:
|
||||
- Provider: stability / other configured HF/Gemini image models.
|
||||
- Inputs: per-scene `image_prompt` derived from outline and template.
|
||||
- Templates:
|
||||
- Realistic marketing visuals (product-focused).
|
||||
- Anime / stylized visuals (for anime mode).
|
||||
|
||||
### 5.2 Audio
|
||||
|
||||
- Use StoryAudioGenerationService:
|
||||
- Providers: gTTS, pyttsx3, or any configured TTS provider.
|
||||
- Inputs: `audio_narration` per scene (or compressed scene text).
|
||||
- For campaign stories:
|
||||
- Voice selection aligned with brand persona (calm, energetic, authoritative).
|
||||
|
||||
### 5.3 Video
|
||||
|
||||
- Use StoryVideoGenerationService for stitching scenes:
|
||||
- Inputs:
|
||||
- `image_urls` or `video_urls` per scene.
|
||||
- `audio_urls` per scene.
|
||||
- `fps`, `transition_duration`.
|
||||
- For advanced/hero scenes, optionally integrate Video Studio models:
|
||||
- text-to-video models (Hunyuan, LTX-2) for key moments.
|
||||
- image-to-video (WAN 2.5, Kandinsky 5 Pro) for stylized sequences.
|
||||
|
||||
### 5.4 Modes
|
||||
|
||||
- **Standard story video**:
|
||||
- Use existing story video pipeline.
|
||||
- Emphasis on timeline, readability, and voiceover.
|
||||
- **Anime / stylized video**:
|
||||
- Switch scene prompts + model selection to anime-friendly setups.
|
||||
- Use more dynamic transitions for “story trailer” feel.
|
||||
|
||||
---
|
||||
|
||||
## 6. Phase Flow (Reframed for Story Studio)
|
||||
|
||||
Core phases remain as in story-writer-architecture.mdc but with clarified responsibilities:
|
||||
|
||||
1. **Setup**:
|
||||
- Choose mode: Marketing Narrative vs Pure Story.
|
||||
- Select template (Product Story, Brand Manifesto, etc.).
|
||||
- Toggle/use brand context (onboarding, personas, SEO).
|
||||
- Configure image/audio/video settings.
|
||||
|
||||
2. **Outline**:
|
||||
- Generate structured outline with scenes via `json_struct`.
|
||||
- For marketing templates:
|
||||
- Ensure each scene maps to a stage in the campaign story arc.
|
||||
- For anime/fiction:
|
||||
- Emphasize “world, characters, conflicts” and visual richness.
|
||||
|
||||
3. **Writing**:
|
||||
- Generate story text based on length settings.
|
||||
- Enforce length controls and completion detection.
|
||||
- Show word count and target in UI.
|
||||
|
||||
4. **Multimedia / Export**:
|
||||
- Generate scene images, audio, and video using configured settings.
|
||||
- Allow:
|
||||
- Export as story video (using StoryVideoGenerationService).
|
||||
- Export scenes and script to:
|
||||
- YouTube Creator (for adaptation to YouTube-specific script + scenes).
|
||||
- Video Studio (for advanced editing or alternative visual styles).
|
||||
|
||||
5. **Reset**:
|
||||
- Keep existing reset semantics (clears state, localStorage, phases).
|
||||
|
||||
---
|
||||
|
||||
## 7. Implementation Plan
|
||||
|
||||
### Phase 1 – Positioning and UX tweaks
|
||||
|
||||
- Rename surface in UI copy to “Story Studio” where appropriate (keep routes/IDs stable to preserve GSC).
|
||||
- Add mode selector in Setup:
|
||||
- Marketing Narratives (default).
|
||||
- Pure Story.
|
||||
- Add template selector for marketing mode:
|
||||
- Product Story, Brand Manifesto, Founder Story, Customer Story.
|
||||
- Adjust landing screen messaging to emphasize:
|
||||
- “Turn your brand and product into narrative campaigns.”
|
||||
|
||||
### Phase 2 – Context integration
|
||||
|
||||
- Extend Story Setup state to store:
|
||||
- `useBrandContext` flag.
|
||||
- `selectedPersonaId` (optional).
|
||||
- `selectedProductId` or product context stub.
|
||||
- Update `getRequest()` to include:
|
||||
- References or snapshots from canonical_profile (where available).
|
||||
- Update backend story_service to:
|
||||
- Accept and pass context into prompts (premise, outline, story start, continuation).
|
||||
|
||||
### Phase 3 – Template-specific prompts
|
||||
|
||||
- Design prompt templates per marketing template:
|
||||
- Product Story, Brand Manifesto, Founder Story, Customer Story.
|
||||
- Encode:
|
||||
- Clear arcs (setup → conflict → resolution).
|
||||
- Use of brand and persona parameters.
|
||||
- Explicit constraints on tone, length, POV.
|
||||
- For pure story mode:
|
||||
- Provide genre/style toggles, including anime.
|
||||
|
||||
### Phase 4 – Multimedia presets
|
||||
|
||||
- Define multimedia presets for:
|
||||
- Marketing Story (clean product visuals, muted transitions).
|
||||
- Anime Story (bold colors, stylized prompts, dynamic transitions).
|
||||
- Trailer Mode (shorter, punchier video composition).
|
||||
- Map presets to:
|
||||
- Image generation prompt scaffolds.
|
||||
- Video settings (fps, transition_duration).
|
||||
- Potential future mapping into Video Studio model choices.
|
||||
|
||||
### Phase 5 – Cross-surface exports
|
||||
|
||||
- Implement export from Story Studio to:
|
||||
- YouTube Creator:
|
||||
- Export scenes as a JSON payload that can seed plan/scenes.
|
||||
- Video Studio:
|
||||
- Export scenes with associated image/audio references to pre-populate a project.
|
||||
- Add UI affordances:
|
||||
- “Send to YouTube Creator”.
|
||||
- “Open in Video Studio”.
|
||||
|
||||
### Phase 6 – Hardening and metrics
|
||||
|
||||
- Ensure:
|
||||
- Subscription / usage checks per story generation and multimedia call.
|
||||
- Caching behavior for repeated outline/story generations when context unchanged.
|
||||
- Task management and polling remain consistent with existing story_writer architecture.
|
||||
- Add metrics:
|
||||
- How many Story Studio sessions start in marketing vs pure modes.
|
||||
- How many flows export to YouTube/Video Studio.
|
||||
|
||||
---
|
||||
|
||||
## 9. Current Implementation Status (Feb 2026)
|
||||
|
||||
- Modes and templates:
|
||||
- Marketing vs Pure Story modes are wired through Story Setup state and backend prompts.
|
||||
- Template identifiers (e.g. `product_story`, `brand_manifesto`, `founder_story`, `customer_story`, `short_fiction`, `long_fiction`, `anime_fiction`, `experimental_fiction`) are resolved in service components for label-aware prompts.
|
||||
- Context integration:
|
||||
- Story context endpoint exposes `canonical_profile`-derived `brand_context` and `brand_assets` for Story Studio.
|
||||
- Story Setup modal can toggle use of onboarding brand persona when in marketing mode, and surfaces avatar and voice preview where available.
|
||||
- AI Setup modal:
|
||||
- Dedicated AI Setup modal drives an idea → setup flow.
|
||||
- “Enhance Story Idea” uses a dedicated backend endpoint to enhance only the freeform idea text, preserving intent and avoiding premature setup field generation.
|
||||
- “Continue to Story Setup” generates exactly three structured setup options and, on selection, pre-fills Story Setup fields (persona, setting, characters, plot, style, tone, POV, audience, rating, ending, length, premise) plus image/video/audio defaults.
|
||||
- UI includes rotating, mode-aware idea placeholders and an AI-style, animated glow frame around the story idea textarea to emphasize the primary input surface while keeping the overall theme light and readable.
|
||||
- Multimedia defaults:
|
||||
- Story Setup state carries image/video/audio settings and can accept per-option overrides from generated setups.
|
||||
- Story Outline and StoryImageGenerationModal integrate with the shared image generation stack, including provider/model selection and cost-awareness.
|
||||
- Video integrations:
|
||||
- HD video configuration section exposes provider/model dropdowns with story-based defaults, aligned with the broader Video Studio architecture.
|
||||
|
||||
---
|
||||
|
||||
## 10. Next Steps for Story Studio
|
||||
|
||||
- Deepen template-specific prompts:
|
||||
- Tighten prompt templates per marketing template so setup options encode clearer arcs and brand positioning, especially for product_story and customer_story.
|
||||
- Expand fiction/anime prompt variants to better control pacing and recurring characters across scenes.
|
||||
- Refine idea-to-setup bridge:
|
||||
- Add light-touch guidance in the AI Setup modal to help users iterate between idea enhancement and setup generation (e.g. suggested follow-up edits based on the last enhancement).
|
||||
- Capture telemetry on how often users enhance the idea before generating setups to tune prompts and thresholds.
|
||||
- Multimedia and cross-surface flows:
|
||||
- Wire Story Studio’s outline and scene structures into “Send to YouTube Creator” and “Open in Video Studio” actions, including carrying over multimedia presets.
|
||||
- Introduce preset bundles for “Standard Story”, “Anime Story”, and “Trailer Mode” that automatically adjust image/video defaults and, where available, Video Studio model choices.
|
||||
- Reliability and guardrails:
|
||||
- Add targeted monitoring around the AI Setup endpoints (idea enhancement and setup generation) for latency, error rates, and JSON validity.
|
||||
- Continue to enforce subscription and usage checks consistently across Story Studio, Blog Writer, and Video Studio so billing and limits remain unified.
|
||||
|
||||
---
|
||||
|
||||
## 8. Guardrails and Non-Goals
|
||||
|
||||
- Do not:
|
||||
- Turn Story Studio into a separate codebase; keep it within the existing Story Writer architecture and services.
|
||||
- Fragment context handling; continue to use canonical_profile and existing persona services as SSOTs.
|
||||
- Guardrails:
|
||||
- Maintain strict story length controls.
|
||||
- Keep subscription handling centralized via `triggerSubscriptionError`.
|
||||
- Continue to use structured JSON for outlines and scene metadata wherever possible.
|
||||
|
||||
This document, together with `story-writer-architecture.mdc` and the existing Story Writer implementation docs in `docs/story writer/`, serves as the SSOT for evolving Story Writer into Story Studio.
|
||||
Reference in New Issue
Block a user