Files

Will Chen 30b5c0d0ef Replace thinking with native Gemini thinking summaries (#400 )

This uses Gemini's native [thinking
summaries](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking#thought-summaries)
which were recently added to the API.

Why? The grafted thinking would sometimes cause weird issues where the
model, especially Gemini 2.5 Flash, got confused and put dyad tags like
`<dyad-write>` inside the `<think>` tags.

This also improves the UX because you can see the native thoughts rather
than having the Gemini response load for a while without any feedback.

I tried adding Anthropic extended thinking, however it requires temp to
be set at 1, which isn't ideal for Dyad's use case where we need precise
syntax following.

2025-06-16 17:29:32 -07:00

chatCompletionHandler.ts

E2E test for fixing errors (#351 )

2025-06-05 23:29:52 -07:00

index.ts

Replace thinking with native Gemini thinking summaries (#400 )

2025-06-16 17:29:32 -07:00

package-lock.json

Implement saver mode (#154 )

2025-05-13 15:34:41 -07:00

package.json

Implement saver mode (#154 )

2025-05-13 15:34:41 -07:00

README.md

Implement saver mode (#154 )

2025-05-13 15:34:41 -07:00

tsconfig.json

Implement saver mode (#154 )

2025-05-13 15:34:41 -07:00

README.md

Fake LLM Server

A simple server that mimics the OpenAI streaming chat completions API for testing purposes.

Features

Implements a basic version of the OpenAI chat completions API
Supports both streaming and non-streaming responses
Always responds with "hello world" message
Simulates a 429 rate limit error when the last message is "[429]"
Configurable through environment variables

Installation

npm install

Usage

Start the server:

# Development mode
npm run dev

# Production mode
npm run build
npm start

Example usage

curl -X POST http://localhost:3500/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Say something"}],"model":"any-model","stream":true}'

The server will be available at http://localhost:3500 by default.

API Endpoints

POST /v1/chat/completions

This endpoint mimics OpenAI's chat completions API.

Request Format

{
  "messages": [{ "role": "user", "content": "Your prompt here" }],
  "model": "any-model",
  "stream": true
}

Set stream: true to receive a streaming response
Set stream: false or omit it for a regular JSON response

Response

For non-streaming requests, you'll get a standard JSON response:

{
  "id": "chatcmpl-123456789",
  "object": "chat.completion",
  "created": 1699000000,
  "model": "fake-model",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "hello world"
      },
      "finish_reason": "stop"
    }
  ]
}

For streaming requests, you'll receive a series of server-sent events (SSE), each containing a chunk of the response.

Simulating Rate Limit Errors

To test how your application handles rate limiting, send a message with content exactly equal to [429]:

{
  "messages": [{ "role": "user", "content": "[429]" }],
  "model": "any-model"
}

This will return a 429 status code with the following response:

{
  "error": {
    "message": "Too many requests. Please try again later.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}

Configuration

You can configure the server by modifying the PORT variable in the code.

Use Case

This server is primarily intended for testing applications that integrate with OpenAI's API, allowing you to develop and test without making actual API calls to OpenAI.