# CME OI Scraper

Python scraper to pull Open Interest data from CME Group QuikStrike and current gold price from investing.com.

## What It Extracts

1. **OI Levels (from CME QuikStrike):**
   - Top 3 CALL strikes by OI volume
   - Top 3 PUT strikes by OI volume

2. **Gold Price (from investing.com):**
   - Current gold futures price (e.g., 4345.50)

## Prerequisites

- Python 3.9+
- CME Group QuikStrike account with login credentials

## Installation

1. Copy environment variables:
```bash
cp .env.example .env
```

2. Edit `.env` and add your CME credentials:
```bash
CME_USERNAME=your_username
CME_PASSWORD=your_password
```

3. Install dependencies:
```bash
pip install -r requirements.txt
playwright install chromium
```

## Usage

### Basic Scraping

```bash
python main.py
```

This will:
- Login to CME QuikStrike
- Navigate to OI Heatmap
- Extract top 3 CALL and PUT strikes by OI volume
- Scrape current gold price from investing.com
- Export to `oi_data.csv`

### Session Persistence

The scraper automatically saves your login session to `cookies.json`. This means:

- **First run**: Logs in with your credentials, saves cookies
- **Subsequent runs**: Uses saved cookies if session is still valid
- **Session expired**: Automatically logs in again and saves new cookies

Benefits for scheduled runs:
- Faster execution (skips login when session is valid)
- Reduces login attempts to CME servers
- CME sessions typically last several days/weeks

To force a fresh login, delete `cookies.json`:
```bash
rm cookies.json
```

### Output Format

The CSV output is compatible with the EA's `LoadOIFromCSV()` and `LoadFuturePriceFromCSV()` functions:

```csv
Type,Strike,OI
CALL,4345,155398
CALL,4350,229137
CALL,4360,90649
PUT,4300,227936
PUT,4290,270135
PUT,4280,65839

[Price]
FuturePrice,4345.50
```

**Note:** The `[Price]` section contains the current gold futures price scraped from investing.com. The EA reads this value for Delta calculation.

## Configuration

Edit `.env` to customize:

- `PRODUCT_URL` - QuikStrike product page URL (requires login)
- `TOP_N_STRIKES` - Number of top strikes to export (default: 3)
- `HEADLESS` - Run browser in headless mode (default: false for debugging)
- `CSV_OUTPUT_PATH` - Output CSV file path
- `TIMEOUT_SECONDS` - Page load timeout

### Available Products

**Gold (XAUUSD/COMEX Gold - OG|GC):**
```
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool
```

**Silver:**
```
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=41&viewitemid=IntegratedOpenInterestTool
```

**SOFR (3M SOFR):**
```
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=476&viewitemid=IntegratedOpenInterestTool
```

**Note:** You must be logged in to access QuikStrike data. The scraper will automatically login using credentials from `.env`.

## Integration with EA

The EA reads OI data from CSV when `InpOISource = OI_SOURCE_CSV_FILE`.

Place the generated `oi_data.csv` in MetaTrader's `MQL5/Files` directory.

## Scheduling

Use cron or Windows Task Scheduler to run periodically:

```bash
# Run every hour
0 * * * * cd /path/to/oi_scraper && python main.py
```

## Troubleshooting

**Login fails:**
- Verify credentials in `.env`
- Check if CME requires 2FA
- Set `HEADLESS=false` to see what's happening
- Check screenshots: `login_failed.png`, `login_error.png`, `login_success.png`

**No data extracted:**
- Check if table structure changed
- Increase `TIMEOUT_SECONDS`
- Check logs for detailed errors
- Screenshot saved as `login_debug.png` or `login_failed.png`

**Login page selectors changed:**
- If the scraper can't find username/password inputs, CME may have updated their login page
- Update the selectors in `login_to_cme()` function in `main.py`:
  ```python
  # Example: update to match current CME login form
  page.fill('input[id="username"]', CME_USERNAME)
  page.fill('input[id="password"]', CME_PASSWORD)
  page.click('button[type="submit"]')
  ```

**Browser issues:**
- Install Chromium dependencies: `playwright install chromium`
- Try different browser: Change `p.chromium.launch()` to `p.firefox.launch()`

## Notes

- The scraper targets the OI Heatmap table structure
- Only exports top N strikes by OI volume
- Login session is not persisted (login each run)
- Cookies could be saved for faster subsequent runs

### Finding Product IDs

To find product IDs for other instruments:
1. Visit https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html
2. Login to your CME account
3. Select a product from the "Products" menu
4. The URL will update with the `pid` parameter
5. Copy that URL to your `.env` file

Example: `https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html?pid=40` (Gold)