MeanRevisionEA/oi_scraper/README.md

# CME OI Scraper

Python scraper that extracts Open Interest data from CME Group QuikStrike and current gold price from investing.com.

## What It Extracts

1. **OI Levels (from CME QuikStrike):**
   - Top 3 CALL strikes by OI volume (unique strikes)
   - Top 3 PUT strikes by OI volume (unique strikes)

2. **Gold Price (from investing.com):**
   - Current gold futures price (e.g., 4476.50)

## Prerequisites

- Python 3.9 or higher
- CME Group QuikStrike account (free registration at https://www.cmegroup.com)
- Windows 10/11 (for batch files) or Linux/macOS

## Quick Start

### Windows

1. **Run one-time setup:**
   ```cmd
   cd C:\Path\To\oi_scraper
   setup_env.bat
   ```

2. **Run the scraper:**
   ```cmd
   run_with_venv.bat
   ```

### Linux/macOS

1. **Setup:**
   ```bash
   cd /path/to/oi_scraper
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   playwright install chromium
   ```

2. **Run:**
   ```bash
   source venv/bin/activate
   python main.py
   ```

## Configuration

### Edit `.env` File

Copy and edit the environment file:

```cmd
copy .env.example .env
notepad .env
```

Required settings:
```env
CME_USERNAME=your_cme_username
CME_PASSWORD=your_cme_password
```

Optional settings:
```env
# Number of top strikes to export (default: 3)
TOP_N_STRIKES=3

# Run browser without window (default: false)
HEADLESS=false

# Page timeout in seconds (default: 30)
TIMEOUT_SECONDS=30

# Output CSV path
CSV_OUTPUT_PATH=./oi_data.csv

# Logging level: DEBUG, INFO, WARNING, ERROR
LOG_LEVEL=INFO
```

## Output Format

The scraper exports to `oi_data.csv`:

```csv
Type,Strike,OI
CALL,4375.0,147
CALL,4450.0,173
CALL,4500.0,176
PUT,4435.0,49
PUT,4400.0,102
PUT,4515.0,150

[Price]
FuturePrice,4467.8
```

The `[Price]` section contains the current gold futures price scraped from investing.com.

## Session Persistence

The scraper saves login sessions to `cookies.json`:

- **First run:** Logs in with credentials, saves cookies
- **Subsequent runs:** Uses saved cookies if session is valid
- **Session expired:** Automatically re-logs in and saves new cookies

This makes scheduled runs faster and reduces login attempts to CME servers.

To force a fresh login:
```cmd
del cookies.json
```

## Integration with EA

The EA reads OI data from CSV when configured:
```mql5
input ENUM_OI_SOURCE InpOISource = OI_SOURCE_CSV_FILE;
```

Copy `oi_data.csv` to your MT5 `MQL5/Files` directory:
```
C:\Users\YourUsername\AppData\Roaming\MetaQuotes\Terminal\Common\MQL5\Files\oi_data.csv
```

## Automatic Daily Scheduling

### Windows Task Scheduler

1. **Create scheduled task:**
   - Open Task Scheduler (`taskschd.msc`)
   - Click "Create Task"

2. **Configure General tab:**
   - Name: `CME OI Scraper - Daily`
   - ✅ Run whether user is logged on or not
   - ✅ Run with highest privileges

3. **Configure Triggers tab:**
   - New → On a schedule → Daily
   - Start time: 9:00 AM (or your preferred time)
   - ✅ Enabled

4. **Configure Actions tab:**
   - Action: Start a program
   - Program/script:
     ```
     C:\Path\To\oi_scraper\run_scheduled.bat
     ```
   - Start in:
     ```
     C:\Path\To\oi_scraper
     ```

5. **Click OK to save**

### Linux/macOS (cron)

```bash
# Edit crontab
crontab -e

# Add line to run every day at 9 AM
0 9 * * * cd /path/to/oi_scraper && /path/to/venv/bin/python main.py
```

## Batch Files Reference

| File | Purpose |
|------|---------|
| `setup_env.bat` | One-time setup (creates virtual environment) |
| `run_with_venv.bat` | Manual run with visible window |
| `run_scheduled.bat` | For Task Scheduler (no window, no pause) |

## Troubleshooting

### Module Not Found Errors

**Error:** `ModuleNotFoundError: No module named 'playwright'`

**Solution:**
```cmd
run_with_venv.bat
```

The virtual environment ensures all dependencies are isolated.

### Login Fails

- Verify credentials in `.env`
- Check if CME requires 2FA (manual intervention needed)
- Set `HEADLESS=false` to see browser activity
- Check screenshots: `login_failed.png`, `login_error.png`

### No Data Extracted

- Check if CME table structure changed
- Increase `TIMEOUT_SECONDS=60` in `.env`
- Check logs for errors
- Screenshot saved as `login_debug.png`

### Browser Issues

```cmd
# Reinstall Chromium
python -m playwright install chromium
```

### Session Expires Frequently

Delete cookies to force fresh login:
```cmd
del cookies.json
```

### Check Python Path Issues (Windows)

```cmd
# Check which Python is being used
where python

# Use Python launcher
py -3 main.py

# Or use the virtual environment
run_with_venv.bat
```

## Finding Product IDs

To scrape other instruments (Silver, Crude Oil, etc.):

1. Visit CME QuikStrike OI Heatmap
2. Login to your CME account
3. Select a product from the dropdown
4. The URL updates with the `pid` parameter
5. Note: This scraper is configured for Gold by default

## Notes

- Targets the OI Heatmap table structure
- Exports top N unique strikes by OI volume
- Uses session cookies for faster subsequent runs
- CME sessions typically last several days to weeks
- Virtual environment recommended to avoid Python path conflicts

## Files

```
oi_scraper/
├── main.py              # Main scraper script
├── requirements.txt     # Python dependencies
├── .env.example         # Environment template
├── .env                 # Your credentials (create from example)
├── setup_env.bat        # Windows: Create virtual environment
├── run_with_venv.bat    # Windows: Manual run
├── run_scheduled.bat    # Windows: Task Scheduler run
├── oi_data.csv          # Output file (generated)
├── cookies.json         # Session cookies (generated)
└── scraper.log          # Log file (generated)
```