MeanRevisionEA/oi_scraper/README.md

CME OI Scraper

Python scraper that extracts Open Interest data from CME Group QuikStrike and current gold price from investing.com.

What It Extracts

1. OI Levels (from CME QuikStrike):

   • Top 3 CALL strikes by OI volume (unique strikes)
   • Top 3 PUT strikes by OI volume (unique strikes)

2. Gold Price (from investing.com):

   • Current gold futures price (e.g., 4476.50)

Prerequisites

• Python 3.9 or higher
• CME Group QuikStrike account (free registration at https://www.cmegroup.com)
• Windows 10/11 (for batch files) or Linux/macOS

Quick Start

Windows

1. Run one-time setup:

   cd C:\Path\To\oi_scraper
   setup_env.bat
   

2. Run the scraper:

   run_with_venv.bat
   

Linux/macOS

1. Setup:

   cd /path/to/oi_scraper
   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   playwright install chromium
   

2. Run:

   source venv/bin/activate
   python main.py
   

Configuration

Edit .env File

Copy and edit the environment file:

copy .env.example .env
notepad .env

Required settings:

CME_USERNAME=your_cme_username
CME_PASSWORD=your_cme_password

Optional settings:

# Number of top strikes to export (default: 3)
TOP_N_STRIKES=3

# Run browser without window (default: false)
HEADLESS=false

# Page timeout in seconds (default: 30)
TIMEOUT_SECONDS=30

# Output CSV path
CSV_OUTPUT_PATH=./oi_data.csv

# Logging level: DEBUG, INFO, WARNING, ERROR
LOG_LEVEL=INFO

Output Format

The scraper exports to oi_data.csv:

Type,Strike,OI
CALL,4375.0,147
CALL,4450.0,173
CALL,4500.0,176
PUT,4435.0,49
PUT,4400.0,102
PUT,4515.0,150

[Price]
FuturePrice,4467.8

The [Price] section contains the current gold futures price scraped from investing.com.

Session Persistence

The scraper saves login sessions to cookies.json:

• First run: Logs in with credentials, saves cookies
• Subsequent runs: Uses saved cookies if session is valid
• Session expired: Automatically re-logs in and saves new cookies

This makes scheduled runs faster and reduces login attempts to CME servers.

To force a fresh login:

del cookies.json

Integration with EA

The EA reads OI data from CSV when configured:

input ENUM_OI_SOURCE InpOISource = OI_SOURCE_CSV_FILE;

Copy oi_data.csv to your MT5 MQL5/Files directory:

C:\Users\YourUsername\AppData\Roaming\MetaQuotes\Terminal\Common\MQL5\Files\oi_data.csv

Automatic Daily Scheduling

Windows Task Scheduler

1. Create scheduled task:

   • Open Task Scheduler (taskschd.msc)
   • Click "Create Task"

2. Configure General tab:

   • Name: CME OI Scraper - Daily
   • ✅ Run whether user is logged on or not
   • ✅ Run with highest privileges

3. Configure Triggers tab:

   • New → On a schedule → Daily
   • Start time: 9:00 AM (or your preferred time)
   • ✅ Enabled

4. Configure Actions tab:

   • Action: Start a program
   • Program/script:

     C:\Path\To\oi_scraper\run_scheduled.bat
     

   • Start in:

     C:\Path\To\oi_scraper
     

5. Click OK to save

Linux/macOS (cron)

# Edit crontab
crontab -e

# Add line to run every day at 9 AM
0 9 * * * cd /path/to/oi_scraper && /path/to/venv/bin/python main.py

Batch Files Reference

File	Purpose
setup_env.bat	One-time setup (creates virtual environment)
run_with_venv.bat	Manual run with visible window
run_scheduled.bat	For Task Scheduler (no window, no pause)

Troubleshooting

Module Not Found Errors

Error: ModuleNotFoundError: No module named 'playwright'

Solution:

run_with_venv.bat

The virtual environment ensures all dependencies are isolated.

Login Fails

• Verify credentials in .env
• Check if CME requires 2FA (manual intervention needed)
• Set HEADLESS=false to see browser activity
• Check screenshots: login_failed.png, login_error.png

No Data Extracted

• Check if CME table structure changed
• Increase TIMEOUT_SECONDS=60 in .env
• Check logs for errors
• Screenshot saved as login_debug.png

Browser Issues

# Reinstall Chromium
python -m playwright install chromium

Session Expires Frequently

Delete cookies to force fresh login:

del cookies.json

Check Python Path Issues (Windows)

# Check which Python is being used
where python

# Use Python launcher
py -3 main.py

# Or use the virtual environment
run_with_venv.bat

Finding Product IDs

To scrape other instruments (Silver, Crude Oil, etc.):

1. Visit CME QuikStrike OI Heatmap
2. Login to your CME account
3. Select a product from the dropdown
4. The URL updates with the pid parameter
5. Note: This scraper is configured for Gold by default

Notes

• Targets the OI Heatmap table structure
• Exports top N unique strikes by OI volume
• Uses session cookies for faster subsequent runs
• CME sessions typically last several days to weeks
• Virtual environment recommended to avoid Python path conflicts

Files

oi_scraper/
├── main.py              # Main scraper script
├── requirements.txt     # Python dependencies
├── .env.example         # Environment template
├── .env                 # Your credentials (create from example)
├── setup_env.bat        # Windows: Create virtual environment
├── run_with_venv.bat    # Windows: Manual run
├── run_scheduled.bat    # Windows: Task Scheduler run
├── oi_data.csv          # Output file (generated)
├── cookies.json         # Session cookies (generated)
└── scraper.log          # Log file (generated)