MeanRevisionEA/oi_scraper/README.md

CME OI Scraper

Python scraper to pull Open Interest data from CME Group QuikStrike and current gold price from investing.com.

What It Extracts

1. OI Levels (from CME QuikStrike):

   • Top 3 CALL strikes by OI volume
   • Top 3 PUT strikes by OI volume

2. Gold Price (from investing.com):

   • Current gold futures price (e.g., 4345.50)

Prerequisites

• Python 3.9+
• CME Group QuikStrike account with login credentials

Installation

1. Copy environment variables:

cp .env.example .env

2. Edit .env and add your CME credentials:

CME_USERNAME=your_username
CME_PASSWORD=your_password

3. Install dependencies:

pip install -r requirements.txt
playwright install chromium

Usage

Basic Scraping

python main.py

This will:

• Login to CME QuikStrike
• Navigate to OI Heatmap
• Extract top 3 CALL and PUT strikes by OI volume
• Scrape current gold price from investing.com
• Export to oi_data.csv

Session Persistence

The scraper automatically saves your login session to cookies.json. This means:

• First run: Logs in with your credentials, saves cookies
• Subsequent runs: Uses saved cookies if session is still valid
• Session expired: Automatically logs in again and saves new cookies

Benefits for scheduled runs:

• Faster execution (skips login when session is valid)
• Reduces login attempts to CME servers
• CME sessions typically last several days/weeks

To force a fresh login, delete cookies.json:

rm cookies.json

Output Format

The CSV output is compatible with the EA's LoadOIFromCSV() and LoadFuturePriceFromCSV() functions:

Type,Strike,OI
CALL,4345,155398
CALL,4350,229137
CALL,4360,90649
PUT,4300,227936
PUT,4290,270135
PUT,4280,65839

[Price]
FuturePrice,4345.50

Note: The [Price] section contains the current gold futures price scraped from investing.com. The EA reads this value for Delta calculation.

Configuration

Edit .env to customize:

• PRODUCT_URL - QuikStrike product page URL (requires login)
• CME_LOGIN_URL - CME login page URL (default: SSO URL)
• TOP_N_STRIKES - Number of top strikes to export (default: 3)
• HEADLESS - Run browser in headless mode (default: false for debugging)
• CSV_OUTPUT_PATH - Output CSV file path
• TIMEOUT_SECONDS - Page load timeout

Available Products

Gold (XAUUSD/COMEX Gold - OG|GC):

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool

Silver:

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=41&viewitemid=IntegratedOpenInterestTool

SOFR (3M SOFR):

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=476&viewitemid=IntegratedOpenInterestTool

Note: You must be logged in to access QuikStrike data. The scraper will automatically login using credentials from .env.

Integration with EA

The EA reads OI data from CSV when InpOISource = OI_SOURCE_CSV_FILE.

Place the generated oi_data.csv in MetaTrader's MQL5/Files directory.

Scheduling

Use cron or Windows Task Scheduler to run periodically:

# Run every hour
0 * * * * cd /path/to/oi_scraper && python main.py

Troubleshooting

Login fails:

• Verify credentials in .env
• Check if CME requires 2FA
• Set HEADLESS=false to see what's happening
• Check screenshots: login_failed.png, login_error.png, login_success.png

No data extracted:

• Check if table structure changed
• Increase TIMEOUT_SECONDS
• Check logs for detailed errors
• Screenshot saved as login_debug.png or login_failed.png

Login page selectors changed:

• If the scraper can't find username/password inputs, CME may have updated their login page
• Update the selectors in login_to_cme() function in main.py:

  # Example: update to match current CME login form
  page.fill('input[id="username"]', CME_USERNAME)
  page.fill('input[id="password"]', CME_PASSWORD)
  page.click('button[type="submit"]')
  

Browser issues:

• Install Chromium dependencies: playwright install chromium
• Try different browser: Change p.chromium.launch() to p.firefox.launch()

Notes

• The scraper targets the OI Heatmap table structure
• Only exports top N strikes by OI volume
• Login session is not persisted (login each run)
• Cookies could be saved for faster subsequent runs

Finding Product IDs

To find product IDs for other instruments:

1. Visit https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html
2. Login to your CME account
3. Select a product from the "Products" menu
4. The URL will update with the pid parameter
5. Copy that URL to your .env file

Example: https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html?pid=40 (Gold)