MeanRevisionEA/oi_scraper/WINDOWS_SETUP.md

CME OI Scraper - Windows Setup Guide

Complete guide for setting up and running the CME OI scraper on Windows with automatic daily updates.

Table of Contents

• Prerequisites
• Installation
• Configuration
• Manual Testing
• Automatic Daily Updates
• MetaTrader 5 Integration
• Troubleshooting

---

Prerequisites

Required Software

1. Python 3.9 or higher

   • Download: https://www.python.org/downloads/
   • During installation: ✅ Check "Add Python to PATH"

2. CME Group QuikStrike Account

   • Free account required: https://www.cmegroup.com/
   • Register for QuikStrike access
   • Save your username and password

3. MetaTrader 5 (for EA integration)

   • Download: https://www.metatrader5.com/
   • Install on your Windows machine

Verify Python Installation

python --version

Expected output: Python 3.9.x or higher

If not found, install Python or use py or python3 commands.

---

Installation

Step 1: Navigate to Scraper Directory

Open Command Prompt (cmd) and navigate:

cd C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper

Replace YourUsername with your actual Windows username.

Step 2: Create Environment File

copy .env.example .env

Step 3: Edit .env File

Open .env with Notepad:

notepad .env

Update with your credentials:

# CME Group QuikStrike Login Credentials
CME_USERNAME=your_actual_username_here
CME_PASSWORD=your_actual_password_here

# Product Configuration (Gold)
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool

# Output Settings
CSV_OUTPUT_PATH=./oi_data.csv
TOP_N_STRIKES=3

# Scraping Settings
HEADLESS=false
TIMEOUT_SECONDS=30
RETRY_ATTEMPTS=3

# Logging
LOG_LEVEL=INFO

Save and close (Ctrl+S, then Alt+F4).

Step 4: Install Python Dependencies

pip install -r requirements.txt

Expected output: Successfully installed playwright, python-dotenv, pandas

Step 5: Install Playwright Browser

playwright install chromium

Expected output: Downloading Chromium... [progress bar]

---

Configuration

Available Products

Gold (XAUUSD/COMEX Gold - OG|GC):

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool

Silver:

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=41&viewitemid=IntegratedOpenInterestTool

SOFR (3M SOFR):

PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=476&viewitemid=IntegratedOpenInterestTool

Configuration Options

Setting	Description	Default
TOP_N_STRIKES	Number of top strikes to export	3
HEADLESS	Run browser without window (true/false)	false
TIMEOUT_SECONDS	Page load timeout in seconds	30
CSV_OUTPUT_PATH	Output CSV file path	./oi_data.csv
LOG_LEVEL	DEBUG, INFO, WARNING, ERROR	INFO

---

Manual Testing

Run Scraper Manually

python main.py

Expected output:

INFO:__main__:Cookies loaded from file
INFO:__main__:Using existing session (cookies)
INFO:__main__:Navigating to OI Heatmap: https://...
INFO:__main__:Extracting OI data from Gold matrix table...
INFO:__main__:Extracted 6 OI levels
INFO:__main__:Exported OI data to ./oi_data.csv

Check Output

1. Verify CSV created:

dir oi_data.csv

2. View CSV content:

notepad oi_data.csv

Expected format:

Type,Strike,OI
CALL,4400,6193
CALL,4300,3826
CALL,4350,1983
PUT,4400,5559
PUT,4300,2988
PUT,4350,1214

Check Logs

type scraper.log

Or view in Notepad:

notepad scraper.log

---

Automatic Daily Updates

Option 1: Windows Task Scheduler (Recommended)

Step 1: Create Batch File Wrapper

Create run_scraper.bat in the oi_scraper directory:

@echo off
cd /d C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper

echo Starting CME OI Scraper at %date% %time% >> scraper.log
echo ---------------------------------------- >> scraper.log

python main.py >> scraper.log 2>&1

if %ERRORLEVEL% EQU 0 (
    echo %date% %time%: Scraper completed successfully >> scraper.log
) else (
    echo %date% %time%: Scraper failed with error %ERRORLEVEL% >> scraper.log
)

echo ---------------------------------------- >> scraper.log

Replace YourUsername with your actual username.

Step 2: Open Task Scheduler

Press Win + R, type taskschd.msc, press Enter

Or: Start → Windows Administrative Tools → Task Scheduler

Step 3: Create Task

1. Click "Create Basic Task" on right sidebar
2. Name: CME OI Scraper - Daily
3. Description: Update OI data from CME QuikStrike every day at 9 AM
4. Click Next

Step 4: Set Trigger

1. Trigger: Select "Daily"
2. Start date: Today's date
3. Start time: 9:00:00 AM (or your preferred time)
4. Click Next

Step 5: Set Action

1. Action: Select "Start a program"
2. Program/script:

   C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper\run_scraper.bat
   

3. Start in (optional):

   C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper
   

4. Click Next

Step 6: Finish

1. Review settings
2. Check "Open the Properties dialog for this task when I click Finish"
3. Click Finish

Step 7: Configure Advanced Settings (Optional)

In the Properties dialog:

• General tab:

  • ✅ Run whether user is logged on or not
  • ✅ Do not store password (if using Windows authentication)
  • ✅ Run with highest privileges

• Conditions tab:

  • ✅ Start the task only if the computer is on AC power
  • ✅ Stop if the computer switches to battery power
  • ✅ Wake the computer to run this task

• Settings tab:

  • ✅ Allow task to be run on demand
  • ❌ Stop the task if it runs longer than: 30 minutes
  • ✅ If the task fails, restart every: 5 minutes (up to 3 times)

Click OK to save settings.

Step 8: Test Task

1. In Task Scheduler, find "CME OI Scraper - Daily"
2. Right-click → Run
3. Check scraper.log after a minute:

   type scraper.log
   

---

Option 2: PowerShell Script (Advanced)

Step 1: Create PowerShell Script

Save as run_scraper.ps1:

# Script configuration
$scriptPath = "C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper"
$logFile = "$scriptPath\scraper.log"
$timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

# Navigate to script directory
cd $scriptPath

try {
    # Run Python scraper
    Write-Output "$timestamp: Starting CME OI Scraper" | Add-Content $logFile
    & python main.py *>> $logFile 2>&1
    
    # Check if CSV was created
    if (Test-Path "oi_data.csv") {
        $fileInfo = Get-Item "oi_data.csv"
        Write-Output "$timestamp: Scraper completed successfully (CSV updated: $($fileInfo.LastWriteTime))" | Add-Content $logFile
    } else {
        Write-Output "$timestamp: WARNING - CSV file not created" | Add-Content $logFile
    }
} catch {
    $errorMsg = $_.Exception.Message
    Write-Output "$timestamp: ERROR - $errorMsg" | Add-Content $logFile
    exit 1
}

Step 2: Update Task Scheduler to Use PowerShell

Same steps as Option 1, but:

• Program/script: powershell.exe
• Add arguments:

  -ExecutionPolicy Bypass -File "C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper\run_scraper.ps1"
  

---

MetaTrader 5 Integration

Find MT5 Files Directory

MT5 data directory location:

C:\Users\YourUsername\AppData\Roaming\MetaQuotes\Terminal\[Terminal_ID]\MQL5\Files\

To find your Terminal_ID:

1. Open MT5
2. Click File → Open Data Folder
3. Navigate to Terminal\[Your_Terminal_ID]\MQL5\Files\

Update Batch File to Copy to MT5

Edit run_scraper.bat:

@echo off
cd /d C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper

echo Starting CME OI Scraper at %date% %time% >> scraper.log
echo ---------------------------------------- >> scraper.log

python main.py >> scraper.log 2>&1

if %ERRORLEVEL% EQU 0 (
    if exist oi_data.csv (
        echo Copying OI data to MT5... >> scraper.log
        copy oi_data.csv "C:\Users\YourUsername\AppData\Roaming\MetaQuotes\Terminal\[Your_Terminal_ID]\MQL5\Files\oi_data.csv"
        
        if %ERRORLEVEL% EQU 0 (
            echo %date% %time%: Scraper completed - OI data copied to MT5 >> scraper.log
        ) else (
            echo %date% %time%: ERROR - Failed to copy to MT5 >> scraper.log
        )
    ) else (
        echo %date% %time%: ERROR - oi_data.csv not found >> scraper.log
    )
) else (
    echo %date% %time%: ERROR - Scraper failed with error %ERRORLEVEL% >> scraper.log
)

echo ---------------------------------------- >> scraper.log

Replace [Your_Terminal_ID] with your actual MT5 terminal ID.

Update EA Configuration

In your EA (OI_MeanReversion_Pro_XAUUSD_A.mq5), set:

input ENUM_OI_SOURCE InpOISource = OI_SOURCE_CSV_FILE;  // Load from CSV file

The EA will automatically read oi_data.csv from its Files directory.

---

Troubleshooting

Python Not Found

Error: 'python' is not recognized as an internal or external command

Solutions:

1. Use full path to Python:

   C:\Users\YourUsername\AppData\Local\Programs\Python\Python312\python.exe main.py
   

2. Use py launcher:

   py main.py
   

3. Reinstall Python with "Add to PATH" option

Module Import Errors

Error: ModuleNotFoundError: No module named 'playwright'

Solution:

pip install -r requirements.txt

Login Fails

Error: Login failed - still on login page

Solutions:

1. Check credentials in .env file:

   notepad .env
   

2. Check login screenshots:

   • login_failed.png - Shows login page
   • login_error.png - Shows error during login
   • login_success.png - Confirms successful login

3. Manually test login at: https://www.cmegroup.com/

4. Check if 2FA is required (CME may require additional authentication)

No Data Extracted

Warning: No CALL OI data extracted or No PUT OI data extracted

Solutions:

1. Check if you're logged in:

   • Delete cookies.json to force fresh login
   • Run scraper manually with HEADLESS=false in .env

2. Check if page structure changed:

   • View screenshots to see actual page content
   • Check if Gold product URL is correct

3. Increase timeout:

   TIMEOUT_SECONDS=60
   

Task Not Running

Issue: Task Scheduler doesn't execute the task

Solutions:

1. Check task history:

   • Task Scheduler → Right-click task → Properties → History tab
   • Look for errors in the log

2. Test manually:

   • Right-click task → Run
   • Check scraper.log for output

3. Check account permissions:

   • Ensure task is set to run with your Windows account
   • Check "Run whether user is logged on or not"

4. Check Windows Event Viewer:

   • Event Viewer → Windows Logs → Application
   • Look for Task Scheduler errors

Session Expiration

Issue: Session expires after some time

Solution: The scraper will automatically re-login when cookies expire. No manual action needed.

To force fresh login:

del cookies.json

Check Logs

View recent logs:

type scraper.log | more

View last 20 lines:

powershell "Get-Content scraper.log -Tail 20"

Search for errors:

findstr /C:"ERROR" scraper.log

Verify CSV Output

Check if CSV is valid:

python -c "import pandas as pd; print(pd.read_csv('oi_data.csv'))"

Check file size:

dir oi_data.csv

---

Advanced Options

Run Multiple Times Per Day

Edit Task Scheduler Trigger:

1. Open task properties → Triggers tab
2. Edit existing trigger → Click "New" to add additional
3. Set different times:
   • 9:00 AM
   • 12:00 PM
   • 3:00 PM
   • 6:00 PM

Run on Market Days Only

Create separate batch file:

@echo off
cd /d C:\Users\YourUsername\Gitea\MeanRevisionEA\oi_scraper

REM Check if today is weekday (1=Monday, 5=Friday)
for /f "skip=1 tokens=*" %%a in ('wmic path win32_localtime get dayofweek /value') do set DAY=%%a

if %DAY% LSS 1 goto END
if %DAY% GTR 5 goto END

REM Run scraper
python main.py >> scraper.log 2>&1

:END

Email Notifications

Use PowerShell to send email on completion:

# Add to run_scraper.ps1 at the end
$smtpServer = "smtp.gmail.com"
$smtpPort = 587
$smtpUser = "your_email@gmail.com"
$smtpPass = "your_password"
$from = "CME OI Scraper <your_email@gmail.com>"
$to = "your_email@gmail.com"
$subject = "CME OI Scraper - %date%"

if ($errorOccurred) {
    $body = "CME OI Scraper failed. Check logs for details."
} else {
    $body = "CME OI Scraper completed successfully.`n`nUpdated files:`n- oi_data.csv"
}

$message = New-Object System.Net.Mail.MailMessage $from, $to
$message.Subject = $subject
$message.Body = $body

$smtp = New-Object System.Net.Mail.SmtpClient $smtpServer, $smtpPort
$smtp.EnableSsl = $true
$smtp.Credentials = New-Object System.Net.NetworkCredential $smtpUser, $smtpPass

$smtp.Send($message)

---

Summary

Quick Start Checklist:

• Python 3.9+ installed
• CME QuikStrike account created
• .env file configured with credentials
• Dependencies installed (pip install -r requirements.txt)
• Playwright browser installed (playwright install chromium)
• Manual test successful (python main.py)
• oi_data.csv created and valid
• Task Scheduler task created
• Task tested manually
• CSV copied to MT5 Files directory
• EA configured to use CSV file

Daily Workflow:

1. Task Scheduler runs at 9:00 AM
2. Batch file executes Python scraper
3. Scraper logs in with saved cookies (or fresh login)
4. OI data extracted and saved to oi_data.csv
5. CSV copied to MT5 Files directory
6. EA reads updated OI data
7. EA uses new OI levels for trading

---

Support

For issues or questions:

1. Check scraper.log for detailed error messages
2. Review screenshots (login_failed.png, login_error.png)
3. Verify .env configuration
4. Test manually without Task Scheduler
5. Check Windows Event Viewer for system errors

---

Last Updated: January 4, 2026 Version: 1.0 Platform: Windows 10/11