28a4546cd8
add new oi_scraper directory for collecting open interest data and update the main EA to integrate with the scraper functionality
178 lines
4.7 KiB
Markdown
178 lines
4.7 KiB
Markdown
# CME OI Scraper
|
|
|
|
Python scraper to pull Open Interest data from CME Group QuikStrike and current gold price from investing.com.
|
|
|
|
## What It Extracts
|
|
|
|
1. **OI Levels (from CME QuikStrike):**
|
|
- Top 3 CALL strikes by OI volume
|
|
- Top 3 PUT strikes by OI volume
|
|
|
|
2. **Gold Price (from investing.com):**
|
|
- Current gold futures price (e.g., 4345.50)
|
|
|
|
## Prerequisites
|
|
|
|
- Python 3.9+
|
|
- CME Group QuikStrike account with login credentials
|
|
|
|
## Installation
|
|
|
|
1. Copy environment variables:
|
|
```bash
|
|
cp .env.example .env
|
|
```
|
|
|
|
2. Edit `.env` and add your CME credentials:
|
|
```bash
|
|
CME_USERNAME=your_username
|
|
CME_PASSWORD=your_password
|
|
```
|
|
|
|
3. Install dependencies:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
playwright install chromium
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Basic Scraping
|
|
|
|
```bash
|
|
python main.py
|
|
```
|
|
|
|
This will:
|
|
- Login to CME QuikStrike
|
|
- Navigate to OI Heatmap
|
|
- Extract top 3 CALL and PUT strikes by OI volume
|
|
- Scrape current gold price from investing.com
|
|
- Export to `oi_data.csv`
|
|
|
|
### Session Persistence
|
|
|
|
The scraper automatically saves your login session to `cookies.json`. This means:
|
|
|
|
- **First run**: Logs in with your credentials, saves cookies
|
|
- **Subsequent runs**: Uses saved cookies if session is still valid
|
|
- **Session expired**: Automatically logs in again and saves new cookies
|
|
|
|
Benefits for scheduled runs:
|
|
- Faster execution (skips login when session is valid)
|
|
- Reduces login attempts to CME servers
|
|
- CME sessions typically last several days/weeks
|
|
|
|
To force a fresh login, delete `cookies.json`:
|
|
```bash
|
|
rm cookies.json
|
|
```
|
|
|
|
### Output Format
|
|
|
|
The CSV output is compatible with the EA's `LoadOIFromCSV()` and `LoadFuturePriceFromCSV()` functions:
|
|
|
|
```csv
|
|
Type,Strike,OI
|
|
CALL,4345,155398
|
|
CALL,4350,229137
|
|
CALL,4360,90649
|
|
PUT,4300,227936
|
|
PUT,4290,270135
|
|
PUT,4280,65839
|
|
|
|
[Price]
|
|
FuturePrice,4345.50
|
|
```
|
|
|
|
**Note:** The `[Price]` section contains the current gold futures price scraped from investing.com. The EA reads this value for Delta calculation.
|
|
|
|
## Configuration
|
|
|
|
Edit `.env` to customize:
|
|
|
|
- `PRODUCT_URL` - QuikStrike product page URL (requires login)
|
|
- `TOP_N_STRIKES` - Number of top strikes to export (default: 3)
|
|
- `HEADLESS` - Run browser in headless mode (default: false for debugging)
|
|
- `CSV_OUTPUT_PATH` - Output CSV file path
|
|
- `TIMEOUT_SECONDS` - Page load timeout
|
|
|
|
### Available Products
|
|
|
|
**Gold (XAUUSD/COMEX Gold - OG|GC):**
|
|
```
|
|
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool
|
|
```
|
|
|
|
**Silver:**
|
|
```
|
|
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=41&viewitemid=IntegratedOpenInterestTool
|
|
```
|
|
|
|
**SOFR (3M SOFR):**
|
|
```
|
|
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=476&viewitemid=IntegratedOpenInterestTool
|
|
```
|
|
|
|
**Note:** You must be logged in to access QuikStrike data. The scraper will automatically login using credentials from `.env`.
|
|
|
|
## Integration with EA
|
|
|
|
The EA reads OI data from CSV when `InpOISource = OI_SOURCE_CSV_FILE`.
|
|
|
|
Place the generated `oi_data.csv` in MetaTrader's `MQL5/Files` directory.
|
|
|
|
## Scheduling
|
|
|
|
Use cron or Windows Task Scheduler to run periodically:
|
|
|
|
```bash
|
|
# Run every hour
|
|
0 * * * * cd /path/to/oi_scraper && python main.py
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
**Login fails:**
|
|
- Verify credentials in `.env`
|
|
- Check if CME requires 2FA
|
|
- Set `HEADLESS=false` to see what's happening
|
|
- Check screenshots: `login_failed.png`, `login_error.png`, `login_success.png`
|
|
|
|
**No data extracted:**
|
|
- Check if table structure changed
|
|
- Increase `TIMEOUT_SECONDS`
|
|
- Check logs for detailed errors
|
|
- Screenshot saved as `login_debug.png` or `login_failed.png`
|
|
|
|
**Login page selectors changed:**
|
|
- If the scraper can't find username/password inputs, CME may have updated their login page
|
|
- Update the selectors in `login_to_cme()` function in `main.py`:
|
|
```python
|
|
# Example: update to match current CME login form
|
|
page.fill('input[id="username"]', CME_USERNAME)
|
|
page.fill('input[id="password"]', CME_PASSWORD)
|
|
page.click('button[type="submit"]')
|
|
```
|
|
|
|
**Browser issues:**
|
|
- Install Chromium dependencies: `playwright install chromium`
|
|
- Try different browser: Change `p.chromium.launch()` to `p.firefox.launch()`
|
|
|
|
## Notes
|
|
|
|
- The scraper targets the OI Heatmap table structure
|
|
- Only exports top N strikes by OI volume
|
|
- Login session is not persisted (login each run)
|
|
- Cookies could be saved for faster subsequent runs
|
|
|
|
### Finding Product IDs
|
|
|
|
To find product IDs for other instruments:
|
|
1. Visit https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html
|
|
2. Login to your CME account
|
|
3. Select a product from the "Products" menu
|
|
4. The URL will update with the `pid` parameter
|
|
5. Copy that URL to your `.env` file
|
|
|
|
Example: `https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html?pid=40` (Gold) |