Add files
This commit is contained in:
Kunthawat Greethong
268
OI EA/oi_scraper/README.md
Normal file
268
OI EA/oi_scraper/README.md
Normal file
@@ -0,0 +1,268 @@
|
||||
# CME OI Scraper
|
||||
|
||||
Python scraper that extracts Open Interest data from CME Group QuikStrike and current gold price from investing.com.
|
||||
|
||||
## What It Extracts
|
||||
|
||||
1. **OI Levels (from CME QuikStrike):**
|
||||
- Top 3 CALL strikes by OI volume (unique strikes)
|
||||
- Top 3 PUT strikes by OI volume (unique strikes)
|
||||
|
||||
2. **Gold Price (from investing.com):**
|
||||
- Current gold futures price (e.g., 4476.50)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.9 or higher
|
||||
- CME Group QuikStrike account (free registration at https://www.cmegroup.com)
|
||||
- Windows 10/11 (for batch files) or Linux/macOS
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Windows
|
||||
|
||||
1. **Run one-time setup:**
|
||||
```cmd
|
||||
cd C:\Path\To\oi_scraper
|
||||
setup_env.bat
|
||||
```
|
||||
|
||||
2. **Run the scraper:**
|
||||
```cmd
|
||||
run_with_venv.bat
|
||||
```
|
||||
|
||||
### Linux/macOS
|
||||
|
||||
1. **Setup:**
|
||||
```bash
|
||||
cd /path/to/oi_scraper
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
2. **Run:**
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
python main.py
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Edit `.env` File
|
||||
|
||||
Copy and edit the environment file:
|
||||
|
||||
```cmd
|
||||
copy .env.example .env
|
||||
notepad .env
|
||||
```
|
||||
|
||||
Required settings:
|
||||
```env
|
||||
CME_USERNAME=your_cme_username
|
||||
CME_PASSWORD=your_cme_password
|
||||
```
|
||||
|
||||
Optional settings:
|
||||
```env
|
||||
# Number of top strikes to export (default: 3)
|
||||
TOP_N_STRIKES=3
|
||||
|
||||
# Run browser without window (default: false)
|
||||
HEADLESS=false
|
||||
|
||||
# Page timeout in seconds (default: 30)
|
||||
TIMEOUT_SECONDS=30
|
||||
|
||||
# Output CSV path
|
||||
CSV_OUTPUT_PATH=./oi_data.csv
|
||||
|
||||
# Logging level: DEBUG, INFO, WARNING, ERROR
|
||||
LOG_LEVEL=INFO
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
The scraper exports to `oi_data.csv`:
|
||||
|
||||
```csv
|
||||
Type,Strike,OI
|
||||
CALL,4375.0,147
|
||||
CALL,4450.0,173
|
||||
CALL,4500.0,176
|
||||
PUT,4435.0,49
|
||||
PUT,4400.0,102
|
||||
PUT,4515.0,150
|
||||
|
||||
[Price]
|
||||
FuturePrice,4467.8
|
||||
```
|
||||
|
||||
The `[Price]` section contains the current gold futures price scraped from investing.com.
|
||||
|
||||
## Session Persistence
|
||||
|
||||
The scraper saves login sessions to `cookies.json`:
|
||||
|
||||
- **First run:** Logs in with credentials, saves cookies
|
||||
- **Subsequent runs:** Uses saved cookies if session is valid
|
||||
- **Session expired:** Automatically re-logs in and saves new cookies
|
||||
|
||||
This makes scheduled runs faster and reduces login attempts to CME servers.
|
||||
|
||||
To force a fresh login:
|
||||
```cmd
|
||||
del cookies.json
|
||||
```
|
||||
|
||||
## Integration with EA
|
||||
|
||||
The EA reads OI data from CSV when configured:
|
||||
```mql5
|
||||
input ENUM_OI_SOURCE InpOISource = OI_SOURCE_CSV_FILE;
|
||||
```
|
||||
|
||||
Copy `oi_data.csv` to your MT5 `MQL5/Files` directory:
|
||||
```
|
||||
C:\Users\YourUsername\AppData\Roaming\MetaQuotes\Terminal\Common\MQL5\Files\oi_data.csv
|
||||
```
|
||||
|
||||
## Automatic Daily Scheduling
|
||||
|
||||
### Windows Task Scheduler
|
||||
|
||||
1. **Create scheduled task:**
|
||||
- Open Task Scheduler (`taskschd.msc`)
|
||||
- Click "Create Task"
|
||||
|
||||
2. **Configure General tab:**
|
||||
- Name: `CME OI Scraper - Daily`
|
||||
- ✅ Run whether user is logged on or not
|
||||
- ✅ Run with highest privileges
|
||||
|
||||
3. **Configure Triggers tab:**
|
||||
- New → On a schedule → Daily
|
||||
- Start time: 9:00 AM (or your preferred time)
|
||||
- ✅ Enabled
|
||||
|
||||
4. **Configure Actions tab:**
|
||||
- Action: Start a program
|
||||
- Program/script:
|
||||
```
|
||||
C:\Path\To\oi_scraper\run_scheduled.bat
|
||||
```
|
||||
- Start in:
|
||||
```
|
||||
C:\Path\To\oi_scraper
|
||||
```
|
||||
|
||||
5. **Click OK to save**
|
||||
|
||||
### Linux/macOS (cron)
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add line to run every day at 9 AM
|
||||
0 9 * * * cd /path/to/oi_scraper && /path/to/venv/bin/python main.py
|
||||
```
|
||||
|
||||
## Batch Files Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `setup_env.bat` | One-time setup (creates virtual environment) |
|
||||
| `run_with_venv.bat` | Manual run with visible window |
|
||||
| `run_scheduled.bat` | For Task Scheduler (no window, no pause) |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Module Not Found Errors
|
||||
|
||||
**Error:** `ModuleNotFoundError: No module named 'playwright'`
|
||||
|
||||
**Solution:**
|
||||
```cmd
|
||||
run_with_venv.bat
|
||||
```
|
||||
|
||||
The virtual environment ensures all dependencies are isolated.
|
||||
|
||||
### Login Fails
|
||||
|
||||
- Verify credentials in `.env`
|
||||
- Check if CME requires 2FA (manual intervention needed)
|
||||
- Set `HEADLESS=false` to see browser activity
|
||||
- Check screenshots: `login_failed.png`, `login_error.png`
|
||||
|
||||
### No Data Extracted
|
||||
|
||||
- Check if CME table structure changed
|
||||
- Increase `TIMEOUT_SECONDS=60` in `.env`
|
||||
- Check logs for errors
|
||||
- Screenshot saved as `login_debug.png`
|
||||
|
||||
### Browser Issues
|
||||
|
||||
```cmd
|
||||
# Reinstall Chromium
|
||||
python -m playwright install chromium
|
||||
```
|
||||
|
||||
### Session Expires Frequently
|
||||
|
||||
Delete cookies to force fresh login:
|
||||
```cmd
|
||||
del cookies.json
|
||||
```
|
||||
|
||||
### Check Python Path Issues (Windows)
|
||||
|
||||
```cmd
|
||||
# Check which Python is being used
|
||||
where python
|
||||
|
||||
# Use Python launcher
|
||||
py -3 main.py
|
||||
|
||||
# Or use the virtual environment
|
||||
run_with_venv.bat
|
||||
```
|
||||
|
||||
## Finding Product IDs
|
||||
|
||||
To scrape other instruments (Silver, Crude Oil, etc.):
|
||||
|
||||
1. Visit CME QuikStrike OI Heatmap
|
||||
2. Login to your CME account
|
||||
3. Select a product from the dropdown
|
||||
4. The URL updates with the `pid` parameter
|
||||
5. Note: This scraper is configured for Gold by default
|
||||
|
||||
## Notes
|
||||
|
||||
- Targets the OI Heatmap table structure
|
||||
- Exports top N unique strikes by OI volume
|
||||
- Uses session cookies for faster subsequent runs
|
||||
- CME sessions typically last several days to weeks
|
||||
- Virtual environment recommended to avoid Python path conflicts
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
oi_scraper/
|
||||
├── main.py # Main scraper script
|
||||
├── requirements.txt # Python dependencies
|
||||
├── .env.example # Environment template
|
||||
├── .env # Your credentials (create from example)
|
||||
├── setup_env.bat # Windows: Create virtual environment
|
||||
├── run_with_venv.bat # Windows: Manual run
|
||||
├── run_scheduled.bat # Windows: Task Scheduler run
|
||||
├── oi_data.csv # Output file (generated)
|
||||
├── cookies.json # Session cookies (generated)
|
||||
└── scraper.log # Log file (generated)
|
||||
```
|
||||
Reference in New Issue
Block a user