kunthawat/MeanRevisionEA
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(oi): update scraper for new QuikStrike website structure

Browse Source 
- Replace direct product URL navigation with fixed heatmap URL and UI product selection
- Implement cookie validation with automatic session cleanup
- Update login flow to use SSO authentication and new form selectors
- Improve data extraction with iframe context and better table parsing
- Add multiple fallback selectors for gold price scraping
- Enhance error handling, logging, and timeout management
This commit is contained in:
Kunthawat Greethong
2026-01-06 12:16:53 +07:00
parent 28a4546cd8
commit 2e8e07ed17
5 changed files with 1411 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
oi_scraper/.env.example
View File

@@ -1,15 +1,10 @@
# CME Group QuikStrike Login Credentials
CME_USERNAME=your_username_here
CME_PASSWORD=your_password_here
CME_LOGIN_URL=https://login.cmegroup.com/sso/accountstatus/showAuth.action
# Product Configuration
# Gold (XAUUSD/COMEX Gold - OG|GC): pid=40
# Default product for XAUUSD trading
PRODUCT_URL=https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=40&viewitemid=IntegratedOpenInterestTool
# Alternative products:
# SOFR (3M SOFR): https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=476&viewitemid=IntegratedOpenInterestTool
# Silver: https://cmegroup.quikstrike.net/User/QuikStrikeView.aspx?pid=41&viewitemid=IntegratedOpenInterestTool
# QuikStrike URL (fixed - always same page)
QUIKSTRIKE_URL=https://www.cmegroup.com/tools-information/quikstrike/open-interest-heatmap.html
# Gold Price Source (investing.com)
INVESTING_URL=https://www.investing.com/commodities/gold