Selenium Scraper Browser: Headless Chrome & Firefox in Python
Selenium WebDriver remains the bedrock of browser automation for Python and enterprise testing. However, running standard Selenium Headless against protected websites triggers immediate bot blocks due to navigator.webdriver flags. Learn how to configure modern ChromeOptions or connect Selenium to cloud scraper browsers.
2Captcha Browser API
Bypasses Cloudflare Turnstile and DataDome challenges that consistently block standard Python Selenium scripts. Supports remote Selenium grid connection.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Modern Selenium Python Headless Chrome configuration
options = Options()
options.add_argument("--headless=new")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options)
driver.get("https://quotes.toscrape.com")
print("Page Title:", driver.title)
driver.quit()ChromeOptions for Selenium Python: `--headless=new` Syntax
Always use options.add_argument('--headless=new') in Selenium 4+. The legacy '--headless' flag runs the obsolete headless engine which lacks modern Chrome features and is instantly detected by Cloudflare and DataDome.
Why Selenium Headless Gets Blocked and How to Evade Detection
Anti-bot scripts detect Selenium through navigator.webdriver = true, default window sizes (800x600), missing media codecs, and datacenter IP ASNs. Adding stealth arguments and routing traffic through residential proxies is mandatory for reliable data extraction.
Selenium Chrome vs Selenium Firefox Headless
While Firefox headless (firefox_options.add_argument('-headless')) has fewer CDP-specific detection vectors, Chrome offers superior extension compatibility, Chrome DevTools Protocol access, and broader cloud browser endpoint support.
Architectural Benchmark & Pricing Comparison
| Configuration | Engine | Detection Rate | Proxy Support |
|---|---|---|---|
| Legacy Selenium (--headless) | Old Headless Engine | 90% Blocked | Manual HTTP |
| Modern Selenium (--headless=new) | Genuine Chrome Engine | 40-60% Blocked | Manual HTTP/Socks5 |
| 2Captcha Scraper Browser API (#1) | Managed Cloud Chrome | 0.6% (99.4% Pass) | 220+ Countries Built-in |
Frequently Asked Questions
What is the Robot Framework syntax for headless Chrome?
In Robot Framework, set Open Browser ${URL} chrome options=add_argument("--headless=new");add_argument("--no-sandbox").
How do I hide navigator.webdriver in Selenium Python?
Use driver.execute_cdp_cmd('Page.addScriptToEvaluateOnNewDocument', {'source': 'Object.defineProperty(navigator, "webdriver", {get: () => undefined})'}).
Launch Managed Cloud Headless Chrome in Seconds
Get instant access to genuine Chrome DevTools Protocol endpoints, automated CAPTCHA clearing, and residential proxies in 220+ countries.