I am trying to use BrowsermobProxy to Sniff network logs from my UI automation tests. I have to implement it for both HEAD and HEADLESS RemoteDriver. For Head-Driver it is working like a charm.
Code for my Head-Remote Driver:
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(browserMobProxy);
seleniumProxy.setNoProxy("<-loopback>");
seleniumProxy.setHttpProxy("localhost" + ":" + browserMobProxy.getPort());
seleniumProxy.setSslProxy("localhost" + ":" + browserMobProxy.getPort());
desiredCapabilities = DesiredCapabilities.chrome();
desiredCapabilities.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.IGNORE);
desiredCapabilities.setCapability(CapabilityType.ACCEPT_SSL_CERTS, Boolean.TRUE);
desiredCapabilities.setAcceptInsecureCerts(Boolean.TRUE);
desiredCapabilities.setJavascriptEnabled(Boolean.TRUE);
ChromeOptions options = getChromeOptions(driverConfig.getUserAgent(), Boolean.TRUE);
options.setCapability(CapabilityType.PROXY, seleniumProxy);
desiredCapabilities.setCapability(ChromeOptions.CAPABILITY, options);
desiredCapabilities.setCapability(ChromeOptions.CAPABILITY, options);
return RemoteWebDriver(remoteAddress, desiredCapabilities)
For Headless-RemoteDriver code:
ChromeOptions options = new ChromeOptions();
options.addArguments("--user-agent=" + "HeadlessChrome");
options.setAcceptInsecureCerts(Boolean.TRUE);
options.setHeadless(true);
options.addArguments("--allow-insecure-localhost", "--no-sandbox", "--disable-extensions", "--window-size=1920,1080");
options.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
List<String> optionsArg = ImmutableList.of("--ignore-certificate-errors", "--proxy-bypass-list=<-loopback>",
"--proxy-server=http://localhost:" + mobProxy.getPort(), "--remote-debugging-port=9222");
options.addArguments("--ssl-protocol=any");
options.addArguments("--allow-running-insecure-content");
options.addArguments(optionsArg);
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
driver = new RemoteWebDriver(driverUrl, capabilities);
In case of my headless driver the config for driver are as follows:
Driver URL -> http://localhost:<port_number> [eg: http://localhost:29515]
BrowserMob Proxy Port -> <dynamic_port> [eg:33173]
Proxy Server Address -> http://localhost:<dynamic_port> [eg: http://localhost:33173]
The full capabilities list before driver creation for headless is as follows:
{acceptInsecureCerts=true, acceptSslCerts=true, browserName=chrome,
goog:chromeOptions={args=[--user-agent=HeadlessChrome, --headless, --disable-gpu,
--allow-insecure-localhost, --no-sandbox, --disable-extensions,
--window-size=1920,1080, --ssl-protocol=any, --allow-running-insecure-content,
--ignore-certificate-errors, --proxy-bypass-list=<-loopback>,
--proxy-server=http://localhost:33173, --remote-debugging-port=9222], extensions=[]},
platform=ANY, version=}
Result
When taken screenshot I am seeing a white page only on test failure, Implicit waits are already applies for page load and on top of that I have tried with 5 seconds static sleep time. No fix. I get a HTML dump:
<html><head></head><body></body></html>
NOTES
I have tried with the argument --proxy-bypass-list=*, but in this case the network is not routed through mob proxy, as I am unable to record any logs. The pages are loading fine in this case.
The capabilities: --ssl-protocol=any, --remote-debugging-port=9222 and --allow-running-insecure-content are extras, I have tried without them also, no avail.
I am using browser-mob-proxy = 2.1.5
ChromeDriver version = 92.0.4515.159
I am running this on a Remote Linux CLI System
When starting the browsermob proxy I am doing the following:
System.setProperty("bmp.allowNativeDnsFallback", "true");
BrowserMobProxy browserMobProxy = new BrowserMobProxyServer();
browserMobProxy.setTrustAllServers(Boolean.TRUE);
Kindly, someone help to resolve this issue, I am stuck for at least a week now and my deadline is nearing fast.
For headless chrome , useragent is set as chromeheadless or something , this makes website to detect that you are using headless chrome and behave differently
You can vent this by specifying hardcoded useragent,
open a normal chrome , goto network tab , open request header and copy the user agent part and replace in your code