Selenium is a very popular automation tool, Selenium 4 has introduced many advantages which are helpful for testers. Chrome DevTools Protocol (CDP) is one of the major features which selenium has introduced recently. As the Chrome DevTools Protocol is now available with selenium 4 we can control the network on Chrome browsers.  In this tutorial let’s discuss some of the features of Network Interception

  • Capture Request with Selenium 4
  • Capture Response with Selenium 4
  • Block Network URLs with Selenium 4
  • Load Insecure website in Selenium 4
  • Mock Network using Selenium 4
  • Control Network Speed/Emulation with Selenium4

Capture Network Request in Selenium 4

Selenium 4 has introduced CDP, using CDP you get access to browser Network features. You can get access to the Network HTTP requests when you make the call to the particular domain. For example, if you type URL https://www.rahulshettyacademy.com internally it makes requests to a lot of assets such as CSS, images, javascript, it may even call the API to load the data on the browser.

To capture Network Request in Selenium 4, you need to follow below steps.

The CDP features are made available in the selenium using DevTool package/class so we need to create the instance.

ChromeDriver driver = new ChromeDriver();

DevTools devTool = driver.getDevTools();

devTool.createSession();

In the above code, we care about creating driver instance first, using driver instance we are getting dev tools and then we are creating session.

The DevTool provides the command send() which allows us to send the command to CDP, CDP has documented list of Chrome DevTools Protocol command, by referring their documentation, to capture network request we need to

  • Enable the Network
  • Add the Listener to requestWillBeSent event

So the code looks like below

devTool.send(Network.enable(Optional.empty(), Optional.empty(),

Optional.empty()));

      devTool.addListener(Network.requestWillBeSent(), requestSent -> {

      System.out.println("Request URL => " + requestSent.getRequest().getUrl());

      System.out.println("Request Method => " + requestSent.getRequest().getMethod());

      System.out.println("Request Headers => " + requestSent.getRequest().getHeaders().toString());

      System.out.println("-------------------------------------------------");

});

Once we are done with Network capturing, we need to navigate to the URL

driver.get("https://rahulshettyacademy.com/#/index");

Putting Everything Together

Example: Capture Network Request Using CDP in Selenium 4

import java.util.Optional;

import org.junit.Test;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.network.Network;

public class CaptureRequestSelenium {

      @Test

      public void captureRequestSelenium() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver ");

            ChromeDriver driver = new ChromeDriver();

            DevTools devTool = driver.getDevTools();

            devTool.createSession();

            devTool.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

            devTool.addListener(Network.requestWillBeSent(), requestSent -> {

                  System.out.println("Request URL => " + requestSent.getRequest().getUrl());

                  System.out.println("Request Method => " + requestSent.getRequest().getMethod());

                  System.out.println("Request Headers => " + requestSent.getRequest().getHeaders().toString());

                  System.out.println("------------------------------------------------------");

            });

            driver.get("https://rahulshettyacademy.com/#/index");

      }

}

Once you Execute the above Code the Output looks like below

Output : Network Request CDP Selenium 4

Capturing Network Response with CDP in Selenium 4

Above we have captured network requests similarly we can capture network response, For example, if you are looking for status for API, we can get it that as well.

To Capture Network response we need to do

  • Enable Network
  • Add Listener to Network.responseReceived event

Code Looks like below

Example: Capture Network Response with Chrome DevTools Protocol in Selenium 4

import java.util.Optional;

import org.junit.Test;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.network.Network;

public class CaptureResponseSelenium {

      @Test

      public void captureRequstSelenium() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver");

            ChromeDriver driver = new ChromeDriver();

            DevTools devTool = driver.getDevTools();

            devTool.createSession();

            devTool.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

            devTool.addListener(Network.responseReceived(), responseReceieved -> {

                  System.out.println("Response Url => " + responseReceieved.getResponse().getUrl());

                  System.out.println("Response Status => " + responseReceieved.getResponse().getStatus());

                  System.out.println("Response Headers => " + responseReceieved.getResponse().getHeaders().toString());

                  System.out.println("Response MIME Type => " + responseReceieved.getResponse().getMimeType().toString());

                  System.out.println("------------------------------------------------------");

            });

            driver.get("http://google.com");

      }

}

Output: Network Response CDP Selenium

Blocking URL using Chrome DevTools Protocol in Selenium

Consider example, if you want to validate the website blocking all the .css files (that is website without styles), you need to block all the urls/files which are having extensions with .css. Similarly, you might have a scenario to test blocking certain API calls. These scenarios can be tested using Chrome DevToos Protocol(CDP) in selenium 4.

The Network.setBlockedURLs () provides option to block the List of URLs

If you want to block all the .css you can simply use

devTool.send(Network.setBlockedURLs(List.of("*.css"))); // Blocks all css

If you want to block a particular API or URL you can use like below

devTool.send(Network.setBlockedURLs(List.of(

    "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"

)));

Example: Blocking API/URL/CSS in Selenium 4

Before Blocking CSS website looks like below

import org.junit.Test;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.network.Network;

public class BlockURL {

      @Test

      public void blockUrl() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver");

            ChromeDriver driver = new ChromeDriver();

            DevTools devTool = driver.getDevTools();

            devTool.createSession();

            devTool.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

            devTool.send(Network.setBlockedURLs(List.of("*.css"))); // Blocks all css files

           

            /*

             * Block request by URL below code

             devTool.send(Network.setBlockedURLs( List.of(

             "https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"

              )));

             */

           

            devTool.addListener(Network.loadingFailed(), loadingFailed -> {

                  System.out.println("Blocking reason: " + loadingFailed.getBlockedReason().get());

            });

            driver.get("https://rahulshettyacademy.com/#/index");

      }

}

After Executing the above code it blocks all the styles and the Website will be loaded like below.

Output : Blocking URL in Selenium with CDP

Loading Insecure Website using CDP in Selenium 4

The browser usually blocks, we are trying to load the HTTPS website if the certificate has some errors. Many times, we will not be able to proceed with such website automation. CDP provides the option to ignore those certificate errors and we can load that website on the browser and test it.

The setIgnoreCertificateErrors command in CDP can be used with Selenium 4 and we can proceed with testing.

Security.setIgnoreCertificateErrors(true)

Example : Load Insecure webpage with CDP in Selenium 4

import org.junit.Test;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.security.Security;

public class InsecureWebsite {

      @Test

      public void ignoreCertificate() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver");

            ChromeDriver driver = new ChromeDriver();

            DevTools devTool = driver.getDevTools();

            devTool.createSession();

            devTool.send(Security.setIgnoreCertificateErrors(true));

            driver.get("https://untrusted-root.badssl.com/");

      }

}

The above code loads the website though it's not secure and has an HTTPS certificate error.

Network Emulation and Controlling Network Speed using CDP in Selenium 4

Consider we have a scenario, we need to test our website how it loads under slow internet connection, like 2G, 3G, 4G, etc. If our website is tested under slow bandwidth conditions and if it is optimized to work in slow bandwidth it works normally. The Selenium 4 with Chrome DevTools Protocol provides option to control the network bandwidth it is called network emulation.

Network emulation can be done using the command Network.emulateNetworkConditions, The selenium DevTools wrapper takes argument such as offline, latency, downloadThroughput, uploadThroughput, connectionType. We need to call this method to emulate network conditions.

import java.util.Optional;

import org.junit.Test;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.network.Network;

import org.openqa.selenium.devtools.v94.network.model.ConnectionType;

public class NetworkSpeed {

      @Test

      public void networkSpeed() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver");

            ChromeDriver driver = new ChromeDriver();

            DevTools devTool = driver.getDevTools();

            devTool.createSession();

            devTool.send(Network.enable(Optional.empty(), Optional.empty(), Optional.empty()));

            devTool.send(Network.emulateNetworkConditions(false, 20, 20, 50, Optional.of(ConnectionType.CELLULAR3G)));

            driver.get("https://rahulshettyacademy.com/#/index");

      }

}

Once we execute the above code, it emulates the network as per said parameters and loads the website. This helps to test website performance under slow network conditions.

Mock API using Chrome DevTools Protocol in Selenium 4

Consider the example, we have a website which internally calls some API, now what happens if we change the API parameters, what happens if we change the request information. Mocking can be done in selenium using CDP commands

Let’s consider the example,

Navigate to https://rahulshettyacademy.com/angularAppdemo/

Click on Virtual Library

It loads the table with book details.

If you open the Chrome Network Tab, internally it is making call to /GetBook.php?AuthorName=Shetty

Now, let’s change the network URL parameters programmatically,

We have /GetBook.php?AuthorName=Shetty API call for that lets replace Shetty with Unkown and lets see what happens

Example : Mock API Requests in Selenium 4 Using CDP

import java.util.Optional;

import org.junit.Test;

import org.openqa.selenium.By;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v94.fetch.Fetch;

public class MockNetwork {

      @Test

      public void mockAPISelenium() {

            System.setProperty("webdriver.chrome.driver", "path to chromedriver");

            ChromeDriver driver = new ChromeDriver();

            DevTools devtools = driver.getDevTools();

            devtools.createSession();

            devtools.send(Fetch.enable(Optional.empty(), Optional.empty()));

            devtools.addListener(Fetch.requestPaused(), req -> {

                  if (req.getRequest().getUrl().contains("=Shetty")) {

                        String mock = req.getRequest().getUrl().replace("=shetty", "=Unknown");

                        devtools.send(Fetch.continueRequest(req.getRequestId(), Optional.of(mock), Optional.empty(),

                                    Optional.empty(), Optional.empty(), Optional.empty()));

                  } else {

                        devtools.send(Fetch.continueRequest(req.getRequestId(), Optional.of(req.getRequest().getUrl()),

                                    Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty()));

                  }

            });

            driver.get("https://www.rahulshettyacademy.com/angularAppdemo/");

            driver.findElement(By.xpath("//button[contains(text(),'Virtual Library')] ")).click();

      }

}

Let’s understand the Code,

devtools.send(Fetch.enable(Optional.empty(), Optional.empty())); : We are enabling Featch

devtools.addListener(Fetch.requestPaused(), req -> { : In this line we are adding listener to the CDP evernt Fetch.requestPaused, this helps to get the each request that is made from the browser.

if (req.getRequest().getUrl().contains("=Shetty")) { : Our scenario is monitor only request having parameter “Shetty”, and we are going to replace that with Unkown, so we are verifuing the request is having parameter “Shetty”

String mock = req.getRequest().getUrl().replace("=shetty", "=Unknown"); : If the URL is having Shetty then we need replace original url with Unknown that is what we are doing here, this is mock URL we are building from orginal URL

devtools.send(Fetch.continueRequest(req.getRequestId(), Optional.of(mock), Optional.empty(),Optional.empty(), Optional.empty(), Optional.empty())); : Now, we need to continue with Mock Request, we have replaced original request with mock request in this line and continuing with mock request. That means browser sends the mocked request in place of original request.

else {devtools.send(Fetch.continueRequest(req.getRequestId(), Optional.of(req.getRequest().getUrl()),Optional.empty(), Optional.empty(), Optional.empty(), Optional.empty())); }

We are not bothered about the request if the request doesn’t contain the “Shetty” parameter so else part we are just continuing with the original URL

Let’s execute the above code see what happens

Output : Mock API Requests in Selenium CDP