How to get HTTP responseBody using Selenium, CDP, JavaScript

Question

I am an AQA and testing an app. According to the test, after the button is clicked, I need to get the responseBody returned from the server, like we have in devtools - network tab. I have tried multiple Java and Python code examples found here, tried to transform them to JavaScript, but nothing worked for me. I've been trying smth like this:

try {

    const url = 'http://someUrl';

    const driver = await new Builder().forBrowser('chrome').build();

    const cdpConnection = await driver.createCDPConnection('page');

    await cdpConnection.execute('Network.responseReceived()', response => { 
        // Network.getResponseBody(), etc.
        const res = response.getResponse();
        console.log(res);
    };

await driver.get(url);
await driver.quit();

} catch (error) {
    console.log(error);
}

Answer 1

I am close to getting this to work... but not quite there. See my code here: https://github.com/SeleniumHQ/seleniumhq.github.io/issues/1155 If anyone can figure out the last step I'm missing, that'd be so amazing. ie.

let test = await cdpConnection.execute('Fetch.getResponseBody', {
                requestId: obj.params.requestId,
              });
              console.log(test); // ------>  THIS RETURNS UNDEFINED !!!!

Answer 2

1. Network.responseReceived is an Event. So you have to listen to the message from the underlying CDP connection.

wsConnection.on('message', message => {
    let data = JSON.parse(message);
    if (data.method === 'Network.loadingFinished') {
        // ... load response body here
    }
});

I use Network.loadingFinished event instead of Network.responseReceived, as the response is then completely loaded.

2. The problem is, that the CDPConnection class is not properly implemented yet: CDPConnection.js#L18 It doesn't return any promise. This is message-based communication, though it adds the Message ID, to retrieve later the Response Message from the WebSocket, but it doesn't handle that response message here webdriver.js#L1239

Until it is implemented you can use custom CDPConnection class. Here is the TypeScript implementation.

let ID = 0;

type TAwaiter = {
    id: number
    resolve: (value: any) => void
    reject: (reason?: any) => void
};

export class BiDiCDPConnection {

    private requests: Map<number, TAwaiter> = new Map();

    constructor(private wsConnection, private sessionId: string) {
        wsConnection.on('message', this.onMessage.bind(this));
        wsConnection.on('close', this.onClose.bind(this));
        wsConnection.on('error', this.rejectAll.bind(this));
    }

    execute <T = any> (method, params, onMessageSent: (err) => any = null): Promise<T> {

        let message = {
            sessionId: this.sessionId,
            method,
            params,
            id: ++ID,
        };
        let listener = {
            id: message.id,
            resolve: null,
            reject: null,
        };
        let promise = new Promise<T>((resolve, reject) => {
            listener.resolve = resolve;
            listener.reject = reject;
        });

        this.requests.set(listener.id, listener);

        this.wsConnection.send(JSON.stringify(message), onMessageSent)
        return promise;
    }

    private onMessage (message: Buffer) {
        let params = JSON.parse(message.toString());
        let { id, result } = params;

        if (id != null && this.requests.has(id)) {
            this.requests.get(id)?.resolve?.(result);
            this.requests.delete(id);
        }
    }

    private onClose () {
        this.rejectAll(new Error(`CDPConnection: The underlying connection was closed`));
    }

    private rejectAll(error: Error) {

        let awaiters = this.requests.values();
        this.requests = new Map();

        for (let awaiter of awaiters) {
            awaiter.reject(error);
        }
    }
}

Then you initialize the class and use it for your calls, after you create the inner cdp connection, as createCDPConnection establishes the WebSocket connection.

const cdpConnection = await driver.createCDPConnection('page');
const wsConnection = driver._wsConnection;
const bidiCdpConnection = new BiDiCDPConnection(wsConnection, driver.sessionId);


wsConnection.on('message', message => {
    let data = JSON.parse(message);
    if (data.method === 'Network.loadingFinished') {
                
        let response = await bidiCdpConnection.execute('Network.getResponseBody', {
            requestId: data.params.requestId,
        });
        console.log(response)
    }
});

---

I use this to monitor (selenium-query/BrowserNetworkMonitor.ts) and intercept (selenium-query/BrowserNetworkInterceptor.ts) requests. You can take and modify those classes for your initial needs.