Splash doesn't render the full page

36 Views Asked by At

I'm not able to figure out how to get full page rendered with splash. I tried to set proxies and differents user-agent, without success. The main div with the content is not rendered.. If someone could help on this, it would be appreciated.

Rendering from Splash (url https://www.finishers.com/courses)

enter image description here

There are some X-Cache Error from cloudfront.. I don't know if that's the issue, either how to solve this.

This is the page from the browser:

enter image description here

I tried to set differents proxies, user-agent and others parameters.

1

There are 1 best solutions below

1
Alireza Fathi On

Contents of this page load dynamically and you should send POST request method to get content. You can find data and headers from Inspect > Network > XHR and use

from scrapy.http import JsonRequest

And send request like this:

data = {
"name1": "value1",
"name2": "value2",
}
yield JsonRequest(url="http://www.example.com/post/action", data=data)