Scraping all links using BeautifulSoup

Question

I am trying to scrape all match reports links from the page but there is 'load more' button, and I don't want to use selenium. Is there any solution to collect all links without selenium. Thanks in advance.

Here what I tried:

 from bs4 import BeautifulSoup as bs
 import requests
 r=requests.get('https://www.iplt20.com/news/match-reports')
 soup = bs(r.text,'lxml')

 for match in soup.find_all('div',class_='latest-slider-wrap 
 position-relative'):
      links = match.find('a')
      print(links['href'])

Answer 1

Try:

import requests
from bs4 import BeautifulSoup

url = "https://www.iplt20.com/news/match-reports"

soup = BeautifulSoup(requests.get(url).content, "html.parser")

for a in soup.select("#div-match-report a:has(li)"):
    print(a["href"])

Prints:

https://www.iplt20.com/news/4014/tata-ipl-2024-match-11-lsg-vs-pbks-match-report
https://www.iplt20.com/news/4012/tata-ipl-2024-match-10-rcb-vs-kkr-match-report
https://www.iplt20.com/news/4011/tata-ipl-2024-match-09-rr-vs-dc-match-report
https://www.iplt20.com/news/4009/tata-ipl-2024-match-08-srh-vs-mi-match-report
https://www.iplt20.com/news/4007/tata-ipl-2024-match-07-csk-vs-gt-match-report
https://www.iplt20.com/news/4006/tata-ipl-2024-match-06-rcb-vs-pbks-match-report
https://www.iplt20.com/news/4004/tata-ipl-2024-match-05-gt-vs-mi-match-report
https://www.iplt20.com/news/4003/tata-ipl-2024-match-04-rr-vs-lsg-match-report
https://www.iplt20.com/news/4001/tata-ipl-2024-match-03-kkr-vs-srh-match-report
https://www.iplt20.com/news/4000/tata-ipl-2024-match-02-pbks-vs-dc-match-report
https://www.iplt20.com/news/3999/tata-ipl-2024-match-01-csk-vs-rcb-match-report
https://www.iplt20.com/news/3976/tata-ipl-2023-final-csk-vs-gt-match-report
https://www.iplt20.com/news/3974/tata-ipl-2023-qualifier-2-gt-vs-mi-match-report
https://www.iplt20.com/news/3973/tata-ipl-2023-eliminator-lsg-vs-mi-match-report
https://www.iplt20.com/news/3972/tata-ipl-2023-qualifier-1-gt-vs-csk-match-report
https://www.iplt20.com/news/3971/tata-ipl-2023-match-70-rcb-vs-gt-match-report
https://www.iplt20.com/news/3970/tata-ipl-2023-match-69-mi-vs-srh-match-report
https://www.iplt20.com/news/3969/tata-ipl-2023-match-68-kkr-vs-lsg-match-report
https://www.iplt20.com/news/3968/tata-ipl-2023-match-67-dc-vs-csk-match-report
https://www.iplt20.com/news/3967/tata-ipl-2023-match-66-pbks-vs-rr-match-report
https://www.iplt20.com/news/3966/tata-ipl-2023-match-65-srh-vs-rcb-match-report

---

EDIT: To get all links you can use their Ajax pagination API:

import requests

api_url = "https://www.iplt20.com/add-more-match-report?page={page}&type=match-reports"

for page in range(1, 4):  # <-- adjust number of pages here
    print(f"{page=}")
    data = requests.get(api_url.format(page=page)).json()
    for d in data["newsResponce"]["data"]:
        print(f'https://www.iplt20.com/news/{d["id"]}/{d["titleUrlSegment"]}')

Prints:

...

page=2
https://www.iplt20.com/news/3964/tata-ipl-2023-match-64-pbks-vs-dc-match-report
https://www.iplt20.com/news/3963/tata-ipl-2023-match-63-lsg-vs-mi-match-report
https://www.iplt20.com/news/3962/tata-ipl-2023-match-62-gt-vs-srh-match-report
https://www.iplt20.com/news/3960/tata-ipl-2023-match-61-csk-vs-kkr-match-report
https://www.iplt20.com/news/3959/tata-ipl-2023-match-60-rr-vs-rcb-match-report
https://www.iplt20.com/news/3958/tata-ipl-2023-match-59-dc-vs-pbks-match-report
https://www.iplt20.com/news/3956/tata-ipl-2023-match-58-srh-vs-lsg-match-report
https://www.iplt20.com/news/3955/tata-ipl-2023-match-57-mi-vs-gt-match-report
https://www.iplt20.com/news/3953/tata-ipl-2023-match-56-kkr-vs-rr-match-report
https://www.iplt20.com/news/3952/tata-ipl-2023-match-55-csk-vs-dc-match-report
https://www.iplt20.com/news/3951/tata-ipl-2023-match-54-mi-vs-rcb-match-report
https://www.iplt20.com/news/3947/tata-ipl-2023-match-53-kkr-vs-pbks-match-report
https://www.iplt20.com/news/3946/tata-ipl-2023-match-52-rr-vs-srh-match-report
https://www.iplt20.com/news/3945/tata-ipl-2023-match-51-gt-vs-lsg-match-report
https://www.iplt20.com/news/3944/tata-ipl-2023-match-50-dc-vs-rcb-match-report
https://www.iplt20.com/news/3943/tata-ipl-2023-match-49-csk-vs-mi-match-report
https://www.iplt20.com/news/3942/tata-ipl-2023-match-48-rr-vs-gt-match-report
https://www.iplt20.com/news/3940/tata-ipl-2023-match-47-srh-vs-kkr-match-report
https://www.iplt20.com/news/3938/tata-ipl-2023-match-46-pbks-vs-mi-match-report
https://www.iplt20.com/news/3937/tata-ipl-2023-match-45-lsg-vs-csk-match-report
https://www.iplt20.com/news/3936/tata-ipl-2023-match-44-gt-vs-dc-match-report
page=3
https://www.iplt20.com/news/3934/tata-ipl-2023-match-43-lsg-vs-rcb-match-report
https://www.iplt20.com/news/3932/tata-ipl-2023-match-42-mi-vs-rr-match-report
https://www.iplt20.com/news/3931/tata-ipl-2023-match-41-csk-vs-pbks-match-report
https://www.iplt20.com/news/3930/tata-ipl-2023-match-40-dc-vs-srh-match-report

...