How to make sure when scraping Bing URLs, that the URLs are actually found

36 Views Asked by At

I'm trying to scrape some URLs from Bing for a project. Once the Bing search has been found, I try to get all the links that were on the bing page into a list for further analysis. However, when using random keywords as the search, the list of URLs Found either work as expected or there are no links found. The keywords that no links change every time I use it.

def getBingLink(query):
    l=[]
    count = 0
    num = random.random()
    time.sleep(num)
    target_url="https://www.bing.com/search?q=" +query 
    headers = {'User-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Norton/121.0.0.0'}
    resp=requests.get(target_url, headers )
    print(resp.status_code)
    soup = BeautifulSoup(resp.text, 'html.parser')
    completeData = soup.find_all("li",{"class":"b_algo"})
    if (len(completeData) == 0):
        return l
    for i in range(0, len(completeData)):       
        o=completeData[i].find("a").get("href")    
        l.append(o)
        count = count +1
    return l

Today, when I use the word "doctor" as the keyword I get : [] When I use the words "doctor who" as the keyword I get:

[[https://en.wikipedia.org/wiki/Doctor_Who,...] 

and other links which is what is expected. What am I doing wrong?

0

There are 0 best solutions below