I'm using the googlesearch library in Colab, but I can't import my websites to a list. How can I do that?

12 Views Asked by At

What I am trying to do is search for specific term, get the list of websites that come up with when searching for that term, and store them as a list for future reference and analysis. Here is the code that I have so far that is working:

from googlesearch import search
print("Googlesearch package installed successfully!")

# set query to search for in Google
query = "Lysotracker"
# execute query and store search results
results = search(query, tld="com", lang="en", stop=25, pause=2)
# iterate over all search results and print them
for result in results:
 print(result)

So this just gets the list of URLs and prints the results. What I then try to do is remove all of the extraneous address text and input the website title to a list with this code:

# Initialize a list to store website titles
website_titles = []

# Iterate over all search results
for result in results:
    # Use regular expressions to extract the website title
    title = re.search('(?<=://)(.*?)(?=/|$)', result).group(1)
    
    # Append the website title to the website_titles list
    website_titles.append(title)

# Display the list of website titles
print(website_titles)

However, my list just winds up having no entries. I am sure there has to be a way to do this. I have tried web scraping tutorials but I just get error after error when running even first few steps, so I tried this method. I can at least get websites, but now I can't make a list.

Thanks for you time.

0

There are 0 best solutions below