BeautifulSoup AttributeError: 'get_text' sometimes in the same code

Question

Does anyone know where this problem comes from? I run the same code in a few seconds and sometimes it gives me that error and sometimes error it doesn't.

page = requests.get(URL, headers=headers)

soup1 = BeautifulSoup(page.content, 'html.parser')
soup2 = BeautifulSoup(soup1.prettify(), 'html.parser')
title = soup2.find(id='productTitle').get_text()
price = soup2.find(class_='aok-offscreen').get_text()

print(title)
print(price)

It really is the same code and I run it and sometimes it shows an error, works I continue searching for elements and working one yes and one no, thank you.

---> 10 title = soup2.find(id='productTitle').get_text()
     11 price = soup2.find(class_='aok-offscreen').get_text()
     14 print(title)

AttributeError: 'NoneType' object has no attribute 'get_text'

Answer 1

The problem is that there isn't an element that have the id "productTitle". You can try running it on two sperate commands like

title_elem = soup2.find(id='productTitle')
print(title_elem)

This will print None. So when you try to access get_text() method on title_elem like

print(title_elem.get_text())

This will generate an error.