Scraping data from Reddit - No API Sept 2023

Question

I know recently Reddit changed their way to handle APIs and it is very restrictive now. I am working on a school project and need Reddit data on Stocks (subredits: Wallstreetbets, StockMarket). I am currently trying to scrape the pages from Old Reddit but only get a few records out. I was expecting a lot more data.

I have the following code, and even though I have the num_pages_to_scrapre set to 5000, I only get 138 records out. I thought that maybe next_button is not working correctly, or I should change the time.sleep(2) but still I get the same results. Please help!

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

url = "https://old.reddit.com/r/wallstreetbets"
headers = {'User-Agent': 'Mozilla/5.0'}

data = []  # List to store post data

#Set the desired number of pages
num_pages_to_scrape = 5000

for counter in range(1, num_pages_to_scrape + 1):
    page = requests.get(url, headers=headers)
    soup = BeautifulSoup(page.text, 'html.parser')

    posts = soup.find_all('div', class_='thing', attrs={'data-domain': 'self.wallstreetbets'})

    for post in posts:
        title = post.find('a', class_='title').text
        author = post.find('a', class_='author').text
        comments = post.find('a', class_='comments').text.split()[0]

        if comments == "comment":
            comments = 0

        likes = post.find("div", class_="score likes").text

        if likes == "•":
            likes = "None"
        
        # Extract the date information from the HTML
        date_element = post.find('time', class_='live-timestamp')
        date = date_element['datetime'] if date_element else "N/A"
        formatted_date = pd.to_datetime(date, utc=True).strftime('%Y-%m-%d %H:%M:%S')

        data.append([formatted_date, title, author, comments, likes])

    next_button = soup.find("span", class_="next-button")
    if next_button:
        next_page_link = next_button.find("a").attrs['href']
        url = next_page_link
    else:
        break

    time.sleep(2)

#Create df
columns = ['Date', 'Title', 'Author', 'Comments', 'Likes']
df = pd.DataFrame(data, columns=columns)

#Print the DataFrame
df

Answer 1

Here is an example skeleton, how you can use their Json API to download multiple pages of data (note: to get data in Json form, add .json at the end of the URL):

import json

import requests

# api doc: https://old.reddit.com/dev/api/
url = "https://reddit.com/r/wallstreetbets.json"
headers = {"User-Agent": "Mozilla/5.0"}

data = requests.get(url, headers=headers).json()

while True:
    # uncomment this to print all data:
    # print(json.dumps(data, indent=4))
    for c in data["data"]["children"]:
        print(c["data"]["title"])

    url = "https://reddit.com/r/wallstreetbets.json?after=" + data["data"]["after"]
    data = requests.get(url, headers=headers).json()

    if not data:
        break

Prints:

...

Remember kids, bankruptcies are good for stonks                                                                                                                                                                    
Bullish news - Tesla mcflurry cyber spoon                                                                
King of dilution strikes again                                                                           
Unleashing the Power of YouTube for Alpha Gains    
When is PayPal going                                                                               
He's been right so far                                                                                                                                                                                             
FuelCell teaming up with Toyota. First ever Tri-gen plant up and running
Bad day to be an Ape                                                                                     

...