Error in loading JSON data -JSONDecodeError

Question

I am trying to get json data in a container using following code:

import requests
import pandas as pd
import json
from bs4 import BeautifulSoup

url = 'https://www.nseindia.com/api/quote-derivative?symbol=NIFTY&identifier=FUTIDXNIFTY29-02-2024XX0.00'

headers = {
        'accept':'*/*',
        'accept-encoding':'gzip, deflate, br',
        'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36'
         }

session = requests.Session()

data = session.get(url, headers = headers).json()

It throws me following error:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Answer 1

TL;DR --> Add cookies to the request

The website just checks if the request you have sent has the right cookies. So a good approach to get the information is to copy your cookies and adding it into your request. Here's a PoC (I didn't use requests.session, and I've redacted the actual values of cookies, but you should be able to implement this approach in your code):

import requests

cookies = {
    "nsit": "Redacted",
    "AKA_A2": "Redacted",
    "_ga": "Redacted",
    "nseQuoteSymbols": "Redacted",
    "ak_bmsc": "Redacted",
    "_ga_QJZ4447QD3": "Redacted",
    "defaultLang": "Redacted",
    "nseappid": "Redacted",
    "RT": "Redacted",
    "_ga_87M7PJ3R97": "Redacted",
    "bm_sv": "Redacted"
}

headers = {
    'authority': 'www.nseindia.com',
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9',
    'referer': 'https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY&identifier=FUTIDXNIFTY29-02-2024XX0.00',
    'sec-ch-ua': '"Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"',
    'sec-ch-ua-mobile': '?0',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A',
}
response = requests.get('https://www.nseindia.com/api/quote-derivative?symbol=NIFTY&identifier=FUTIDXNIFTY29-02-2024XX0.00', cookies=cookies, headers=headers)

Now of course the cookies won't be redacted, I hid it because of privacy reasons. If you do similar approach your code will work out.