speed up json parsing with rapidjson

116 Views Asked by At

In this code snippet, I parse json in a loop which makes the process slower. Instead of json, I tried using the rapidjson library for .loads and .dumps functions. However, the overall results are the same and no time is saved. Why could that be?

  def get_data(taskid, article_list, **kwargs):

        logging.info(f"Total number of ids: {len(article_list)}")

        chunks = [article_list[x:x + 50] for x in range(0, len(article_list), 50)]

        logging.info("Starting content API requests")
        content = []
        for i in chunks:
            try:
                response = requests.post(
                    'https://api-serverxxx',
                    headers=headers, json=i)
                content_chunk = response.text
                content.append(rapidjson.loads(response.text))
                if ('{"message":"Forbidden"}' in content_chunk):
                    logging.info(f"Error during lean data request: {content_chunk}")
                    exit()
            except Exception as e:
                logging.info(f"Exception: {e}")

        logging.info("Finished API requests")
        logging.info("Starting writing file to S3")
        try:
            filename = datetime.now().strftime("%Y%m%d_%H-%M-%S").replace("-", "_").replace(":", "_")
            json_data = rapidjson.dumps(content)
             ...

Is there any way I can use the rapidjson library to further speed up my code?

0

There are 0 best solutions below