PagerDuty how to extract data for more than 1000 records for analytics endpoint

Question

I am struggling to extract data from PagerDuty API from weeks and there is no direct solution to my problem. This is a crucial task for my Job and any immediate help would be appreciated.

I want to extract data from PagerDuty analytics dashboard which displays insights of incident activity, service performance, responder, team, escalation policy. My manager want me to write a Python script that extract all the data that we usually find from the PagerDuty UI with the "export CSV" button.

I initially started with extracting list of incidents. I thought I will save all the team ids in a JSON file and use that to extract all the incidents from endpoint https://api.pagerduty.com/analytics/raw/incidents. Though I am able to extract the data quickly but I can't extract more than 1000 records, we have records more than 10000. What I understood from PagerDuty documentation is we can't use traditional pagination (limit + offset) to extract data from all the pages. I tried to read about starting_date and use that to loop over until the "more" is false but I am not able to think of code execution. The code I tried so far:

import requests
import csv
import json

url = "https://api.pagerduty.com/analytics/raw/incidents"

with open("team_data.json", 'r') as file:
    team_data = json.load(file)
    team_ids = team_data['pagerduty_id']

payload = {
    "filters": {
        "created_at_start": "2023-11-30T00:00:00Z",
        "created_at_end": "2023-12-03T00:00:00Z",
        "team_ids": team_ids,
    },
    "limit": 1000,
    "order": "desc",
    "order_by": "created_at",
    "time_zone": "Etc/UTC"
}

headers = {
    "Accept": "application/vnd.pagerduty+json;version=2",
    "Content-Type": "application/json",
    "Authorization": "Token token=api_key"
}

response = requests.post(url, json=payload, headers=headers)

if response.status_code == 200:
    incidents = response.json().get('incidents', [])
    for incident in incidents:
        incident_number = incident.get('incident_number')
        print(f"Incident Number: {incident_number}")

    raw_data = response.json().get('data', [])

    # Specify the file path for the CSV file
    csv_file_path = "incident_data.csv"

    # Writing incident data to CSV file
    with open(csv_file_path, mode='w', newline='') as csv_file:
        fieldnames = ["incident_number", "urgency", "priority_name", "service_name",
                      "team_name", "created_at", "resolved_at", "auto_resolved",
                      "assignment_count", "escalation_policy_name",
                      "business_hour_interruptions", "created_at",
                      "escalation_policy_id", "resolved_by_user_name",
                      "business_hour_interruptions"]

        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
        writer.writeheader()

        for d in raw_data:
            writer.writerow({
                "incident_number": d.get("incident_number"),
                "urgency": d.get("urgency"),
                "priority_name": d.get("priority_name"),
                "service_name": d.get("service_name"),
                "team_name": d.get("team_name"),
                "created_at": d.get("created_at"),
                "resolved_at": d.get("resolved_at"),
                "auto_resolved": d.get("auto_resolved"),
                "assignment_count": d.get("assignment_count"),
                "escalation_policy_name": d.get("escalation_policy_name"),
                "business_hour_interruptions": d.get("business_hour_interruptions"),
                "created_at": d.get("created_at"),
                "escalation_policy_id": d.get("escalation_policy_id"),
                "resolved_by_user_name": d.get("resolved_by_user_name"),
                "business_hour_interruptions": d.get("business_hour_interruptions"),
            })

    print(f"Data written to {csv_file_path}")
else:
    print(f"Error: {response.status_code}")

I don't know how to execute the code to take starting_after filter to extract all the records for given date range.

Need to display all the records from PagerDuty analytics endpoint.