I'm currently working on a project where I'm pulling data from the Aviation Week API and saving it into a database. However, I've encountered a challenge with the update process.
Here's the situation:
- The Aviation Week API returns a large dataset with 248,764 records just for 1 endpoint.
- Due to API limitations, I can only retrieve 150 records per call.
Now, the issue arises when I need to update these records. There isn't a field in the data indicating the last updated time, so I can't efficiently identify which records have been updated since the last pull. Has anyone else encountered a similar challenge? I'm seeking advice on how to efficiently handle updates without resorting to pulling the entire dataset repeatedly.
I'm trying to avoid making redundant API calls, so any insights or alternative approaches would be greatly appreciated.
Thank you in advance!
As a result, I find myself needing to pull the entire dataset again to check for updates.