My application receives links from various sources, and I'm trying to read the opengraph data from the websites in order to populate metadata about the link, similar to have applications like telegram, whatsapp, messenger, etc do it
What is the best way to be doing this? I've just been making simple serverside http requests from my node.js backend to the websites to just get the raw html, however I worry about hitting rate limits for links to public resources like facebook pages or twitter profiles, as I've found they sometimes do not work.
I've tried searching as much as I can but I can't find an agreed upon method of doing this. Should I be using a specific user agent, throttling requests, making these requests on the client? I'm not sure how best to do this