Block Bots visiting paid URLs

Question

I have Laravel application which is serving some content from external links, paid links. Problem is that bots are visiting all those links and gettting bill at end of a month is not so fun when those bots make 100000 requests.

How can i restrict access for bots and other unwanted visitors?

Answer 1

Completely agree with @CharlesDuffy, now if you want something simple you could modify the way Laravel limit rate request. Check this package made by Graham Campbell.

Answer 2

You should only allow authenticated users to access those premium links using the auth middleware on the appropriate route in routes/web.php:

Route::get('linkWhichCostsYouMoney', function () {
    // Only authenticated users may enter...
})->middleware('auth');

See the Laravel docs.

You could also try disallowing robots to access the paid pages using robots.txt. This will only stop good bots, not bad bots.

Finally, depending on your application, it may be possible to cache the content locally and avoid using the paid service for every single request. If you want search engines to index your pages including paid content, this is the path you should follow.