Fetching data on a schedule and storing in AWS S3

Question

I've read many discussions and learnt this is possible using a lambda. My question is whether there's a managed service to facilitate this? I'm looking at grabbing CSVs off multiple http endpoints on a schedule and then processing in Glue. Ideally capturing the content in S3 in the process.

Content is less than 100mb, won't take more than the 15mins limit.

Thanks

Explored App Sync connectors and lambda.

Answer 1

Ok, lets me summarize this, you want to fetch data from HTTP endpoints on a schedule store it in S3 and then process it in Glue, right.

That can be done using Lambda + EventBridge:

• Create Glue ETL Job with S3 source and add other steps you need.
• Create lambda that will be triggered on a schedule example Schedule AWS Lambda using EventBridge. Lambda will load data into S3.
• Trigger Glue ETL job from Lambda example Lambda staring an AWS Glue job