So I'm trying to develop a solution using AWS Glue that basically takes some files that are present in one S3 bucket and then moves them to a different bucket in S3. At first sight this is very simple but I want this solution to be totally generic - i.e I want it to work for any file I point it to in S3 - so I can't hardcode the creation of the source database and the source table in the glue job script.
What I'm trying to do then is to create a glue crawler - via cloudformation template- and point it to the desired S3 location so it automatically creates the table for me and adds it to the data catalog. The problem is that I can't find a way to pass the s3 location path parameter to the crawler in such a way that it can change when I need to. For example:
{"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"gluejobcrawler": {
"Type": "AWS::Glue::Crawler",
"Properties": {
"Role": "gluejobrole",
"Targets": {
"S3Targets":[
{
"Path": "Here I want a generic path so the crawler can inferr the schema of any S3 location I pass
.
.
.
}
}
}
}
}
Any advice is usefull even if you come up with another solution to perform this task.
Thanks!