So I'm trying to develop a solution using AWS Glue that basically takes some files that are present in one S3 bucket and then moves them to a different bucket in S3. At first sight this is very simple but I want this solution to be totally generic - i.e I want it to work for any file I point it to in S3 - so I can't hardcode the creation of the source database and the source table in the glue job script.

What I'm trying to do then is to create a glue crawler - via cloudformation template- and point it to the desired S3 location so it automatically creates the table for me and adds it to the data catalog. The problem is that I can't find a way to pass the s3 location path parameter to the crawler in such a way that it can change when I need to. For example:

{"AWSTemplateFormatVersion": "2010-09-09",
 "Resources": {
  "gluejobcrawler": {
   "Type": "AWS::Glue::Crawler",
   "Properties": {
    "Role": "gluejobrole",
    "Targets": {
      "S3Targets":[
        {
          "Path": "Here I want a generic path so the crawler can inferr the schema of any S3 location I pass
       .
       .
       .
      }
     }
    }
   }
}

Any advice is usefull even if you come up with another solution to perform this task.

Thanks!

0

There are 0 best solutions below