I want to call GlueCrawler from the Glue job. I see there is an API https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-crawling.html#aws-glue-api-crawler-crawling-StartCrawler
But I totally lost how I can call it? I can't find examples in internet as well. I found some Python samples using boto3 and it is working fine, but I need scala call.
If I understand correctly, I need some how instantiate GlueClient and call startCrawler operation on it, but how to do it? All examples that I found operates only on GlueContext and it is not clear how to retrieve GlueClient from it(if it is possible at all?).
How to call AWS Glue crawler from AWS Glue job using Scala API?
147 Views Asked by Andrei Markhel At
1
There are 1 best solutions below
Related Questions in SCALA
- Mocking AmazonS3 listObjects function in scala
- Last SPARK Task taking forever to complete
- How to upload a native scala project to local repo by sbt like using "maven install"
- Folding a list of OR clauses in io.getquill
- How to get latest modified file using scala from a folder in HDFS
- Enforce type bound for inferred type parameter in pattern matching
- can't write pyspark dataframe to parquet file on windows
- spark streaming and kafka integration dependency problem
- how to generate fresh singleton literal type in scala using macros
- exception during macro expansion: type T is not a class, play json
- Is there any benefit of converting a List to a LazyList in Scala?
- Get all records within a window in spark structured streaming
- sbt publishLocal of a project with provided dependencies in build.sbt doesn't make these dependencies visible to projects using the project as library
- Scala composition of partially-applied functions
- How to read the input json using a schema file and populate default value if column not being found in scala?
Related Questions in AWS-GLUE
- AWS GLUE child node execution order of same level
- Is there a way to import Redshift Connection in PySpark AWS Glue Job?
- Retrieving a list of all failed Glue jobs via CLI
- How do I change the data type in a Glue Crawler?
- Loading around 50gb of parquet data to Redshift taking indefinite time to load
- Glue Notebook not starting: Failed to start notebook
- old aws-glue libraries in the Glue streaming ETL job 4.0?
- Add File name column to Dynamic Frame
- How to test Glue jobs and Athena queries locally on dummy data?
- AWS Glue throws AWSBadRequestException when loading DynamicFrame from s3 with local Glue docker
- AWS Glue Insert and update into oracle table
- SQL query to extract incremental data from a table in SQL Server
- redshift spectrum type conversion from String to Varchar
- Apply transformation on nested json column in dataframe
- Access Denied while creating crawler
Related Questions in GLUE-CRAWLER
- How to pass a generic parameter value as the path of the Glue Crawler into the CloudFormation templates that creates it?
- Unable to create S3 Crawler in AWSGLUE. Access Denied
- Query RDS table with Redshift
- Unknown parameter in Targets DeltaTargets must be one of: S3Targets, JdbcTargets, MongoDBTargets, DynamoDBTargets, CatalogTargets
- AWS Glue crawler only crawl the column name not the data
- PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 104857600 bytes. Pass allowDiskUse:true to opt in
- How to call AWS Glue crawler from AWS Glue job using Scala API?
- Glue crawler only doing top level of DynamoDb Export
- aws glue create-crawler fails on Configuration settings
- How to specify glue version 3.0 for an AWS crawler with boto3?
- How to load only metadata in data catalog table using aws crawler
- Classify custom date string as date in AWS Glue Crawler
- Glue Catalog w/ Delta Tables Connected to Databricks SQL Engine
- Glue Crawler/Athena array of strings handling
- Why is Kinesis or Crawler creating partitions in my data?
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?
I haven´t workded with glue, but looks like there is a JDK SDK for glue with some examples of how to start a crawler. One of them, shows how to start a crawler, but as you said the
GlueClientis provided. Based on the thread How to Get AWS Glue Client in Java, you should be able to create an instance of the client doing something likeThe API Doc of GlueClient shows that the it should be possible to do that.
Hope this helps