How can we effectively call a Python UDF function for multiple CSV files in S3 stage? I have like ~450K CSV files (each in size of few KBs) coming in daily and I need to select only certain columns from each file and load it in table. I'm using a UDF to read the header and select only required columns. Right now it's taking ~10 mins to read and load the file. Is there any optimization technique available that can speed this process?
Calling Python UDF function for multiple files
61 Views Asked by Selva At
0
There are 0 best solutions below
Related Questions in SNOWFLAKE-CLOUD-DATA-PLATFORM
- Are there poor practices in this use of python cryptography package to generate RSA keypair?
- snowflake cost management page limited warehouse access to role
- How to make FLATTEN function in Snowflake return PATH in Dot Notation instead of Brackets Notation
- How to overwrite a single partition in Snowflake when using Spark connector
- snowflake enforce unsorted json into variant column
- Spark connectors from Azure Databricks to Snowflake using AzureAD login
- Load data from csv in airflow docker container to snowflake DB
- Snowflake ODBC xdg-open Missing X server or $DISPLAY
- How can I reduce table scan time in snowflake
- API INTEGRATION for azure devops git on snowflake
- When will "create or alter" be available to all accounts?
- Event_date reference in CTE
- Problem decorating Python stored procedure handler with @functools.cache
- How to add a 1 to a phone number and remove the dashes?
- DBT - Merge - Only update condition
Related Questions in SNOWFLAKE-STAGE
- Snowflake loading file from stage subfolder not working
- Pattern misbehaving in Copy into for file in snowflake stage
- How to pass parameters when calling an sql script from a stage in snowflake
- Compilation error using snowflake COPY INTO internal stage
- Snowflake Stage Error "OSError: [Errno 28] No space left on device "
- GET file from internal stage via NodeJS SDK 1.9.2
- How to update the rows using dynamic table in snowflake?
- How to remove specific characters from select column in Snowflake
- Can we extend the validity of presigned URL for snowflake?
- Download or Move Snowflake Worksheet
- Snowflake - How to pivot 2 or multiple columns without aggregation
- Create snowflake view for a csv file stored on S3
- How can I implement this method to prevent from sql injection? Any help would be very much appreciated. Thank you all
- snowflake-csv fileformat to read only 2 line and rest of the data
- Save Snowpark DataFrame as text file in Snowflake Stage
Trending Questions
- UIImageView Frame Doesn't Reflect Constraints
- Is it possible to use adb commands to click on a view by finding its ID?
- How to create a new web character symbol recognizable by html/javascript?
- Why isn't my CSS3 animation smooth in Google Chrome (but very smooth on other browsers)?
- Heap Gives Page Fault
- Connect ffmpeg to Visual Studio 2008
- Both Object- and ValueAnimator jumps when Duration is set above API LvL 24
- How to avoid default initialization of objects in std::vector?
- second argument of the command line arguments in a format other than char** argv or char* argv[]
- How to improve efficiency of algorithm which generates next lexicographic permutation?
- Navigating to the another actvity app getting crash in android
- How to read the particular message format in android and store in sqlite database?
- Resetting inventory status after order is cancelled
- Efficiently compute powers of X in SSE/AVX
- Insert into an external database using ajax and php : POST 500 (Internal Server Error)
Popular # Hahtags
Popular Questions
- How do I undo the most recent local commits in Git?
- How can I remove a specific item from an array in JavaScript?
- How do I delete a Git branch locally and remotely?
- Find all files containing a specific text (string) on Linux?
- How do I revert a Git repository to a previous commit?
- How do I create an HTML button that acts like a link?
- How do I check out a remote Git branch?
- How do I force "git pull" to overwrite local files?
- How do I list all files of a directory?
- How to check whether a string contains a substring in JavaScript?
- How do I redirect to another webpage?
- How can I iterate over rows in a Pandas DataFrame?
- How do I convert a String to an int in Java?
- Does Python have a string 'contains' substring method?
- How do I check if a string contains a specific word?