List questions from scala-spark | Devhide

org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus can't be cast to org.apache.spark.sql.execution.datasources.FileStatusWithMetadat

Published on 14 March 2024 at 08:15

How to setup and run scala-spark in intellij?

Published on 10 January 2024 at 10:00

Scala Spark distributed run on Google Cloud Platform but worker not working

Published on 10 December 2023 at 10:31

How to remove duplicate value from parquet file

Published on 20 November 2023 at 15:02

Spark write working with memory errors in Shell but not with spark-submit

Published on 17 November 2023 at 07:29

Explode nested list of objects into DataFrame in Spark

Published on 10 November 2023 at 23:58

Access newly created column in withinColumn

Published on 25 October 2023 at 07:17

Spark Array column - Find max interval between two values

Published on 15 September 2023 at 11:52

Divide a column value into multiple rows by number of months based on start date & end date columns

Published on 02 September 2023 at 13:43

Scala - Create Dataframe with only 1 row from a List using for comprehension

Published on 11 July 2023 at 21:29

How to improve spark filter() performance on an array of struct?

Published on 05 July 2023 at 05:29

AWS Glue scala spark job failing - org.apache.spark.util.collection.CompactBuffer[] not registered in Kryo

Published on 30 June 2023 at 04:24

Spark extract values from Json struct

Published on 30 June 2023 at 03:23

Spark broadcasts right dataset from left join, which causes org.apache.spark.sql.execution.OutOfMemorySparkException

Published on 02 June 2023 at 15:31

How to apply custom logics on spark dataframe using scala

Published on 25 May 2023 at 07:27

Performance Implications of spark read

Published on 04 May 2023 at 12:54

MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-27c9b887c306cb9c-conf-map" not found

Published on 27 April 2023 at 14:29

Exporting file from Google Dataproc Job

Published on 21 April 2023 at 12:55

Convert Vector[String] to Dataframe in Scala Spark

Published on 29 March 2023 at 14:38

spark scala exploding struct array throwing error ambigous reference to fields

Published on 29 March 2023 at 06:16