Spark/Scala transform fails in FindBugs

Question

FindBugs is reporting a "Null pointer dereference of ?" in the following Spark/Scala transform:

val df = Seq((10,20)).toDF
df
  .transform(addCol1) // <-- Null pointer dereference of ?

def addCol1(df: DataFrame): DataFrame =
  df.withColumn("col1", lit("a"))

Report:

Null pointer dereference of ? in new MyClass$...

Bug kind and pattern: NP - NP_ALWAYS_NULL

I don't understand what might be null in this case. Also, the ? may be a lead.

Does anyone have an hint?

Answer 1

Hope below code snippet will help.

scala> val df = List(1,2,3).toDF("id")
df: org.apache.spark.sql.DataFrame = [id: int]

scala> df.show
+---+
| id|
+---+
|  1|
|  2|
|  3|
+---+


scala> :paste
// Entering paste mode (ctrl-D to finish)

def addCol1(df: DataFrame): DataFrame =
df.withColumn("col1", lit("a"))

// Exiting paste mode, now interpreting.

addCol1: (df: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame

scala> df.transform(addCol1).show
+---+----+
| id|col1|
+---+----+
|  1|   a|
|  2|   a|
|  3|   a|
+---+----+