pyspark hive - Insert NULL as DB null through text file

Question

While inserting text file from pyspark shell to hive table.

NULL values treating as string in table.

If i query hive table, records can be retried only with filter condition = 'NULL' rather than is null.

Can any one suggest how to insert data as DB NULLS in table

Answer 1

Check if your spark dataframe is having null or None. And while writing to the hive table set the nullValue option as

df.write.option('nullValue', None).saveAsTable(path)

This will solve your issue.