ubuntu scala ide - spark - toDF Method error value toDF is not a member of org.apache.spark.rdd.RDD[String]

50 Views Asked by At

i m facing issue with scala IDE (ubuntu), method toDF not working correcly see my code

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD

object ScalaTextSearch {
  def main(args: Array[String]) {
    val conf = new SparkConf()
      .setAppName("TextSearch")

    val sc = new SparkContext(conf)
    val texte = sc.textFile(args(0))
    val df = texte.toDF("line")

thanks

1

There are 1 best solutions below

0
Raphael Roth On

you need to import implicit conversions from SparkSession (assuming Spark 2.0 or higher) :

val ss = SparkSession.builder().appName("TextSearch").master("local").getOrCreate()
import ss.implicits._

val texte = ss.sparkContext.textFile(args(0))
val df = texte.toDF("line")