will Spark support Clojure?

Question

I am about to start learning functional programming and Clojure appeals to me the most, I love its community, syntax and concept of immutable data structures. I am also interested in bio inspired ML for rich data Numenta. However, my huge concern is that Spark does not support it as yet, and Spark rocks!!! There is a Flambo Flambo Clojure ,but is it a satisfactory solution?

My course and job is in Scala. Should I defeat and enter Scala world or do you think that I should focus solely on Clojure?

Answer 1

Being the author or Sparkling (thanks to Josh Rosen for pointing that out), I can tell you that we use it at our company for ETL processing.

Here's what's good:

• it provides a Clojuristic way of interacting with Spark, and as you can see in the presentation "Big Data with style - the Clojure/Spark way"
• it's optimized for performance
• it's used in production and there are others also using it

Here's what's missing:

• There's currently no support for Spark Streaming, Spark SQL, Spark Dataframes or Spark ML. That might come in the future, I'm happy to accept pull requests, but it's currently not main focus (at the time of writing, April 2015).

I hope this helps you make up your mind on going with Clojure or starting to learn Scala.

Answer 2

It's hard to say that something like Spark doesn't support Clojure. It would make more sense to ask if there are good libraries to use that project that are easy to use from Clojure. From googling around Flambo looks like a viable option and at the various clojure conferences I hear incidental talk of using Spark in several contexts.

I would say that there is fairly low technical risk in using spark from Clojure so you are free to make this choice based on the other constraints of your working environment and pojects. Being particularly biased toward Clojure I whole heartedly recommend at least trying it and see what parts of the language and ecosystem work well for you.