Comparing Apache Livy with spark-jobserver

Question

I know Apache Livy is the rest interface for interacting with spark from anywhere. So what is the benefits of using Apache Livy instead of spark-jobserver. What are the drawbacks of spark-jobserver for which Livy is used as an alternative. And I couldn't find much on this on the internet. Can you please help me to get clarity on this.

Thanks,

Answer 1

As of September 2018, the only major difference between Livy and the (Ooyala) Spark Job-server is the fact that although (the current version of) Livy is built against Spark 2.2.0, the version of Spark used when running Livy does not need to match the version used to build Livy. Livy internally handles the differences between different Spark versions.

This behaviour is different from Spark Job-server which has tight coupling with the Spark version it is built against.

Answer 2

There are a couple of major differences that were relevant to my use case.

Livy's advantages:

• Livy does not require any changes to your code, while SJS jobs must extend a specific class.
• Livy allows submitting code snippets as well as precompiled jars, while SJS only accepts jars.
• In addition to REST Livy has a Java and Scala APIs. A Python API is in development, SJS has a "python binding"

SJS Advantages:

• SJS can manage the jars as well. It allows you to upload and store Jars, then deploy jobs from these jars with a separate REST call. Livy requires the jar whenever you need to deploy a job.
• SJS jobs can be configured with HOCON format which can be submitted as part of the REST call.

Additionally, SJS has better documentation, although in both cases, it's not comprehensive. And of course, keep in mind that both projects are pre v1, so things could change quickly.

In my case we ended up going with SJS since I had no use for submitting snippets, and Jar management and HOCOCN configuration came in handy. I am, however, considering revisiting Livy in the near future for a more thorough evaluation.

Sources:

• https://livy.incubator.apache.org/
• https://github.com/spark-jobserver/spark-jobserver
• Livy presentation: https://www.youtube.com/watch?v=C_3iEf_KNv8

Answer 3

For people who read this after 4 years, there is no support for Spark 3.x for Spark Jobserver. So with the tightcoupling of Spark version as mentioned above, this seems to be a dead-end in 2023.