I am new to sbt. I have a quite simple project config as follow:
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.12.15"
assemblyMergeStrategy in assembly := {
case PathList("META-INF", _*) => MergeStrategy.discard
case _ => MergeStrategy.first
}
lazy val root = (project in file("."))
.settings(
name := "sifi_pipeline"
)
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "3.3.2" % Provided,
"org.apache.spark" %% "spark-sql" % "3.3.2" % Provided,
"org.apache.spark" %% "spark-streaming" % "3.3.2" % Provided,
"org.apache.spark" %% "spark-sql-kafka-0-10" % "3.3.2",
"com.google.cloud.bigdataoss" % "gcs-connector" % "1.4.3-hadoop2"
)
In my understanding provided scope means you make sure such dependencies are provided in class path and will NOT be include in the fat jar to be generated.
Here I have 5 dependencies, no matter how I config the provided scope. The assembly result were same in size. I even tried to add provided to all of them, still the same. Do I misunderstand provided?