Library Dependencies bundled together to create jar file is not present in the jar file using sbt-assembly

Question

I have multiple libraries dependency in the sbt.build file. I am creating the final jar file using sbt-assembly so it includes all the dependent libraries in the jar itself.

But using jar tvf jarname.jar, I am not able to find all libraries there.

I need this to bundle all libraries in a jar and provide it to spark-shell with spark-shell --jar jarpath and then using the import command to use the libraries there.

This is done because this is not possible for me to import packages directly to spark-shell using the spark-shell --packages command.

Expected: Adding the jar file to the spark-shell and then importing all libraries there which should be present in jar

Answer 1

Found the solution here:

Some of my dependencies includes the "provided" tag and thus it was not getting included in the fat jar.

libraryDependencies += "org.apache.flink" %% "flink-table-planner" % flinkVersion % "provided"