While trying to create a jar for a Flink Batch program to runs on EMR, I am running into conflicts with versions of flink-shaded-guava jar.
pom.xml
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-clients_2.11</artifactId>
<version>1.14.6</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-core</artifactId>
<version>1.18.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-jdbc_2.11</artifactId>
<version>1.9.3</version>
</dependency>
dependency tree
[INFO] +- org.apache.flink:flink-clients_2.11:jar:1.14.6:compile
[INFO] | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for conflict with 1.18.1)
[INFO] | +- org.apache.flink:flink-runtime:jar:1.14.6:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-rpc-core:jar:1.14.6:compile
[INFO] | | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-rpc-akka-loader:jar:1.14.6:compile
[INFO] | | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-rpc-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | | \- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-java:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-queryable-state-client-java:jar:1.14.6:compile
[INFO] | | | +- (org.apache.flink:flink-shaded-netty:jar:4.1.65.Final-14.0:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-shaded-guava:jar:30.1.1-jre-14.0:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-hadoop-fs:jar:1.14.6:compile
[INFO] | | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | +- commons-io:commons-io:jar:2.8.0:compile
[INFO] | | +- org.apache.flink:flink-shaded-netty:jar:4.1.65.Final-14.0:compile
[INFO] | | +- (**org.apache.flink:flink-shaded-guava:jar:30.1.1-jre-14.0:compile - omitted for duplicate)**
[INFO] | | +- (org.apache.flink:flink-shaded-jackson:jar:2.12.4-14.0:compile - omitted for conflict with 2.14.2-17.0)
[INFO] | | +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.4.14-14.0:compile
[INFO] | | +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] | | +- (commons-cli:commons-cli:jar:1.3.1:compile - omitted for duplicate)
[INFO] | | +- org.javassist:javassist:jar:3.24.0-GA:compile
[INFO] | | +- org.xerial.snappy:snappy-java:jar:1.1.8.3:compile
[INFO] | | +- org.lz4:lz4-java:jar:1.8.0:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-optimizer:jar:1.14.6:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-runtime:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-java:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-shaded-guava:jar:30.1.1-jre-14.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-shaded-jackson:jar:2.12.4-14.0:compile - omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-java:jar:1.14.6:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- (org.apache.commons:commons-lang3:jar:3.3.2:compile - omitted for duplicate)
[INFO] | | +- org.apache.commons:commons-math3:jar:3.5:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] | +- org.apache.flink:flink-streaming-java_2.11:jar:1.14.6:compile
[INFO] | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-file-sink-common:jar:1.14.6:compile
[INFO] | | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-runtime:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- org.apache.flink:flink-scala_2.11:jar:1.14.6:compile
[INFO] | | | +- (org.apache.flink:flink-core:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-java:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | | +- (org.apache.flink:flink-shaded-asm-7:jar:7.1-14.0:compile - omitted for duplicate)
[INFO] | | | +- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] | | | | \- (org.scala-lang:scala-library:jar:2.11.12:compile - omitted for duplicate)
[INFO] | | | +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO] | | | +- org.scala-lang:scala-compiler:jar:2.11.12:compile
[INFO] | | | | +- (org.scala-lang:scala-library:jar:2.11.12:compile - omitted for duplicate)
[INFO] | | | | +- (org.scala-lang:scala-reflect:jar:2.11.12:compile - omitted for duplicate)
[INFO] | | | | +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.5:compile
[INFO] | | | | | \- (org.scala-lang:scala-library:jar:2.11.7:compile - omitted for conflict with 2.11.12)
[INFO] | | | | \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] | | | | \- (org.scala-lang:scala-library:jar:2.11.6:compile - omitted for conflict with 2.11.12)
[INFO] | | | +- com.twitter:chill_2.11:jar:0.7.6:compile
[INFO] | | | | \- com.twitter:chill-java:jar:0.7.6:compile
[INFO] | | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-java:jar:1.14.6:compile - omitted for duplicate)
[INFO] | | +- (org.apache.flink:flink-shaded-guava:jar:30.1.1-jre-14.0:compile - omitted for duplicate)
[INFO] | | +- (org.apache.commons:commons-math3:jar:3.5:compile - omitted for duplicate)
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.15:compile - omitted for duplicate)
[INFO] | | +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | | \- (org.apache.flink:flink-shaded-force-shading:jar:14.0:compile - omitted for duplicate)
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.15:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | \- org.apache.flink:flink-shaded-force-shading:jar:14.0:compile
[INFO] +- org.apache.flink:flink-core:jar:1.18.1:compile
[INFO] | +- org.apache.flink:flink-annotations:jar:1.18.1:compile
[INFO] | | +- (org.slf4j:slf4j-api:jar:1.7.36:compile - omitted for conflict with 1.7.15)
[INFO] | | \- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-metrics-core:jar:1.18.1:compile
[INFO] | | \- (org.apache.flink:flink-annotations:jar:1.18.1:compile - omitted for duplicate)
[INFO] | +- org.apache.flink:flink-shaded-asm-9:jar:9.5-17.0:compile
[INFO] | +- org.apache.flink:flink-shaded-jackson:jar:2.14.2-17.0:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.12.0:compile
[INFO] | +- org.apache.commons:commons-text:jar:1.10.0:compile
[INFO] | | \- (org.apache.commons:commons-lang3:jar:3.12.0:compile - omitted for duplicate)
[INFO] | +- com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] | | \- org.objenesis:objenesis:jar:2.1:compile
[INFO] | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | +- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] | +- **org.apache.flink:flink-shaded-guava:jar:31.1-jre-17.0:compile**
[INFO] | +- (org.slf4j:slf4j-api:jar:1.7.36:compile - omitted for conflict with 1.7.15)
[INFO] | \- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted for duplicate)
[INFO] +- org.apache.flink:flink-jdbc_2.11:jar:1.9.3:compile
[INFO] | \- org.apache.flink:force-shading:jar:1.9.3:compile
[INFO] +- org.apache.flink:flink-s3-fs-hadoop:jar:1.18.0:provided
[INFO] \- org.junit.jupiter:junit-jupiter-api:jar:5.10.1:test
[INFO] +- org.opentest4j:opentest4j:jar:1.3.0:test
[INFO] +- org.junit.platform:junit-platform-commons:jar:1.10.1:test
[INFO] | \- (org.apiguardian:apiguardian-api:jar:1.1.2:test - omitted for duplicate)
[INFO] \- org.apiguardian:apiguardian-api:jar:1.1.2:test
Tried adding relocations to handle conflicting versions but this isn't working and results in the dependency tree shared above
<relocations>
<relocation>
<pattern>org.apache.flink:flink-shaded-guava:30.1.1-jre-14.0</pattern>
<shadedPattern>org.apache.flink.shaded.guava30</shadedPattern>
</relocation>
</relocations>
What am I missing?
You can't mix and match Flink versions: they all need to be the same (or compatible with that Flink version, when it comes to externalized Flink connectors)