Bad return type in lambda expression when using Java's Optional.or() with subclasses

Question

I am trying to use Optional.or to get an object of subclass A or, if empty, an object of subclass B:

interface Node {}
class InnerNode implements Node {}
class LeafNode implements Node {}
Optional<InnerNode> createInnerNode() {}
Optional<LeafNode> createLeafNode() {}

Optional<Node> node = createInnerNode().or(() -> createLeafNode());

When doing this, I get the following compiler error:

Bad return type in lambda expression: Optional<LeafNode> cannot be converted to Optional<? extends InnerNode>

If I instead use wildcards to explicitly tell the compiler that the Optionals contain an object extending from Node:

Optional<? extends Node> optionalInner = createInnerNode();
Optional<? extends Node> optionalLeaf = createLeafNode();
Optional<Node> node = optionalInner.or(() -> optionalLeaf);

I get the following compiler error:

Bad return type in lambda expression: Optional<capture of ? extends Node> cannot be converted to Optional<? extends capture of ? extends Node>

The only way I found to make this work, is using an Optional.empty in front:

Optional<Node> node = Optional.<Node>empty() // requires generic argument
    .or(() -> createInnerNode())
    .or(() -> createLeafNode());

But for me, this is confusing to read. Is it somehow possible to instruct the compiler to allow the statement optionalInner.or(() -> optionalLeaf)? Or are there other alternatives to make this work?

Answer 1

The workaround you provided is a pretty simple answer, but you can convert an Optional<InnerNode> to an Optional<Node> trivially with map if you want to avoid a placeholder Optional.

createInnerNode.<Node>map(Function.identity()).or(() -> createLeafNode());

In general, this is also how you conceptually turn a List<Dog> to a List<Animal> or any generic collection to a different generic collection in java as well. It's somewhat annoying, but wildcards don't cut it because you want both read/write on the transformed collection.

The signature of or should tell you why your workaround works

Optional<T> or(Supplier<? extends Optional<? extends T>> supplier)

notice how it's already using the ? extends T in the supplier. Since the first empty Optional's T is Node, the or function can accept a supplier that returns

Optional<? extends Node>

and since both the innernode and leafnode are an Optional<? extends Node>, this works.

So the problem is that you need an Optional<Node> at some point.

I don't think you can avoid doing something since an Optional<InnerNode> never type matches Optional<Node> for reasons listed in other answers (and other questions on stack overflow).

Reading the other answer and trying

Optional<? extends Node> node = createInnerNode();

means that the T is a

? extends Node

which makes or seem to want doubly nested captures? I can't stop getting

Bad return type in lambda expression: Optional<capture of ? extends Node> cannot be converted to Optional<? extends capture of ? extends Node>

when trying basically any cast with the following code

Optional<? extends Node> node = Node.createInnerNode();
node.or(() -> (Optional<? extends Node>) null);

Answer 2

This must be a confusing error. Let me first explain the underlying problem so you actually understand why javac is complaining, and then I'll get to how to fix it.

The underlying problem is lack of use-site variance.

The error you get just doesn't make sense for Optional, but it makes perfect sense when we use a type that produces and consumes typed data (unlike Optional which, as an instance, only produces data, e.g. when you call .get() - there is no .set() method on Optional; it's immutable).

So let's look at lists. Imagine you could do this:

Integer i = 5;
Number n = i; // This is legal java.
List<Integer> ints = new ArrayList<Integer>();
List<Number> numbers = ints; // So this should be too... right?

That 4th line is not legal. Because generics are by default invariant (Neither supertypes nor subtypes will do; in an invariant typing system, if a Number is required, then only a Number will do) - and that is correct. After all, if the above code WAS legal, then everything breaks:

Double d = 5.0;
numbers.add(d); // Fine - must be. doubles are numbers!
Integer i = ints.get(0); // um.. wait. uhoh.

Because numbers and ints are both just referencing the exact same list, if I add a double to numbers, that means I also added one to ints, and - boom. It's broken.

Now you know why generics are invariant.

This also explains a few further compiler errors:

List<Integer> ints = new ArrayList<Integer>();
List<Number> numbers = ints; // nope, won't compile.
List<? extends Number> numbers = ints; // but this will!
numbers.add(anything-except-literally-null); // this wont!

With ? extends you are telling java that you want covariance (covariance = a subtype of X is just as good as X itself. Java's non-generics typing is covariant). However, covariance in generics automatically means that ALL ways to 'send' typed data to that thing are disabled. You CANNOT add anything (except literally .add(null) as null is all types) to a List<?> or a List<? extends>. This makes sense if you think about it:

A List<? extends Number> could be a List<Integer> or a List<Double>. So how can you add anything to this list? There is nothing that is both a Double and an Integer (well, except the literal null), so nothing you could possibly pass to add is neccessarily 'safe'. Hence why the compiler always tells you its a typing violation.

This also means trivially that this won't work either:

List<? extends Number> numbers = ...;
List<Number> numbers2 = numbers; // nope, this won't compile

Back to your code

Now, for optional, none of this seems relevant. Optional can be completely covariant in its type argument because Optional does not 'consume' data - none of its methods take in an 'E'. But the compiler isn't some sort of megabrain that susses this out and lets you 'break' the variance rules of typeargs. Not so - Optional needs to stick to the same rules list does. So, if an instance of List<? extends Number> cannot be assigned to a variable of type List<Number>, then the same applies to Optional:

An instance of type Optional<? extends Node> cannot be assigned to a variable of type Optional<Node>

And that explains your errors (for both of the attempts to solve the problem stated in your question).

Even though for optional specifically these errors don't make much sense. Now you know why you are getting them.

The best fix is to not use optional here; this isn't really what its meant for and it's not very java-like (see below). But the distant second best option is to fix this stuff. The root fix is that all places that take in an Optional should ALWAYS be stated as ? extends. So, you're trying to assign it to a field of type Optional<Node>, but that field is 'wrong'. That field's type should be Optional<? extends NOde>. Then that fixes all problems:

Optional<? extends Node> node = createInnerNode().or(() -> createLeafNode());

Works fine (that's your first snippet, but with the 'type' of the target fixed).

Ah. But, I can't change the API

Well, then the API is just wrong. It happens - sometimes you need to interact with broken code. But make no mistake, that is broken code. You do the same thing you do with any other broken code: Encapsulate and work around.

You don't want to interact with broken API. So make a wrapper type or helper methods that 'paper over' the error. And in that code (the 'bridging' code that works around the buggy code you cannot change), you accept that you need to write wonky code that generates warnings.

You can 'fix things' as follows:

public Optional<Node> fixIt(Optional<? extends Node> node) {
  Optional raw = node;
  return raw;
}

This works because generics in the end is just a figment of javac's imagination - the JVM doesn't know what generics are at all. So we just need to 'fake out' javac to stop refusing to compile the code. The above WILL generate warnings and any errors in such hackery result in getting ClassCastExceptions on lines with zero casts on them - quite confusing. But, works fine if you know what you're doing. You can eliminate the warnings with a @SuppressWarnings annotation.

Or.. just don't use optional

Java has null baked into many many places. Not just in its core model (fields that aren't initialized begin with null values. Arrays of non-primitives are initialized with null values), but also in a great many of its most crucial libraries. The most obvious and simple example is java.util.Map: It has a get() method that returns a V. Not an Optional<V>. And in a world where Optional is the right answer, get() should of course return Optional<V>. But it does not, and it never will - java doesn't break backwards compatibility that drastically.

Thus, the hypothetical world of 'optional everywhere!' sucks - 30 years of built up code and history needs to be tossed in the bin first and the community is rather unlikely to do that. If you dislike null handling in java (and there are plenty of fine reasons to do so!), then a solution is required that lets existing code backwards-compatibly adjust, and Optional cannot deliver. (nullity annotations probably can, though! Much better; or just write better APIs. Map has .getOrDefault which can be used to ensure your map query calls don't ever return null in the first place. Once you start writing your code to never produce null values except in cases where you truly WANT any attempt to deref to throw (i.e. very rarely), then null ceases to be a problem. These more modern addons to the various java core library types make it much easier to write such code. And adding methods to interfaces is backwards compatible, so, as java.util.Map shows, existing stuff can backwards-compatibly add these).

The best thing to do with Optional is to use it solely for why it was introduced in the first place: As a return value for stream API terminals. Something like intList.stream().max() returns an OptionalInt because what is the maximum of an empty list? Its fine enough for that. If you write your own collectors or stream terminating operations, use Optional. If that's not what you are doing, don't use it.