Optional

[Christian Sterzl]

So after revealing the decisions around the next step could you please name some arguments against Optional? Not that I want any special support in lombok for it, but I like the concept of Optional like in other languages as well and never heard any good arguments against.
I don't want to start a flame war. I am just interested and want to learn something I may have not considered until now.
I googled already for arguments against but didn't find any compelling.

[Martin Grajcar]

I'm about to write a lengthy answer as I have a strong opinion on this.

--
You received this message because you are subscribed to the Google Groups "Project Lombok" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-lombo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robbert Jan Grootjans]

I'm not sure exactly if you want general support for Optional in lombok, or that you are interested in adding support in a specific piece, or perhaps you're referring to the recent Builder 2.0 proposal by Reinier. Lombok is built towards removing existing boilerplate, so it seems logical that any new proposal will amend the existing style, and not take a new approach.

I'm sure you're not implying we have a general discussion on the merits of Optional on this group, there are of course other, more suited place to do that.

On Thu, Dec 4, 2014 at 5:12 PM, Christian Sterzl <christia...@gmail.com> wrote:

[Reinier Zwitserloot]

Optional is a bad idea regardless of language, but trying to retrofit it into java means you get all the downsides of Optional in general, and a few additional downsides to boot.

* Optional is infectuous: It just doesn't work right unless EVERYTHING you do is optional based, or nothing is. Even if you think Optional is an improvement over null, surely I don't need to elaborate that it's bad all around if about half of the APIs you use return null to indicate lack of value, and the other half return Optional. This is a general problem with Optional, but specifically in regards to java: It would take a really, _REALLY_ long time for all libraries to update, so for about 10 years it would just be worse, and that's assuming every library builder is 'on board'. I bet many (including myself) would just say no to optional and never port over their stuff. Effectively, someone has to make the royal decree from up on high that Optional is to be used instead of null, everywhere, effectively immediately, please update your APIs ASAP, do it now, don't argue. Who has that kind of power? Nobody does; it would split the community and the pieces will never come back together again. At least with generics, there was nobody who thought APIs without generics would be better than APIs with generics (some thought generics were not a good idea, but _given_ that the language has it, all agreed that you should then use it if appropriate for your API. Unlike Optional, where the language has it, and has had it for years (ever since generics and foreach, Optional is feasible in java if you really want it), but many do not agree that it's good for your APIs.

* A lot of existing interfaces can't change. In particular, name me a better obvious candidate for returning Optional<T> than map.get(). I can't think of a single API where it would be more appropriate than that one. And yet, that one will not be returning Optional. EVER. Java has to be backwards compatible. At best, a new method (getOptional I guess? I can't even think of a good name for this!) can be added that uses default functionality. Note also that this is dubious; if I .get(X) a key which IS in the map, but was added with .put(X, null), then the fact that I get an Optional.NONE is a bit weird. I would think that means the key isn't in the map at all, but that would be wrong.

* Optional is not a valid container type. What I mean by that is, you can't let an Optional<T> continue to exist in your codebase, you have to unpack it immediately, every time. If you did let it continue to exist, then you can't use any API. Let's say, for example, that I have a method that takes a list of strings. Sometimes, such a method must communicate to its callers that it can't handle null values. Sometimes, it must communicate that is explicitly designed to handle nullable strings. But more usually you have a utility method that just doesn't care. It can handle lists of non-null strings perfectly fine (read: It never adds 'null' to this list), and it can also handle lists of nullable strings perfectly fine (read: It will null-check when it reads and do something appropriate). What is the signature of this method? Here's the clincher: You can't write this method with optional! I can take a List<Optional<String>>, but List<String> is not compatible with this, or I can take a List<String>, but I can't pass a List<Optional<String>> to it. In practice, you don't _WANT_ Optional anywhere EXCEPT in return types, which means all returned optionals must be immediately unpacked, because no other APIs (collection utility methods, etcetera) can be passed optionals. language-level nonnull markers (such as fantom-style String! vs. String?) are better at this, but absolutely none of these really did the legwork on thinking it through. There are 4 nullities: There's definitely-not-null, there's definitely-can-be-null, there's "legacy" (analogous to java's raw types, only relevant for languages in transit), and there's "It might or might be valid to have this be null". That last one sounds the same as @Nullable, but it isn't, for the same reason List<Number> and List<? extends Number> aren't actually the same. I CAN put nulls into a List<@Nullable String>, but the compiler should stop me from adding nulls to a List<@MayOrMayNotBeNullable String>. Similarly, the compiler should freely let me dereference-without-nullchecking elements from a List<@NonNull String>, but it should force me to nullcheck a List<@MayOrMayNotBeNullable String>. Outside of generics you don't need @MayOrMayNotBeNullable, but we're not gonna remove generics from java, so, you need it. This is just a highlight on how a different take on optional isn't simple either.

* Optional is just plain worse. NPEs are just plain better. The usual strategy when unpacking optionals is to do nothing when the optional is none. In scala you have pattern matching and it's somewhat easy to supply code to run in case of the NONE case, but in java we don't have it. So, how to 'unpack' an optional? One commonly used trick for optoinal-afficionados in java is to foreach over an optional, i.e. have optional implement iterable, and the iterators that call makes either have 0 elements in them in the NONE case, or one element in the SOME case. But that just means that the most idiomatic way to unpack an optional is to SILENTLY do NOTHING in the NONE case. We're talking specifically about the case where the coder kinda sorta feels that NONE can't happen (and there will be PLENTY of calls into Optional<T> returning methods where the coder thinks that the NONE case won't actually ever occur  because they know exactly what they're doing, such as looking up a key from a template engine out of an internal resource, where it would be considered a straight up bug in the NONE case). NPE is vastly superior, because the default action if the coder didn't wish to consider NONE case should be an exception. It's nice that APIs can communicate more clearly that a NONE return value is possible, but that's not what this point is about. Once a programmer calling such a method has decided to not spend much time thinking about the NONE case, an exception if NONE does occur is much better than silently do nothing.

* Optional is very wordy.

* null won't ever actually go away. While most scala APIs use optional, null still exists, and for example fields that aren't initialized start out as null, and object arrays are initialized to null. It would be extremely difficult to update the java spec for them all to initially point at the Optional.NONE singleton instead. It can't be done without breaking all old libraries, for example.

* The 'problem' isn't worth fixing. NPEs are NOT a big deal. I have yet to meet an NPE that would both (A) never have happened if, during writing some code, my editor would have explicitly forced me to acknowledge the NONE case, and (B) took me more than 5 seconds to find and fix. NPEs are great: They point at the problem ,and the problem is basically always extremely clear. On the contrary, issues arising from optional are often much harder to find, because the 'problem' (Heyyy, this optional is NONE and I don't understand how that could have happened!) occurs at a point where the 'stacktrace' of the original violation of expectation is long gone. This gets a lot better if you make it a point of unpacking all optionals immediately, but that's far less 'nice' than optional fans tend to paint the picture. In particular, nulls are kept around for a long time (passing them around, etc, treating it as a transparent entity) ALL THE TIME in java code right now, and all that code would have to be updated to more explicitly acknowledge NONE as something you explicitly handle (with a flag or some such I guess, or a default value), that's a LOT of extra code! Also, how often have you ran around with your hair on fire because production code crashed with an NPE that would have been prevented with Optional? (Note that a lot of NPE-production-crashes would not have been helped by Optional, because the assumption was that the NONE case couldn't occur. Your code crashes just as hard if map.get() returned an Optional<T> if your code explicitly derefs the Optional because you thought NONE couldn't happen!) Given that serious application of Optional requires a TONNE of work to 'fix' legacy code, it better be totally worth it. Generics at least had tons of upside to offer as a tradeoff for the considerable downside of having to split the community in two for YEARS until all libraries caught up. Optional gets nowhere near.

* Optional in general is a full blown type and it just shouldn't be. If I use reflection to inspect some method's return type, getting back "Optional.class" is utterly useless to me and effectively a 'lie'. That's not really what that method returns, it returns the generics type. You CAN still get this with reflection (those aren't erased), but it's really difficult to do. Optional is really, really ugly. It's sometimes called 'elegant', but that's only by people who consider a language solving a problem based solely on the basics of the language design itself, instead of an explicit fix built into the language design, 'elegant'. I don't. Generally 'scalable languages' require inferior solutions. After all, it HAS to fit in the language basics, because adding an explicit bit of spec to deal with it is, according to this design principle, 'inelegant'. If I can add new language syntax I can always do a better job.

* Optional is painting with far too broad a brush: For the vast majority of API calls where one should handle the case of "well, one potential return value of this method is 'NONE'", you should actually create 2 overloaded calls. There should be map.get(key), sure, but there should also have been a map.get(key, defaultValue), specced as returning defaultValue if key is not present in the map. With such an API, you don't need optional, and there's no more 'silently ignore NONE' at all: In most cases, the appropriate solution to a NONE value is to use some known (to you) default instead. Sure, java generally does NOT have these APIs. But defending Optional as 'it fixes crappy APIs' is not fair unless we then also add into consideration that people will misuse Optional (if 'Optional can fix crappy API' is a valid argument, then a counter argument of 'Optional will lead to brand new crappy APIs' is just as relevant!). And they will: I'm sure we'd see tons of Optional<X> in method parameters, and even wintered scala fans (especially wintered scala fans, in fact!) argue that Optional shouldn't ever appear there. Just check out scala's core libraries; Optional is in tons of return types, but almost never shows up in a parameter spec, as it should be.

* Given that introduction of optional is a humongous, herculean undertaking, I can think of far better options if you really really MUST solve the null issue with the language and type system. For example, imagine that we abolish nulls altogether, and instead all types (even interfaces) can choose to declare a default value, which should be immutable. strings will pick "" (the empty string) as default, Integer would pick 0, List would pick Collections.emptyList, InputStream would perhaps do a dummy stream that always acts as if EOF has been reached and ignores close calls, but some types, such as java.lang.Object itself, would not HAVE a default choice. Where-ever java currently initialized to null (object arrays, initial field value), the default is used instead, and if the type has no default, then it is a compilation error to try and create an array of this type without specifying the 'fill' value, or to define a field without an initializer. You now have gotten rid of null without ever having to mess with optional. In many cases, the 'right answer' is to work with sensible defaults and avoid null altogether. For example, if I have an API that reeturns a collection of all 'steves' in a population of people, the right response if no steves are in the population are to return an empty list, NOT to return null or Optional<List<Person>>. So, optional isn't much of a fix, this would be. I don't advocate this either, but, hey, if we're going to move heaven and earth to abolish nulls, lets at least go with something awesome and useful instead of a 'fix' that doesn't really do anything useful.

 --Reinier Zwitserloot

On Thu, Dec 4, 2014 at 5:12 PM, Christian Sterzl <christia...@gmail.com> wrote:

[Marius Kruger]

With regards to improving crappy APIs, it may also make sense to support @NonNull/@NotNull on methods so that it can fail fast if for some reason/bug the method returns null. i.e. explicitly declare NonOptional returns (since Object return types are implicitly optional by default:)

[Martin Grajcar]

Well, Reinier's super-post made my effort pretty obsolete.