Should Enums be backwards compatible?

[Maxim Zaks]

Hi Group,

i stumbled upon the fact that if you add a new enum case you might lose backward compatibility.

Lets say we have 

enum E1 : byte {A, B}
table T1 {
  e : E1
}

and it got changed to 

enum E1 : byte {A, B, C}
table T1 {
  e : E1
}

if I would have an old client which would read new state which have E1.C stored it what the result would be?

Funny fact is that I stumbled upon it, because I write a Swift port. And in Swift it will turn unknown value into a nil.

let e3 : E1? = E1(rawValue: 2)
XCTAssert(e3 == nil)

Which is kind of ok I guess. If the old client will than save the state again it will erase the fact that e was set to E1.C 

In C# the cast will work out of the box without the client losing data. (http://stackoverflow.com/questions/6413804/why-does-casting-int-to-invalid-enum-value-not-throw-exception)

And honestly I am not sure what will happen in another languages.

Is it an issue, Should we have something like UNKNOWN case in every enum definition?

[Wouter van Oortmerssen]

What you describe is a problem with forwards compatibility actually (backwards compatibility is a newer program reading old data, which is usually easier). And yes, this breaks forwards compatibility, but in a mostly benign way.

Generally, in most code bases, enums are things that are expected to grow, meaning you need to deal with unknown values gracefully (either an error or ignore it).

An enum is an integer value (even in Java, we don't use enum classes). Implementations should cast this integer to the enum type. The result for unknown values is thus a value of an enum type for which the current program will not have an enum identifier, thus can't be explicitly tested against (will end up in the "default" of a switch).

--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maxim Zaks]

Hi Wouter,

Am Montag, 4. Januar 2016 22:46:23 UTC+1 schrieb Wouter van Oortmerssen:

What you describe is a problem with forwards compatibility actually (backwards compatibility is a newer program reading old data, which is usually easier). And yes, this breaks forwards compatibility, but in a mostly benign way.

Sure sorry for mixing them up. 

Generally, in most code bases, enums are things that are expected to grow, meaning you need to deal with unknown values gracefully (either an error or ignore it).

Totally agree, this is why I started this thread.

 

An enum is an integer value (even in Java, we don't use enum classes). Implementations should cast this integer to the enum type. The result for unknown values is thus a value of an enum type for which the current program will not have an enum identifier, thus can't be explicitly tested against (will end up in the "default" of a switch).

Converting an unknown value into nil means that I am losing data. Which is bad, but I am also hesitant about old client overwriting new states. Reading is fine but writing back is very questionable.

I guess I will stick with nil for now. It would be easy to change, because the base reader reads an int. It is only the generated code which converts the int to an enum.

Btw. here is the infrastructure code for the FlatBuffersSwift

https://github.com/mzaks/FlatBuffersSwift/blob/master/FlatBuffersSwift/FlatBufferBuilder.swift

https://github.com/mzaks/FlatBuffersSwift/blob/master/FlatBuffersSwift/FlatBufferReader.swift

I simplified the Builder implementation and introduced a generic Reader class.

I use those two classes directly in the generated code.

Right now I am focusing on API design. Next step will be performance optimisation.

[Wouter van Oortmerssen]

I would advice against turning them into nil.. they should really just be cast to the enum type.

[Maxim Zaks]

This is however how enum type works in Swift :)

in C, C++ and lots of other languages enum type is just a syntactic sugar for an integer number.
When you use it in a switch statement the value is handled as an integer number, without any special checks.

In Swift enum is a special type which can do lots of crazy stuff (https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/Enumerations.html)

And if you use it in a switch statement compiler will force you to provide exhaustive handling, and will show a warning if some path won't execute.

this is why calling 

MyEnum(rawValue: 7)

is the only way to convert an int to an enum. This method is a so called Failable Initializers

It returns an Optional<MyEnum> or MyEnum? for short. This means that it is either a nil or a valid case of MyEnum

This implies that the generated table class also has to return MyEnum? as enum property type.

This way the user knows that s/he has to unwrap the value before it can be used.

For enum, the value can be nil only in case of forward compatibility problem.

In this case an old client which reads a state generated by a new client want crash, but will lose the information stored in the enum because it can't handle the new case anyways.

It is just unwise for the old client to overwrite the state which was generated by a new client. But this should be true also for other situations. (e.g. a new property was added to a table)

This is why I think that going with Optional<MyEnum> and not just MyEnum as enum property type, is a good idea.

[Wouter van Oortmerssen]

Yes, same in Java where enums are not really compatible with integers, which is why in Java we use "const int" rather than the built-in enum type.

I suggest you do the same in Swift. FlatBuffers defines enums to be integers, so should each implementation.

[Maxim Zaks]

I would like to disagree on this matter. By using "const int" in Swift I would take away some comfort and security from the user.

Switch statements on an enum value has to be exhaustive. Otherwise the code does not compile.

This kind of error can help user to identify bugs which can occur while they introduce new cases for enums.

A switch case on an int has to provide a default case, because there is no way of figuring out if the switch statement is exhaustive. 

This means that new values will fall through and cause a logical bug.

As I read through posts about Java and in C++ exhaustive switch behaviors, I see that C++ and Java compilers provide a warning, but it kind of weird. 

I keep seeing that people have to provide default case anyways. This defeats the whole idea of secure switch statement.

This is why I feel like Swift is pretty much different from C++ and Java. 

One more thing is a bit sad. A table which provides an optional scalar property is returning a default value. And the default default value is 0 or false for boolean.

This implies that it is not really an optional value, as it is impossible to say if the data was actually set to default value, or it was not present in the first place.

Swift has an Optional type which fits perfectly for this problem, however because of the default values I can't use it for scalars.

I mentioned before that I had a feeling that ports of FlatBuffers are a bit C++ centric. I guess it is because of such fine nuances.

[Wouter van Oortmerssen]

I agree that enum safety is a nice thing to have, but the problem is that Swift's enums are implemented in a way that is not compatible with how we do thing in other languages. It is not helpful to people using FlatBuffers cross-platform that Swift represent the data in a different way, or a slower way.

FlatBuffers isn't necessarily C++ centric, it is low-level, and requires that its implementations work at a bits and bytes level. There is no way we could represent enums or optional values in a high level way such a way that it would make it equal in all languages without overhead.