[erlang-questions] Erlang 21, Stability and the murder of an innocent Statemachine

[Heinz N. Gies]

Since this might get long I want to start with an anecdote.

When talking to people about Erlang and why it is great one of the things I mention is stability. I’m not talking about systems not crashing but rather about the fact that change are slow, gentle and breaking changes are rare or nearly unheard off. The example I use is a non trivial toy program I wrote back in 2012 and that it took me a whole of 20 minutes to get code form 2012 working 4 and a half years later  (most of the time spent on updating dependencies).

Having used other languages and the constant catch up game with changes I always found that impressive  and it gives me a warm and fuzzy feeling inside that the code I finish will be finished and don’t require a endless cat and mouse with the language.

Even changes that happen like random vs rand are well based with reason - security and performance in that case, and the changes are minor enough that for the most case changing a single function call here and there is enough to even keep complex programs as long as they’re not (ab)using internal knowledge of the implementation (I’m looking at you riak_core).

This stability has always be a core value of the Erlang eco system to me, and an important one.

While I have the feeling the frequency of deprivations and breaking cross version compatibility has increased over the last releases for most of them I can, as said before, see the reason and the tradeoff made.

Now lets talk about senseless murder of gen_fsm.

I understand that gen_statem is nicer, and more user friendly, and I don’t want to advocate against it at all, it’s great that we have it and I see forward to useing it for the next state machine I have to write!

The next one. I see forward using it for the next one, not the last 100, or even last 15 not even for the last two. I don’t see forward to being forced to go through all libraries that have perfectly fine working gen_fsm without problems or troubles and have to change them.

Ripping out a widely used and not broken behaviour just because there is a more fancy one goes against everything I value in erlang as an eco system. It forces developers to do the kind of busy work I detest in other platforms.

Worst of all forcing people to fundamentally re-write perfectly good code means bugs, bugs we wouldn’t have had if we would just have left the code alone. Bugs that are probably non obvious as differences are subtile. And in the end for absolutely no benefit.

Now it’s just a rant without a proposal. And I detest whining without constructive criticism as much as I detest platforms forcing changes that are not required. So here is what I’d like to suggest:

- Keep gem_fsm around, it has been a friend to many for a long time and there is a lot of code using it that shouldn’t have to change to use newer erlang versions.

- Encourage people to use gen_statm for new code, and if it is really that much better then gen_fsm that it’d warrant to rip it out that should happen on it’s own.

Heinz - member of team gen_fsm 😂

PS: I’m not just writing this out of no-where I’ve been fighting subtile bugs on gen_fsm -> gem_statem code that was ported by people a lot smarter then me and I see no chance in hell that if they can’t pull that off on simple code I will ever be able to do that on complex code.

PPS: if the this plea falls on deaf ears (which I really hope it does’t) here is a plan-b for team gen_fsm https://gitlab.com/Project-FiFo/gen_fsm_compat

[Loïc Hoguin]

What's the issue exactly? Is there plans to remove gen_fsm already, or
is it just about its deprecation?

> The *next* one. I see forward using it *for the next one*, not the last

> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://erlang.org/mailman/listinfo/erlang-questions
>

--
Loïc Hoguin
https://ninenines.eu
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

[Heinz N. Gies]

Oops I forget to reply all again :/

Welp.

Yes it’s early. But I honestly don’t see much reason to ever deprecate it and force people to change working code. Perhaps there is a good reason other then ‘gen_statm is nicer’ I just don’t see.

> On 4. May 2018, at 12:48, Loïc Hoguin <es...@ninenines.eu> wrote:
>
> Ah I was looking in the wrong places...
>
> Yeah that's a bit early.
>
> On 05/04/2018 12:45 PM, Heinz N. Gies wrote:
>> The code says it’s going to be removed in the next major release:
>> https://github.com/erlang/otp/blob/master/lib/stdlib/src/gen_fsm.erl#L132
>> The problem I encountered was very odd edge case of different behaviour on shut down, no idea why I killed it with fire and went back to gen_fsm. It’s subtle, didn’t show up in any tests for the lib itself until it ended up in a bigger project where things got out of hand.

>>>> erlang-q...@erlang.org <mailto:erlang-q...@erlang.org>

[Karolis Petrauskas]

Hi,

I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP 19 and I dont see any reasonable path to upgrade.

There are a lot of dependencies using gen_fsm and I don't see, how one can make code be compatible between releases not providing gen_statem and deprecating gen_fsm.

With simple functions a macro approach is good enough, although it does not work with the gen_fsm.

The deprecation of the erlang:get_stacktrace() feels similarly.

Karolis

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org

http://erlang.org/mailman/listinfo/erlang-questions

[Roger Lipscombe]

On 4 May 2018 at 12:07, Karolis Petrauskas <k.petr...@gmail.com> wrote:

I agree with Heinz. The deprecation of gen_fsm forced me to stuck on OTP 19 and I dont see any reasonable path to upgrade.

We upgraded successfully to OTP 20 (and compiled against OTP 21) by simply adding the following to the top of our gen_fsm modules:

-compile([nowarn_deprecated_function]).  % gen_fsm is deprecated; use gen_statem

It'll be a problem when gen_fsm is *removed*, but until then, we'll stick with this.

[Heinz N. Gies]

I have thought of that but I don’t think that’s a real option. Deprication warnings are important, this would turn all of them off which doesn’t help code quality. I want to know if something is deprecated and fix it in most cases. But as Karolis pointed out, it’s easy for a function or two but for a whole behaviour it’s nearly impossible.

It basically creates two erlang worlds, pre-statem and post-statem.

The stack trace is not great but it’s not as bad, it’s somewhat possible to just wrap the whole function calling it into a ifdef and call it a day. Not great it still is a decent amount of code duplication but it’s possible. https://github.com/Kyorai/clique/commit/b0e89de3cbb56396b59a1023903c7259ec2ef145#diff-7a19efbe2afedfb64aa7ec6690e9a13e would be an example. It’s more annoying then other changes but not nearly as bad as the gen_fsm

[Oliver Korpilla]

Except for the init bug in 19.0 (which our infrastructure was fixed on for a long time) we found migrating to gen_statem a painless experience with decent improvements. But maybe we simply use a different or limited feature set?

Oliver

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Heinz N. Gies]

Hi Oliver,

I am not advocating to forbid migrating code to gen_statem when the maintainer feels it is an improvement for their code. I’m advocating for not forcing maintainers doing so for no reason.

Karolis brought up another very good point I didn’t even think about. Code written with gen_statem will not be able to run on older erlang releases. While that’s probably not a issue on your own code where you control the releases completely it’s a nightmare for open source and libraries.

[Oliver Korpilla]

Ah yes, dependencies! I see your problem now.

We're using gen_statem for a test framework written in elixir for telecoms. The FSM has been the least of our problems.

I'm always amazed at the range of Erlang versions still in use. I keep forgetting that.

Oliver

[Mahesh Paolini-Subramanya]

To echo Heinz's point, is there a *reason* for the deprecation?

Enquiring minds want to know...

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

--

Mahesh Paolini-Subramanya

That tall bald Indian guy..

Twitter | 

Blog | 

G+

 | 

LinkedIn

[Loïc Hoguin]

On 05/04/2018 02:03 PM, Heinz N. Gies wrote:
> I have thought of that but I don’t think that’s a real option.
> Deprication warnings are important

Are they?

Code removal is an important problem which can't be avoided when you
have to support OTP versions before/after. So is fundamental additions
to the language (like maps). But I don't think warnings are nearly as
important, particularly deprecation warnings.

Deprecation warnings signal that your code will stop working one day in
the future so they are useful, but they say nothing of your code TODAY
which is what matters most. The code still builds and works fine
regardless of the warning.

I would say it's important to know that the code will stop working at
some point in the future so you can plan migrating the code. But you
still have at least a year to figure it out, it's not OTP-21 that breaks
your code, it will be OTP-22 or a later release.

If I was in your situation I would adopt the following strategy:

* Disable the deprecation warnings in normal builds
* Setup a separate build task/CI job for compiling with deprecation
warnings (xref checks those too IIRC)
* Open a ticket to remember to fix those and/or change the supported OTP
versions within that timespan

Dropping older versions will work even for many libraries because most
users are on the last 2-3 releases anyway. There's of course exceptions.

In general I would also say that while it's usually a good idea to fix
new warnings, sometimes the fix is uglier than the warning. If you need
to depend on externally defined macros or rewrite huge chunks of the
code to fix a warning, you might want to just leave the warning around
or disable it for now.

Warnings are great to anticipate problems but they should by no means
prevent working code from working.

All that being said, I think the OTP team has been a little too
aggressive with changes lately. It would be better to deprecate two
releases before removal (introduce gen_statem in 20, deprecate gen_fsm
in 21 and remove gen_fsm in 23). That would make transitioning easier
because one could switch from gen_fsm to gen_statem during the OTP-22
era while still supporting the last 3 releases and therefore the
majority of users would be unaffected.

Cheers,

--
Loïc Hoguin
https://ninenines.eu

[Fred Hebert]

Overall, I like some of the aggressiveness that we have seen lately on introducing new useful features and seeing older ones get removed. I hate that I'll have to adapt a bunch of projects to changes in logger interfaces, but it will definitely be a more positive experience for users joining the platform now. Similarly for Unicode support in the strings module.

The thing that I'm a bit unsure about is that deprecation is used as a warning to say something is:

a) no longer the favored mechanism to do something

b) will eventually be removed.

I can tolerate warnings for a) and ignore them, but I don't want to do it for b) unless I do not know how long a thing will be deprecated for.

The thing is that if the warnings are used for a), you can have it deprecated for 10 years and it won't be a big deal; gen_fsm and the old string interface might fall into that category. Maybe get_stacktrace() as well?

But for the case of b), if I don't have a straight timeline of when a feature is going to disappear, then you bet your ass I want warnings_as_errors and to tackle it ASAP. Because if I'm working on a big project, there's no telling if a migration will take me 2 weeks or 2 years (see how long Riak has been stuck on old ass versions), and the more lenient you are as an organisation, the harder it is to stop being so.

I think I have something like 30-40 open-source libraries right now, most of which are considered 'done' and no longer need changes. When the string handling landed, I had to update dozens of them, and also had to shepherd dozens of other ones from the community at large, sometimes because maintainers have left, or sometimes because their upgrade schedule does not align with mine.

If I know a deprecation means "something newer is being used but we're not thinking of removing the old one" then fine. But right now I don't know which it is I am getting, and that is where things get a bit frustrating. I have to assume "Now is better than later": I don't want to be left out of a critical SSL fix because I have a few thousand lines of non-critical code that need patching up to get up there, delaying a deploy or a release by a few precious days.

[Vlad Dumitrescu]

On Fri, May 4, 2018 at 3:31 PM Fred Hebert <mono...@ferd.ca> wrote:

The thing that I'm a bit unsure about is that deprecation is used as a warning to say something is:

a) no longer the favored mechanism to do something

b) will eventually be removed.

I can tolerate warnings for a) and ignore them, but I don't want to do it for b) unless I do not know how long a thing will be deprecated for.

Maybe there could be two different warning for the two use cases? I confess I can't come up with a good word for the other, though...

regards,

Vlad

 

[Oliver Korpilla]

"Being phased out by"?

Oliver

[Scott Ribe]

"endangered" vs "deprecated"?

;-)

--
Scott Ribe
scott...@elevated-dev.com
https://www.linkedin.com/in/scottribe/

[Vlad Dumitrescu]

Maybe "discouraged"? 

/Vlad

[Heinz N. Gies]

I really like discouraged. It emphasises that you probably shouldn’t do it any more but that OTP won’t blow up your program next year if you do. 👍 and it’s shorter then “Please don’t do that any more but if you did before we’re not mad” ;)

[Lloyd R. Prentice]

Hi Heinz,

This exactly how I felt when I found that string functions I relied on had been dropped. 

Perhaps we need a library called “orphans.”

Best wishes,

Lloyd

Sent from my iPad

_______________________________________________

[Pierre Fenoll]

Golang has something that solves this “busy work” problem by writing one off programs that manipulate the AST then pretty print your code, updated: gofix. 

C/Linux has something which I think is more powerful: Coccinelle & semantic patches.

Facebook has a tool called pfff (not sure how many Fs) that implements that “spatch” language and I believe supports Erlang (though the primary usage is targeted at PHP). 

As I have said on this list before I would gladly help writing a linter for Erlang (hopefully for Core Erlang) that would provide code suggestions. Well, I’d gladly work on an Spatch for Erlang and help provide for example an Spatch file to automatically go from a gen_fsm to gen_statem implementation. 

Surely this has rough edges! But life is short and full of terrors. 

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions

--

Cheers,

-- 

Pierre Fenoll

[Ingela Andin]

Hi!

2018-05-04 14:40 GMT+02:00 Mahesh Paolini-Subramanya <mah...@dieswaytoofast.com>:

To echo Heinz's point, is there a *reason* for the deprecation?

Enquiring minds want to know...

Th point is to phase out gen_fsm. The deprecated warning is important to get new users to choose gen_statem. Also if you have a living product you will probably be better of in the long run changing to gen_statem. This

is not a complicated change and an exampel of how to do it is given in the gen_fsm manual page. Of course we give lots of time to adopt. We have no plan of removing the gen_fsm code swiftly, it is there in 20 and 21. 

I can see the deprecating process needs to enhanchend, and yes it sort of has more that one purpose which seems to conflict a bit.

Regards Ingela Erlang/OTP team 

[José Valim]

What about documentation from the OTP team that outlines a deprecation policy? It may be a good opportunity to also outline other compatibility guarantees, such as the compiled bytecode guarantees and the node compatibility guarantees, if such are not yet documented.

We did define such document not long ago for Elixir. In terms of deprecations, we have two terms: "soft-deprecation" and "hard-deprecation". The soft-deprecation is added as soon as a new implementation exists. At this point we don't emit any warnings, but we do update the docs to say the feature will warn in the future and start to point folks towards better ways.

A hard-deprecation is when we finally start emitting warnings. A hard-deprecation can only be added after an alternative exists for at least 2 Elixir releases. This is very important because it means that, once a feature is hard deprecated, you know you can use the proposed alternative and that alternative is supported at least two versions back.

This is only necessary if a feature is being replaced. In case a feature is being removed, such as non-smp VM versions, you may skip directly to the "hard-deprecation" and emit warnings straight-away since there won’t be an alternative in the future anyway.

I am not proposing for Erlang/OTP team to follow those rules and conventions but writing *some* rules as minimum guarantees may improve communication and help the community plan in terms of warnings, deprecations and removals accordingly. I would just avoid emitting deprecation warnings in the same version that an alternative is introduced, because it means library developers need to either only support the latest version or introduce conditionals (either at compile-time or runtime) to support multiple versions.

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

--

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

[Michael Truog]

On 05/06/2018 11:34 PM, Ingela Andin wrote:

Hi!

2018-05-04 14:40 GMT+02:00 Mahesh Paolini-Subramanya <mah...@dieswaytoofast.com>:

To echo Heinz's point, is there a *reason* for the deprecation?

Enquiring minds want to know...

Th point is to phase out gen_fsm. The deprecated warning is important to get new users to choose gen_statem. Also if you have a living product you will probably be better of in the long run changing to gen_statem. This

is not a complicated change and an exampel of how to do it is given in the gen_fsm manual page. Of course we give lots of time to adopt. We have no plan of removing the gen_fsm code swiftly, it is there in 20 and 21. 

I can see the deprecating process needs to enhanchend, and yes it sort of has more that one purpose which seems to conflict a bit.

One thing that might help is if we have the ability to deprecate types which I entered an entry for at https://bugs.erlang.org/browse/ERL-335 .  For example,  when all the time units switched from a format like milli_seconds to millisecond in Erlang/OTP 20.0 it would have been easier to have a deprecated type that can be used for an easier to understand warning.  That may seem like a small issue that can be avoided by switching to other functions or modules, but often changes don't need to be large for small changes that can be limited to type changes.

Best Regards,
Michael

[Mike French]

So you have

-          ‘soft-deprecation’  with doc notes, but no compiler warnings

-          ‘hard-deprecation’ with compiler warnings, but no feature removal

 

I suggest the third stage would be:

-          ‘end-of-life’ with compiler warnings, and there is a
known future release when the original feature will vanish

 

Mike

[Kenneth Lundin]

On Mon, May 7, 2018 at 8:54 AM, José Valim <jose....@gmail.com> wrote:

What about documentation from the OTP team that outlines a deprecation policy? It may be a good opportunity to also outline other compatibility guarantees, such as the compiled bytecode guarantees and the node compatibility guarantees, if such are not yet documented.

I think it is a good idea to describe the OTP policy regarding deprecation, backward compatibility, supported releases etc. in one place. Will try to have it available before the OTP 21 release.

The Elixir policy with hard and soft deperecation is also interesting as we need to make it

clearer what a depercation means.

But in short we use the deprecation mechanism to flag that a function och application is no longer recommended for use in new code since there exist another solution that we recomment to be used instead.

We don't remove deprecated functions or applications just like that and some functions have to stay for a very long time (or forever) since there is so much usage of them.

gen_fsm is an old module which is used a lot and it is not likely to dissapear in a near future. I realize that the deprecated attribute inside the module like this
-deprecated({start, 3, next_major_release}).
is somewhat misleading and we will try to improve this.

/Kenneth, Erlang/OTP

 

We did define such document not long ago for Elixir. In terms of deprecations, we have two terms: "soft-deprecation" and "hard-deprecation". The soft-deprecation is added as soon as a new implementation exists. At this point we don't emit any warnings, but we do update the docs to say the feature will warn in the future and start to point folks towards better ways.

A hard-deprecation is when we finally start emitting warnings. A hard-deprecation can only be added after an alternative exists for at least 2 Elixir releases. This is very important because it means that, once a feature is hard deprecated, you know you can use the proposed alternative and that alternative is supported at least two versions back.

This is only necessary if a feature is being replaced. In case a feature is being removed, such as non-smp VM versions, you may skip directly to the "hard-deprecation" and emit warnings straight-away since there won’t be an alternative in the future anyway.

I am not proposing for Erlang/OTP team to follow those rules and conventions but writing *some* rules as minimum guarantees may improve communication and help the community plan in terms of warnings, deprecations and removals accordingly. I would just avoid emitting deprecation warnings in the same version that an alternative is introduced, because it means library developers need to either only support the latest version or introduce conditionals (either at compile-time or runtime) to support multiple versions.

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

--

José Valim

www.plataformatec.com.br

Founder and 

Director of R&D

[Rickard Green]

On Mon, May 7, 2018 at 12:06 PM, Kenneth Lundin <ken...@erlang.org> wrote:

On Mon, May 7, 2018 at 8:54 AM, José Valim <jose....@gmail.com> wrote:

What about documentation from the OTP team that outlines a deprecation policy? It may be a good opportunity to also outline other compatibility guarantees, such as the compiled bytecode guarantees and the node compatibility guarantees, if such are not yet documented.

I think it is a good idea to describe the OTP policy regarding deprecation, backward compatibility, supported releases etc. in one place. Will try to have it available before the OTP 21 release.

Document available in PR-1839 <https://github.com/erlang/otp/pull/1839>

Regards,

Rickard

--

Rickard Green, Erlang/OTP, Ericsson AB