how to approach front end /back end separation?

[Martin Magdinier]

Hello,

During 2018 we did long due clean up (maven) and made some breaking changes (Jackson) to OpenRefine. We also realize we still have a long road before separating the front and back end (butterfly migration, rewriting of the front end in a modern framework ...)

I wanted to share this article regarding software rewriting stories. It offers different perspectives on how to approach a large refactoring/rewrite project.  

Lessons from 6 software rewrite stories - https://medium.com/@herbcaudill/lessons-from-6-software-rewrite-stories-635e4c8f7c22 - A new take on the age-old question: Should you rewrite your application from scratch, or is that “the single worst strategic mistake that any software company can make”? Turns out there are more than two options for dealing with a mature codebase.

Should we continue to gradually migrate OpenRefine at the risk of losing a part of the community due to incompatibility with some extension? Should we restart a new project based on everything we learned and want for OpenRefine while maintaining the current version?  What do you think?

--

Martin Magdinier

[Thad Guidry]

I seriously hope your kidding on the "scratch everything and rewrite" ?

Those that want to take advantage of new features delivered will ultimately migrate to new version to gain those new features.

Also... Nothing stops them from running version 2.7 , 3.1 , and 4.0  at the same time, if they follow a simple procedure of updating their workspace.

Thad

https://www.linkedin.com/in/thadguidry/

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ettore RIZZA]

at the risk of losing a part of the community due to incompatibility with some extension?

Hello all,

Stop me if I seem to be silly, but I do not feel that extension compatibility is a so critical issue. One of two things: either the extension is always maintained by its/some developer, and in this case solutions are possible to upgrade it (the best example is the Stuart Kenny's work on RDF and NER extensions); or it is no longer maintained, and in this case it is the extension that has a problem, not OpenRefine. Many of these extensions use external APIs and services: without any maintenance, they are in any case doomed to bug some day. There remain some cases like the Stat extension, of which we spoke recently, or VIB-BITS. But is it really so difficult to update these extensions for > OR 3.2? Could we not make a list of extensions of strategic interest (for example based on the 2018 survey of users) for which we should try to contact the developers to explain how to update it? Sorry if all this has already been discussed. 

Ettore Rizza

[Owen Stephens]

Thanks Martin - that's a really interesting article.

Ultimately our biggest problem doing anything is the amount of developer time available to us as a project. This is both limited and unpredictable. Any choices we make have to come with that in mind.

I've re-written this response about 5 times now as I'm struggling to express what I think! I guess ultimately I don't think re-writing from scratch is either viable or desirable. However once the Jackson migration is complete, and when we are able to replace Butterfly - that will have re-written substantial parts of the backend code. If we had the time to carefully re-factor the other code as a result of these changes we could move to a much cleaner set of back end code I think.

In terms of the frontend - I think this is perhaps more of a candidate for "start from scratch" (see the Gmail/Inbox case study in that article). Tweaking what we have is unlikely to bring huge rewards beyond a slightly more modern looking UI (although that wouldn't be the worse thing in the world!) - but we need to have that clear frontend/backend separation + well documented API in position first.

In terms of the extensions I think we should:

* Identify Extension functionality that ought to be part of the core product and work to make that happen

* Identify outstanding Extension functionality that is important to the users and see if we can get extensions updated by the original authors, or adopted by someone in the community

So I'm definitely in favour of evolution not revolution here. 

Owen

On Friday, April 5, 2019 at 8:05:10 AM UTC+1, Ettore RIZZA wrote:

at the risk of losing a part of the community due to incompatibility with some extension?

Hello all,

Stop me if I seem to be silly, but I do not feel that extension compatibility is a so critical issue. One of two things: either the extension is always maintained by its/some developer, and in this case solutions are possible to upgrade it (the best example is the Stuart Kenny's work on RDF and NER extensions); or it is no longer maintained, and in this case it is the extension that has a problem, not OpenRefine. Many of these extensions use external APIs and services: without any maintenance, they are in any case doomed to bug some day. There remain some cases like the Stat extension, of which we spoke recently, or VIB-BITS. But is it really so difficult to update these extensions for > OR 3.2? Could we not make a list of extensions of strategic interest (for example based on the 2018 survey of users) for which we should try to contact the developers to explain how to update it? Sorry if all this has already been discussed. 

Ettore Rizza

Le ven. 5 avr. 2019 à 01:50, Thad Guidry <thadg...@gmail.com> a écrit :

I seriously hope your kidding on the "scratch everything and rewrite" ?

Those that want to take advantage of new features delivered will ultimately migrate to new version to gain those new features.

Also... Nothing stops them from running version 2.7 , 3.1 , and 4.0  at the same time, if they follow a simple procedure of updating their workspace.

Thad

https://www.linkedin.com/in/thadguidry/

On Thu, Apr 4, 2019 at 2:14 PM Martin Magdinier <martin....@gmail.com> wrote:

Hello,

During 2018 we did long due clean up (maven) and made some breaking changes (Jackson) to OpenRefine. We also realize we still have a long road before separating the front and back end (butterfly migration, rewriting of the front end in a modern framework ...)

I wanted to share this article regarding software rewriting stories. It offers different perspectives on how to approach a large refactoring/rewrite project.  

Lessons from 6 software rewrite stories - https://medium.com/@herbcaudill/lessons-from-6-software-rewrite-stories-635e4c8f7c22 - A new take on the age-old question: Should you rewrite your application from scratch, or is that “the single worst strategic mistake that any software company can make”? Turns out there are more than two options for dealing with a mature codebase.

Should we continue to gradually migrate OpenRefine at the risk of losing a part of the community due to incompatibility with some extension? Should we restart a new project based on everything we learned and want for OpenRefine while maintaining the current version?  What do you think?

--

Martin Magdinier

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openref...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openref...@googlegroups.com.

[Ettore RIZZA]

(as we are talking about breaking changes) It's not clear, even for me, when OR started to introduce breaking changes. Users have just noticed that OpenRefine has changed logo by upgrading to version 3, but changes that most likely risk to break an existing workflow are, I believe, in OR 3.1 and 3.2, so in relatively "minor changes" according to the numbering. All this is very confusing. I wonder if there would be interest in completely removing some buggy version, such as OR 3.1.

Ettore Rizza

To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

[Owen Stephens]

This is probably a topic for a different thread! I'd say:

* We need to be careful about what we mean by "breaking changes"

* I don't see 3.1 as particularly buggy

[Martin Magdinier]

Thad, I am not suggesting anything. I am just sharing some perspective as we approach the refactoring. I will take those inputs into consideration when reviewing the Phase 2 doc for Google News Initiative.

Ettore, thanks for bring us to facts and the survey result regarding extension usage. Regarding the release number. Short answer, we decided to release from an end-user perspective (this not a major number change). See the full  discussion here: https://groups.google.com/d/msg/openrefine-dev/h1uSqZUexys/4w5P1AKjCAAJ

Owen, breaking change means we do not support backward compatibility. 3.1 is not buggy for front end and API users but it broke all existing extension. See this message from Antonin: https://groups.google.com/d/msg/openrefine-dev/HpkwPExz30o/8AD_CtdfCQAJ