Releasing 3.4.1 with the Google credentials

[Antonin Delpeuch (lists)]

Hi all,

The released artifacts for 3.4 miss the Google credentials. Sorry about
that, I forgot to add them (it was not mentioned in the documentation,
so I have fixed that).

I propose we release a 3.4.1 with that fix (which does not need a PR
since it is not something we commit anyway).

Alternatively I can also update the artifacts on the existing release.

As a reminder, it is a bad practice to add API keys in this way to
releases, since those credentials are not supposed to be communicated to
users. Sadly, given the way OAuth works, this is the only way we can
ship something workable to users (otherwise they would have to apply for
Google API credentials on their own, which is a pretty lengthy process).
If anyone is aware of a better solution to this problem, I am very
interested.

Sorry again for this oversight!
Antonin

[Thad Guidry]

Agree, let's push out an updated 3.4.1 (and I'd probably remove the 3.4 optionally)

Yes, it can be simpler.

We can use GoogleAuthorizationCodeFlow

instead of our current handling which uses the lower-level classes such as GoogleAuthorizationCodeRequestUrl

Details here: https://developers.google.com/api-client-library/java/google-api-java-client/oauth2#authorization_code_flow

Thad

https://www.linkedin.com/in/thadguidry/

[Tom Morris]

Antonin - Can you provide a link to the documentation that you're talking about? Is the process manual or automated? If it's manual, we need to automate it. e.g. there's a secrets.txt file (or env variable, etc) that's optional for non-release builds, but mandatory for release builds.

Thad - I don't see how that helps since it still requires secrets. Can you explain in more detail?

I'm inclined to stick with the current scheme until we are forced to switch, but two other possibilities are:

- Give up on server access and download to the client and then upload to the server so that all the authentication can be done client side. Obviously not very efficient, but might be the path of least resistance

- ask Google for suggestions on how to deal with our use case (although this has the risk of inviting scrutiny of the current scheme)

Tom

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/8ff79345-caaa-7f70-b9b4-61b73ac94e8e%40antonin.delpeuch.eu.

[Thad Guidry]

Tom,

Sure happy to explain more.

Secrets are in the hands of the user always.  The users credentials with Google (username + password) are the actual secret and that never needs to be stored in our Application in a file or memory.

But what does need to be stored in memory or a file (at least temporarily) is the access token our client is finally granted.

Furthermore, an "auth code", which many think of as a 1st pass secret I guess in pre-Oauth 2.0 days, is generated when a user logs in to authorize our "clientID + their Google username" in order to get an access token (stored for 1 hour or longer) that we store in memory or a file temporarily.

The access token (the final key we use to talk via the Google APIs on the users behalf) is what provides access temporarily to read/write to Sheets API on behalf of the user.

This is explained in OAuth 2.0 (Authorization Code Grant - what some folks in the enterprise refer to as Oauth 2+)

Optionally, a refresh token will also be granted to our client.

This is excellently shown I think in

https://tools.ietf.org/html/rfc6749#section-4.1

Thad

https://www.linkedin.com/in/thadguidry/

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEFoE_2xHHiyLJvX1nS2Wn8jBr-Ykbfn9%3DvmpAAVDLXiXA%40mail.gmail.com.

[Thad Guidry]

Sorry, should have probably started with this statement:

OAuth defines four grant types: authorization code, implicit,
   resource owner password credentials, and client credentials.

You guys are probably thinking of only "resource owner password credentials" which I think we might have framed up in the code that Lu Liu or Jacky made.

I am suggesting instead "authorization code" and using the new higher class provided by Google lib as I mentioned earlier.

Thad

https://www.linkedin.com/in/thadguidry/

[Thad Guidry]

The 4 grant types are fully explained with each of their subsections on Request, Response, Error, Access Token Request, Access Token Reponse.

https://tools.ietf.org/html/rfc6749#section-4.1

https://tools.ietf.org/html/rfc6749#section-4.2

https://tools.ietf.org/html/rfc6749#section-4.3

https://tools.ietf.org/html/rfc6749#section-4.4

Thad

https://www.linkedin.com/in/thadguidry/

[Antonin Delpeuch (lists)]

On 23/09/2020 20:51, Tom Morris wrote:
> Antonin - Can you provide a link to the documentation that you're
> talking about? Is the process manual or automated? If it's manual, we
> need to automate it. e.g. there's a secrets.txt file (or env variable,
> etc) that's optional for non-release builds, but mandatory for release
> builds.

The release process is described here:
https://github.com/OpenRefine/OpenRefine/wiki/Releasing-Version

>
> I'm inclined to stick with the current scheme until we are forced to
> switch

Yes, I am fine with leaving things in this state - the worst that can
happen is that suddenly the Google extension stops working. It's not
great but it is not going to jeopardize anything else.

> - Give up on server access and download to the client and then upload to
> the server so that all the authentication can be done client side.
> Obviously not very efficient, but might be the path of least resistance

I don't understand what you mean. If what you mean is changing the share
of responsibilities between the frontend and the backend, I do not see
how that can solve the problem: we need to ship the frontend and backend
code to the user, so secrets will be embedded there somehow.

Another option would be to run our own web service which would act as a
proxy between OpenRefine and the Google APIs - the credentials would
only need to be stored in that web service. But beyond the fact that it
requires more maintenance, it also means that we can see user data
flowing through that - not great for privacy I would say. And I suspect
that such a service would violate the TOS in the same way - we'd be
providing an open proxy to let anyone bypass their OAuth registration
process…

Antonin

>
> On Wed, Sep 23, 2020 at 12:24 PM Antonin Delpeuch (lists)
> <li...@antonin.delpeuch.eu <mailto:li...@antonin.delpeuch.eu>> wrote:
>
> Hi all,
>
> The released artifacts for 3.4 miss the Google credentials. Sorry about
> that, I forgot to add them (it was not mentioned in the documentation,
> so I have fixed that).
>
> I propose we release a 3.4.1 with that fix (which does not need a PR
> since it is not something we commit anyway).
>
> Alternatively I can also update the artifacts on the existing release.
>
> As a reminder, it is a bad practice to add API keys in this way to
> releases, since those credentials are not supposed to be communicated to
> users. Sadly, given the way OAuth works, this is the only way we can
> ship something workable to users (otherwise they would have to apply for
> Google API credentials on their own, which is a pretty lengthy process).
> If anyone is aware of a better solution to this problem, I am very
> interested.
>
> Sorry again for this oversight!
> Antonin
>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine Development" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to openrefine-de...@googlegroups.com

> <mailto:openrefine-dev%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/openrefine-dev/8ff79345-caaa-7f70-b9b4-61b73ac94e8e%40antonin.delpeuch.eu.
>

> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine-de...@googlegroups.com

> <mailto:openrefine-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEFoE_2xHHiyLJvX1nS2Wn8jBr-Ykbfn9%3DvmpAAVDLXiXA%40mail.gmail.com
> <https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEFoE_2xHHiyLJvX1nS2Wn8jBr-Ykbfn9%3DvmpAAVDLXiXA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Thad Guidry]

Furthermore... direct from this paragraph on the section on Google's OAuth Basic Steps https://developers.google.com/identity/protocols/oauth2#1.-obtain-oauth-2.0-credentials-from-the-dynamic_data.setvar.console_name-.

For example, a JavaScript application does not require a secret, but a web server application does.

Thad

https://www.linkedin.com/in/thadguidry/

[Antonin Delpeuch (lists)]

Ok, I take that back, it does look like it should be possible not to
ship sensitive credentials if we do all the interaction with Google on
the frontend side:
https://developers.google.com/identity/protocols/oauth2/javascript-implicit-flow

I am not particularly keen to work on that, but if anyone is, it would
probably be safer on the long term.

Antonin

> --
> You received this message because you are subscribed to the Google
> Groups "OpenRefine Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openrefine-de...@googlegroups.com
> <mailto:openrefine-de...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openrefine-dev/CAChbWaP0NOY4f%3D1QOq7gmiNL3oN7QpGO37F2RDuMz7H63gs%3DRQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/openrefine-dev/CAChbWaP0NOY4f%3D1QOq7gmiNL3oN7QpGO37F2RDuMz7H63gs%3DRQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Tom Morris]

[+openrefine-dev]

On Wed, Sep 23, 2020 at 3:22 PM Antonin Delpeuch (lists) <li...@antonin.delpeuch.eu> wrote:

On 23/09/2020 20:51, Tom Morris wrote:

> Antonin - Can you provide a link to the documentation that you're
> talking about? Is the process manual or automated? If it's manual, we
> need to automate it. e.g. there's a secrets.txt file (or env variable,
> etc) that's optional for non-release builds, but mandatory for release
> builds.

The release process is described here:
https://github.com/OpenRefine/OpenRefine/wiki/Releasing-Version

That has too many manual steps in it. As I outlined above, I'd expect ./refine dist 3.4 to look for a file containing the secrets, use that info to update the source, then build the kits. For dev builds the secrets file should be optional, but distribution kit builds should fail if it's not present or malformed. I don't have a strong preference as to whether this is updated before or after 3.4.1, but doing it before would provide a good opportunity to test it.

> On Wed, Sep 23, 2020 at 12:24 PM Antonin Delpeuch (lists)
> <li...@antonin.delpeuch.eu <mailto:li...@antonin.delpeuch.eu>> wrote:
>
>     I propose we release a 3.4.1 with that fix (which does not need a PR
>     since it is not something we commit anyway).

Yes, we should do a new release. And, yes, it absolutely needs a new release number since it has different content.

>     Alternatively I can also update the artifacts on the existing release.

Changing the contents of the release without changing the version number is something we should never do. It makes support conversations incredibly confusing. Speaking of which, what do we need to do to get the internal version number fixed for the Mac kit? It currently says 3.4-beta2. Can that be fixed as part of 3.4.1? https://github.com/OpenRefine/OpenRefine/issues/3196

Tom

>
> I'm inclined to stick with the current scheme until we are forced to

> switch

Yes, I am fine with leaving things in this state - the worst that can
happen is that suddenly the Google extension stops working. It's not
great but it is not going to jeopardize anything else.

> - Give up on server access and download to the client and then upload to
> the server so that all the authentication can be done client side.
> Obviously not very efficient, but might be the path of least resistance

[Thad Guidry]

Hi Guys,

I just want to follow up and finalize this thread discussion.

We have fixed the release.  Great!  And for MacOS.  Great!

We have the information to follow for inserting the Google credentials on our Release Version wiki page.  Great!

Anything else left to do, or an issue that needs to be opened because there's still some left over improvements we want to make for the release process?

Is some of that going to be within Tom's effort through our contract work statement or not?

Thad

https://www.linkedin.com/in/thadguidry/

--

You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEHWU5NQp3xGXjzntWdxQ-DYkzfuHFKkyA%3D9npRQDu3HMg%40mail.gmail.com.

[Tom Morris]

Two things which come immediately to mind are:

• a fully automated kit build - https://github.com/OpenRefine/OpenRefine/issues/3241
  
• agreed criteria & processes for releases (including the full lifecycle of beta, rc, final)

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAChbWaPu5L7j7%2BNVFF6Tbtft2Nm5y%3DuXj9kRs%3DxxL6cKirq2CA%40mail.gmail.com.

[Thad Guidry]

Tom - Thanks so much for creating that issue for fully automating the kits for releasing!

All Devs - I am of the opinion that we no longer bother with RC.  We have SNAPSHOTS now that take their place.

So what's left is Beta and Final releases and agreeing on criteria on both.  ( we have already got agreement now on semantic versioning with major.minor.patch )

I'll leave that criteria discussion up to all the devs here but I'd like to see final agreement by the end of this month?  I think that would be nice.

Thad

https://www.linkedin.com/in/thadguidry/

[Thad Guidry]

Following up on this...(been 2 weeks and we are now at the end of the month)

Describing our release criteria and processes for Beta and Final (I'm still of the opinion that we don't confuse users with RC at all and drop that, and instead just Beta 2, 3, etc. until Final)

What else needs to be laid out in docs about our release criteria and processes?

Thad

https://www.linkedin.com/in/thadguidry/

[Tom Morris]

Sorry for the delayed reply. I was sick much of last week. I can write up the release criteria in the form of a PR as a way to force review/consensus, but it's not going to happen this week.

Snapshots and release candidates are at opposite ends of the stability spectrum - snapshots least stable and release candidates most stable. We can make Release Candidates a developer-only thing if you really think our users are that easily confused, but I think it's unwise to eliminate them because otherwise there's no vehicle for peer review of the proposed final bits.

Out of curiosity, is there something special about the end of October as a deadline?

Tom

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAChbWaOd1w4z2DreQCJChDu5KWatsE-x2HLV96Ct5wX3MeiGrA%40mail.gmail.com.

[Thad Guidry]

Hi Tom,

No hard deadline, but was our goal, which is all around improving our governance.  Where Martin also has done updates on the Governance.md

The release criteria is a small part of that (which roles should/can do a release, the criteria, etc.) and having it in the Governance is OK, I think?

I think our Beta's have often served as Release Candidates for users... which is why I don't see a need to have an extra thing for that.

And by us having Snapshots, we get early adopters and developers, and those that run like the wind, to try out the latest features, shape them, and give feedback before a Beta or Final.

So again, I don't see a need for RC's, let's let Beta's fill that position as they seemed to have always done in our past.

Thad

https://www.linkedin.com/in/thadguidry/

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEFKYEddaw0V1sU%2BwS9RPrTdbKn9YHMkjVGOdZ3di_JmqA%40mail.gmail.com.

[Tom Morris]

On Tue, Nov 3, 2020 at 12:07 PM Thad Guidry <thadg...@gmail.com> wrote:

So again, I don't see a need for RC's, let's let Beta's fill that position as they seemed to have always done in our past.

Huh? We've had release candidates since at least 2011 https://groups.google.com/g/openrefine/search?q=rc1 You even announced one in 2017.

Since I'm not doing a very good job of explaining this, perhaps these will help:

Wikipedia https://en.wikipedia.org/wiki/Software_release_life_cycle

StackOverflow https://stackoverflow.com/questions/736433/differences-between-ctp-alpha-beta-gamma-rtm-and-preview-for-software-releas#736475

Drupal https://www.drupal.org/docs/8/understanding-drupal-version-numbers/what-are-alpha-and-beta-releases-and-release-candidates

At the end of the day, it's all about managing risk and is ultimately the release manager's call. If there were no significant changes during the last beta AND you have a fully automated build, you MIGHT consider skipping a release candidate, but the cost is so low it's probably a false economy.

Note that doing a release candidate doesn't mean tacking another month onto the schedule. It could be announced to developer's only. If announced to users, it might only have a testing period of a week (or less).

Tom

[Thad Guidry]

Tom, yes we have had RC's since 2011.  I'm saying we might want to change that.  I agree on all points Tom. (your speaking to the choir)

It's not my burden here, sorry.  I just gave my opinion only from a user perspective and assumptions of testing or even seeing an RC as something that the user community needs to test for the Project for some period of time.

Ultimately it's up to a release manager...who has some expectation that some classified release (RC, Beta, Final) is going to be tested by some part (devs only) or all of the user community.  Sometimes that expectation is very assuming, as we all know.  It is what it is, we do our best around here to put out a quality release, right?  Anyways...

Bowing out of this conversation now respectfully, where I have no more to say or can provide help to you or Antonin or other devs.

Good Luck on your decisions, whatever they may be our dear "release managers"!

Thad

https://www.linkedin.com/in/thadguidry/

--
You received this message because you are subscribed to the Google Groups "OpenRefine Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openrefine-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openrefine-dev/CAE9vqEF%3DZUkTp5wN%3Do0EPchaMK77HO4_Z%3DvE7aHLmgihn04hFQ%40mail.gmail.com.