Future of the Jenkins CI

150 views
Skip to first unread message

Jason Juang

unread,
May 17, 2017, 7:08:02 PM5/17/17
to selenium-...@googlegroups.com
Continuing some thoughts started in IRC...

The Jenkins CI is at https://ci.seleniumhq.org:8443/. It runs on Google Cloud Platform, in the same project that owns the Cloud Storage buckets for Selenium releases. A major complaint is that only Googlers have admin rights to the underlying machines, which makes a lot of maintenance tasks impossible without help from one of us.

Is the general feeling that we would like to move everything over the Travis and/or Appveyor in the long run? Are there benefits to doing so other than independence from Google (e.g., less maintenance work upgrading Jenkins and the host machines)?

I would be pretty happy about giving away ownership of the Jenkins build. If sentiment is that we should eventually turn down Jenkins and move to a hosted solution so we don't have to manage our own machines, that would be a perfectly fine way to accomplish that.

Finally, would there be any objections to limiting access to the Jenkins instance? Given Jenkins' history of security incidents, I would like to limit our exposure a bit, and limiting access to authenticated users -- with access granted very liberally to anyone who asks -- would help. I'd be concerned about making the CI less useful to random Selenium contributors, but then, I'm not sure anybody ever really looks at it anyway.

Jason.

Simon Stewart

unread,
May 18, 2017, 5:27:34 AM5/18/17
to selenium-developers
Inline

On Wed, May 17, 2017 at 11:44 PM, 'Jason Juang' via Selenium Developers <selenium-...@googlegroups.com> wrote:
Continuing some thoughts started in IRC...

The Jenkins CI is at https://ci.seleniumhq.org:8443/. It runs on Google Cloud Platform, in the same project that owns the Cloud Storage buckets for Selenium releases. A major complaint is that only Googlers have admin rights to the underlying machines, which makes a lot of maintenance tasks impossible without help from one of us.

Is the general feeling that we would like to move everything over the Travis and/or Appveyor in the long run? Are there benefits to doing so other than independence from Google (e.g., less maintenance work upgrading Jenkins and the host machines)?

I'm a little surprised Travis let us run as many builds as we do on their infrastructure, and I'd hate to rely on their goodwill. Having said that, the Jenkins builds have been widely ignored by almost everyone. I think that there are edge-cases that Jenkins handles way better than Travis (the Sauce stuff, perhaps --- Luke and Daniel would know more than I do)
 
I would be pretty happy about giving away ownership of the Jenkins build. If sentiment is that we should eventually turn down Jenkins and move to a hosted solution so we don't have to manage our own machines, that would be a perfectly fine way to accomplish that.

I'd like us to have a think about the matrix of tests we consider adds value, and structure our builds appropriately. The shot-gun approach we have now (a not-terribly-close approximation of N languages * M browsers) gives us coverage, but tends to have some pretty clear seams where things go wrong. 
 
Finally, would there be any objections to limiting access to the Jenkins instance? Given Jenkins' history of security incidents, I would like to limit our exposure a bit, and limiting access to authenticated users -- with access granted very liberally to anyone who asks -- would help. I'd be concerned about making the CI less useful to random Selenium contributors, but then, I'm not sure anybody ever really looks at it anyway.

No objections at all, as long as the build summaries and logs are publicly available somehow.

Simon

Jason Juang

unread,
May 19, 2017, 11:50:41 PM5/19/17
to selenium-...@googlegroups.com
On Thu, May 18, 2017 at 2:27 AM, Simon Stewart <simon.m...@gmail.com> wrote:
I'm a little surprised Travis let us run as many builds as we do on their infrastructure, and I'd hate to rely on their goodwill. Having said that, the Jenkins builds have been widely ignored by almost everyone. I think that there are edge-cases that Jenkins handles way better than Travis (the Sauce stuff, perhaps --- Luke and Daniel would know more than I do)

Theoretically, Sauce should work with Travis, right? But we haven't gone to set that up yet.

How much work has it been to maintain the Travis configuration? There's some background level of maintenance required to keep the Jenkins instances running that, theoretically, we wouldn't have to do on a hosted solution.
 
Finally, would there be any objections to limiting access to the Jenkins instance? Given Jenkins' history of security incidents, I would like to limit our exposure a bit, and limiting access to authenticated users -- with access granted very liberally to anyone who asks -- would help. I'd be concerned about making the CI less useful to random Selenium contributors, but then, I'm not sure anybody ever really looks at it anyway.

No objections at all, as long as the build summaries and logs are publicly available somehow.

I'll try to figure out how to make that happen.

Jonathan Lipps

unread,
May 22, 2017, 1:20:57 PM5/22/17
to Selenium Developers
Hi all,

Sauce + Travis support is a thing. See the docs on Travis's site: https://docs.travis-ci.com/user/sauce-connect/ (particularly the part about the JWT encryption which is necessary for builds to be triggered on PRs). Happy to help troubleshoot if something isn't working on the Sauce side.

Cheers,
Jonathan (I work at Sauce)

Jason Juang

unread,
Jun 12, 2017, 10:02:36 PM6/12/17
to selenium-...@googlegroups.com
Sorry I left this thread hanging for a while.

Simon's response about considering the right cross-section of tests to run is interesting and should be considered, but not directly relevant to the decision I'm trying to make, which is whether to continue running this Jenkins instance. Hosting it is a liability, particularly because of Jenkins' security track record and the fact that nobody seems to know any longer how this thing was set up.

Given the current state of the Jenkins build, I am tempted to just delete any build that is currently broken (pretty much all of them). The IE tests haven't passed for over a year, and haven't even been attempted for half a year, which seems like a misconfiguration. It would be better to just get rid of those builds rather than just ignoring them and continuing to pretend that we're going to fix them. Perhaps the one exception would be https://ci.seleniumhq.org:8443/job/SeleniumHQ/, which I think Luke said is responsible for updating the SeHQ website. That should get fixed or migrated to something more reliable.

I would go so far as to say that the entire Jenkins instance should just be deleted wholesale, and if we're insistent on hosting our own Jenkins rather than using Travis or some other hosted solution, then we can rebuild it. Even if we're concerned about Travis's generosity, I would prefer to just cross that bridge if/when it arrives rather than trying to maintain a parallel CI that is ignored.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Alex Rodionov

unread,
Jun 14, 2017, 10:24:27 AM6/14/17
to Selenium Developers
My 2 cents on that is for Travis allowing us to experiment with different operating systems and, when combined with Appveyor, allows us to test on all browsers except Edge for now with zero-to-little maintenance.

For Windows we right now run Ruby tests on IE11 using Appveyor. I don't see why other bindings can't do that as well.

For macOS, SafariDriver is not supported, though I've been experimenting with it (see https://github.com/SeleniumHQ/selenium/tree/travis-safari) and hopefully at some point https://github.com/travis-ci/travis-ci/issues/7187 will be resolved.

Jason Juang

unread,
Jun 19, 2017, 9:14:43 PM6/19/17
to selenium-...@googlegroups.com, luke.s...@gmail.com
Given the discussion here, I think the security liability we incur by hosting our own Jenkins no longer justifies the marginal utility of having it. I will plan to block public access to the Jenkins instance by the end of June, and to thoroughly delete it by the end of July. We can always revive this in the future if other CI services prove not to be sufficient.

We should try to move the existing Windows Java/JS tests to Travis or Appveyor or wherever. But they've been broken on Jenkins for so long that we're already deriving zero value from them, so I won't feel too bad about losing that coverage in the interim.

I think the "SeleniumHQ" is the one build that definitely needs to be migrated somehow. Luke said something about using it to push changes to seleniumhq.org, but I don't really understand how that happens. If anyone knows for sure, LMK and let's figure out a replacement.

Jason.

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

Luke Inman-Semerau

unread,
Jun 19, 2017, 10:28:04 PM6/19/17
to Jason Juang, selenium-...@googlegroups.com
We are using google app engine with maven to deploy. We basically
followed the instructions here:

https://cloud.google.com/appengine/docs/standard/java/tools/uploadinganapp

So, the oauth token is installed on jenkins and that's how it's
working today. The only thing we'd need in another CI solution is to
provide credentials to update the app engine account securely.
>>>> an email to selenium-develo...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Selenium Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to selenium-develo...@googlegroups.com.

Jason Juang

unread,
Jun 20, 2017, 1:09:52 AM6/20/17
to Luke Inman-Semerau, selenium-...@googlegroups.com
I see. It doesn't really make sense for us to put that on something like Travis, then -- we're certainly not going to put your OAuth token in the codebase. Maybe we can rig up something lightweight in our existing GCP project to do that deployment.


>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.
>>>>
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Selenium Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

Luke Inman-Semerau

unread,
Jun 20, 2017, 1:24:21 AM6/20/17
to Jason Juang, selenium-...@googlegroups.com
Travis-ci does have a way to encrypt secrets in the yaml file -
https://docs.travis-ci.com/user/encryption-keys/

Which we could do.
>> >>>> an email to selenium-develo...@googlegroups.com.
>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.
>> >>>>
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Selenium Developers" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to selenium-develo...@googlegroups.com.

Jason Juang

unread,
Jun 20, 2017, 1:36:53 AM6/20/17
to Luke Inman-Semerau, selenium-...@googlegroups.com
Oh, that looks perfect. I'll plan to look into that later in the week, then.


>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.
>> >>>>
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Selenium Developers" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

Jason Juang

unread,
Jun 29, 2017, 9:51:35 PM6/29/17
to Luke Inman-Semerau, selenium-...@googlegroups.com
This is working now.

I've shut down the Jenkins VMs. We still have the disk images if there's anything there that needs to be recovered for some reason. Luke, you can invalidate your old OAuth token if you want.


>> >>>> To view this discussion on the web visit
>> >>>>
>> >>>> https://groups.google.com/d/msgid/selenium-developers/602840d6-3293-4ab5-b431-1ccff1e91122%40googlegroups.com.
>> >>>>
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Selenium Developers" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

Simon Stewart

unread,
Jun 30, 2017, 6:20:42 AM6/30/17
to selenium-developers, Luke Inman-Semerau
Thank you, Jason. I guess our next step is try and update the website and see what breaks :)

Simon

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/selenium-developers/CAN_hBkwPN_PDogx73EDJh8aAW-fD9zvc-%3D%3D4vQ%3DD_Sdgi6pfhQ%40mail.gmail.com.

Jason Juang

unread,
Jun 30, 2017, 8:02:02 PM6/30/17
to selenium-...@googlegroups.com, Luke Inman-Semerau
FWIW, I did already do a couple deploys from Travis -- I made a token change to the website just to make sure it got pushed, and it did. (I even got an email to tell me the build passed.)

By the way, I found credentials for an appe...@seleniumhq.org user in the Jenkins config. I actually used that account to go in to the GAE project and set up service account permissions for Travis. Is that user used for anything else? If not, it should be removed, or at least we should change the password, so that not every Googler with source access can go much with sehq.org.

On Fri, Jun 30, 2017 at 3:20 AM, Simon Stewart <simon.m...@gmail.com> wrote:
Thank you, Jason. I guess our next step is try and update the website and see what breaks :)

Simon
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

Daniel Wagner-Hall

unread,
Jul 1, 2017, 2:59:50 PM7/1/17
to selenium-developers, Luke Inman-Semerau
On 1 July 2017 at 01:01, 'Jason Juang' via Selenium Developers <selenium-...@googlegroups.com> wrote:
FWIW, I did already do a couple deploys from Travis -- I made a token change to the website just to make sure it got pushed, and it did. (I even got an email to tell me the build passed.)

Sounds awesome! Thanks for tearing this stuff down so cleanly :)

By the way, I found credentials for an appe...@seleniumhq.org user in the Jenkins config. I actually used that account to go in to the GAE project and set up service account permissions for Travis. Is that user used for anything else? If not, it should be removed, or at least we should change the password, so that not every Googler with source access can go much with sehq.org.

As far as I know, that's the only thing it's used for. That said, I'm not aware of how releases are pushed to Google Cloud Storage.
 

To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.

Simon Stewart

unread,
Jul 3, 2017, 4:33:25 AM7/3/17
to selenium-developers, Luke Inman-Semerau
On Sat, Jul 1, 2017 at 7:59 PM, Daniel Wagner-Hall <dawa...@gmail.com> wrote:
On 1 July 2017 at 01:01, 'Jason Juang' via Selenium Developers <selenium-developers@googlegroups.com> wrote:
FWIW, I did already do a couple deploys from Travis -- I made a token change to the website just to make sure it got pushed, and it did. (I even got an email to tell me the build passed.)

Sounds awesome! Thanks for tearing this stuff down so cleanly :)

+1
 
By the way, I found credentials for an appe...@seleniumhq.org user in the Jenkins config. I actually used that account to go in to the GAE project and set up service account permissions for Travis. Is that user used for anything else? If not, it should be removed, or at least we should change the password, so that not every Googler with source access can go much with sehq.org.

As far as I know, that's the only thing it's used for. That said, I'm not aware of how releases are pushed to Google Cloud Storage.


Which is called from the `:push_release` target.

Simon
 
 
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsubscribe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Selenium Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to selenium-developers+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages