Gerrit Devops

[Damien Chesneau]

Hello,

As I can see for Gerrit of Gerrit source code, you upgrade versions automaticaly. 

We want to have the new release installed automaticaly in the night after that are published.

How can I do the same mainly for the shema upgrade of database ? 

Thanks per advance.

[luca.mi...@gmail.com]

Have you looked at my zero-downtime upgrades on SlideShare?

https://www.slideshare.net/mobile/lucamilanesio/zerodowntime-gerrit-code-review-upgrades

HTH

Luca

Sent from my iPhone

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Saša Živkov]

On Sat, Nov 19, 2016 at 2:25 PM, <luca.mi...@gmail.com> wrote:

Have you looked at my zero-downtime upgrades on SlideShare?

https://www.slideshare.net/mobile/lucamilanesio/zerodowntime-gerrit-code-review-upgrades

Hi Luca,

thanks for sharing your slides! Nice work!

I have a few questions about your slides:

After the Stage 3 (slide 15) the schema version on the copy is higher than the schema

version on the productive system (for a major upgrade).

During Stage 5 you perform DB export/import again. How does it work when the

schema version is different? I guess that during the init you make sure not to drop

unneeded tables/columns?

In stage 6 you perform an init again? What would that init do when the schema

version is already up-to-date?

Saša

 

HTH

Luca

Sent from my iPhone

On 19 Nov 2016, at 08:26, Damien Chesneau <damien....@nokia.com> wrote:

Hello,

As I can see for Gerrit of Gerrit source code, you upgrade versions automaticaly. 

We want to have the new release installed automaticaly in the night after that are published.

How can I do the same mainly for the shema upgrade of database ? 

Thanks per advance.

--
--

To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Luca Milanesio]

Hi Saša,

see my feedback below.

On 22 Nov 2016, at 12:05, Saša Živkov <ziv...@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:25 PM,  <luca.mi...@gmail.com> wrote:

Have you looked at my zero-downtime upgrades on SlideShare?

https://www.slideshare.net/mobile/lucamilanesio/zerodowntime-gerrit-code-review-upgrades

Hi Luca,

thanks for sharing your slides! Nice work!

I have a few questions about your slides:

After the Stage 3 (slide 15) the schema version on the copy is higher than the schema

version on the productive system (for a major upgrade).

Yes.

During Stage 5 you perform DB export/import again. How does it work when the

schema version is different?

Thanks for the question :-)

Stage 5 will:

- import a Version X of the DB (old version)

- keep an existing Version X + N of the Index (new version)

I guess that during the init you make sure not to drop

unneeded tables/columns?

We don't do init at Stage 5. Week keep version X of the DB.

In stage 6 you perform an init again?

Yep, because at Stage 5 the DB would have come back to version X. So we need to upgrade it again to version X + N.

What would that init do when the schema

version is already up-to-date?

Nothing, keep it as-is.

[Saša Živkov]

On Tue, Nov 22, 2016 at 1:33 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Saša,

see my feedback below.

On 22 Nov 2016, at 12:05, Saša Živkov <ziv...@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:25 PM,  <luca.mi...@gmail.com> wrote:

Have you looked at my zero-downtime upgrades on SlideShare?

https://www.slideshare.net/mobile/lucamilanesio/zerodowntime-gerrit-code-review-upgrades

Hi Luca,

thanks for sharing your slides! Nice work!

I have a few questions about your slides:

After the Stage 3 (slide 15) the schema version on the copy is higher than the schema

version on the productive system (for a major upgrade).

Yes.

During Stage 5 you perform DB export/import again. How does it work when the

schema version is different?

Thanks for the question :-)

Stage 5 will:

- import a Version X of the DB (old version)

Basically this replaces the existing DB (X+N version) with the DB from the productive instance (version X).

I think naming this step "replace" instead of "import" is more precise?

- keep an existing Version X + N of the Index (new version)

I guess that during the init you make sure not to drop

unneeded tables/columns?

We don't do init at Stage 5. Week keep version X of the DB.

Sorry, I was referring to the init from the stage 3.

 

In stage 6 you perform an init again?

Yep, because at Stage 5 the DB would have come back to version X. So we need to upgrade it again to version X + N.

 

This assumes that all schema migrations are idempotent i.e. if something was already migrated from DB to Git then

running the same schema migration will handle that gracefully. Which usually is the case :-)

 

What would that init do when the schema

version is already up-to-date?

Nothing, keep it as-is.

Saša

 

HTH

Luca

Sent from my iPhone

On 19 Nov 2016, at 08:26, Damien Chesneau <damien....@nokia.com> wrote:

Hello,

As I can see for Gerrit of Gerrit source code, you upgrade versions automaticaly. 

We want to have the new release installed automaticaly in the night after that are published.

How can I do the same mainly for the shema upgrade of database ? 

Thanks per advance.

-- 
-- 

To unsubscribe, email repo-discuss+unsub...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
-- 
To unsubscribe, email repo-discuss+unsub...@googlegroups.com

[Luca Milanesio]

Hi Saša, see my replies in-line:

On 22 Nov 2016, at 12:45, Saša Živkov <ziv...@gmail.com> wrote:

On Tue, Nov 22, 2016 at 1:33 PM, Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Saša,

see my feedback below.

On 22 Nov 2016, at 12:05, Saša Živkov <ziv...@gmail.com> wrote:

On Sat, Nov 19, 2016 at 2:25 PM,  <luca.mi...@gmail.com> wrote:

Have you looked at my zero-downtime upgrades on SlideShare?

https://www.slideshare.net/mobile/lucamilanesio/zerodowntime-gerrit-code-review-upgrades

Hi Luca,

thanks for sharing your slides! Nice work!

I have a few questions about your slides:

After the Stage 3 (slide 15) the schema version on the copy is higher than the schema

version on the productive system (for a major upgrade).

Yes.

During Stage 5 you perform DB export/import again. How does it work when the

schema version is different?

Thanks for the question :-)

Stage 5 will:

- import a Version X of the DB (old version)

Basically this replaces the existing DB (X+N version) with the DB from the productive instance (version X).

I think naming this step "replace" instead of "import" is more precise?

it's a drop/create/import.

- keep an existing Version X + N of the Index (new version)

I guess that during the init you make sure not to drop

unneeded tables/columns?

We don't do init at Stage 5. Week keep version X of the DB.

Sorry, I was referring to the init from the stage 3.

It is typically not a problem to keep unneeded / unused tables or columns.

As long as they are not used, they should not create problems.

 

In stage 6 you perform an init again?

Yep, because at Stage 5 the DB would have come back to version X. So we need to upgrade it again to version X + N.

 

This assumes that all schema migrations are idempotent i.e. if something was already migrated from DB to Git then

running the same schema migration will handle that gracefully. Which usually is the case :-)

I have  never seen an "init" in Gerrit that wasn't idempotent.

Are you aware of any?

[Steffen Gebert]

Thanks for the slides, Luca!

I also want to become a true DevOp^^.

I don't want to hijack the thread, but for a server maintenance next week, I'm looking into ways to prohibit any write actions to our Gerrit server. You mention the "Read Only plugin", which would be exactly what I need.
Could you provide that to us, please?

Thanks
Steffen

P.S: The project option "State: Read Only" [1] looked also very promising, as read actions are still possible - only commenting does still work. Bug?


[1] https://gerrit-review.googlesource.com/Documentation/project-configuration.html#project-state

[Damien Chesneau]

Hello,

I have read this, thats a great help for us. 

How to get schema update script automaticaly, do you have in gerrit a folder with all this scripts ?

And you explain 1/2 month of planning for a repo upgrade but we need to do this in 10 min max.

Did you think that are possible ? 

[Luca Milanesio]

Hi Steffen,

the "read only" plugin is a Groovy script and can be found at [2].

With regards to REST API, we just added some simple rewrite rules. The missing bit was the state changes that could be triggered via SSH (review / verified by robots or Jenkins).

The read-only window last however for a couple of minutes, so the risk of losing some of them would not have been high.

Setting the projects with "read only" state would work as well, however you would need to record the status "pre-readonly" and resume it correctly afterwards.

[2] https://gerrit.googlesource.com/plugins/scripts/+/HEAD/admin/readonly-1.0.groovy

[Luca Milanesio]

On 24 Nov 2016, at 09:04, Damien Chesneau <ches....@gmail.com> wrote:

Hello,

I have read this, thats a great help for us. 

How to get schema update script automaticaly, do you have in gerrit a folder with all this scripts ?

The gerrit init does the job for you.

And you explain 1/2 month of planning for a repo upgrade but we need to do this in 10 min max.

I mean, planning a major upgrades takes around 2 weeks in my experience. Minor upgrades can be planned in only 10 mins or less ... you just do it :-)

[hari]

Hello Luca,

This was very helpful. I am going to try this for our upgrades. Between is there a "read-only" plugin already available for Gerrit? Could you let me know from where I can get it? 

[Luca Milanesio]

The one I have is a simple Groovy script, but isn't complete because comments via SSH aren't covered at all :-(

Luca.

[David Pursehouse]

On Thu, Nov 24, 2016 at 6:11 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Steffen,

the "read only" plugin is a Groovy script and can be found at [2].

With regards to REST API, we just added some simple rewrite rules. The missing bit was the state changes that could be triggered via SSH (review / verified by robots or Jenkins).

The read-only window last however for a couple of minutes, so the risk of losing some of them would not have been high.

Setting the projects with "read only" state would work as well, however you would need to record the status "pre-readonly" and resume it correctly afterwards.

[2] https://gerrit.googlesource.com/plugins/scripts/+/HEAD/admin/readonly-1.0.groovy

I've started working on making this into a standalone plugin.

So far I've uploaded 2 changes:

https://gerrit-review.googlesource.com/q/project:plugins%252Freadonly+status:open

The first simply mimics the functionality of the existing groovy plugin, using the commit validator to block all incoming commits.  The following change adds an HTTP filter that blocks POST/PUT/DELETE requests.

The missing piece is still the fact that state can be changed by ssh commands.  I've been looking into the possibility of implementing a repository manager that opens the repositories read-only, but that doesn't seem to be very easy.

[Luca Milanesio]

Sounds good :-) Will add my reviews to it.

[David Pursehouse]

On Tue, Feb 20, 2018 at 10:15 PM David Pursehouse <david.pu...@gmail.com> wrote:

On Thu, Nov 24, 2016 at 6:11 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Steffen,

the "read only" plugin is a Groovy script and can be found at [2].

With regards to REST API, we just added some simple rewrite rules. The missing bit was the state changes that could be triggered via SSH (review / verified by robots or Jenkins).

The read-only window last however for a couple of minutes, so the risk of losing some of them would not have been high.

Setting the projects with "read only" state would work as well, however you would need to record the status "pre-readonly" and resume it correctly afterwards.

[2] https://gerrit.googlesource.com/plugins/scripts/+/HEAD/admin/readonly-1.0.groovy

I've started working on making this into a standalone plugin.

So far I've uploaded 2 changes:

https://gerrit-review.googlesource.com/q/project:plugins%252Freadonly+status:open

The first simply mimics the functionality of the existing groovy plugin, using the commit validator to block all incoming commits.  The following change adds an HTTP filter that blocks POST/PUT/DELETE requests.

These changes are now merged, so with the read only plugin it's possible to prevent changes being pushed (either directly or for review) and prevent any REST calls other than GET.

 

The missing piece is still the fact that state can be changed by ssh commands.  I've been looking into the possibility of implementing a repository manager that opens the repositories read-only, but that doesn't seem to be very easy.

This is still the missing piece.  Making a read-only repository manager likely won't solve it either, since some of the ssh commands cause changes in the database rather than in the git repository.

If anyone has any ideas how to solve this, I would like to hear them :)

[David Ostrovsky]

On Wednesday, February 28, 2018 at 6:01:04 AM UTC+1, David Pursehouse wrote:

On Tue, Feb 20, 2018 at 10:15 PM David Pursehouse <david.pu...@gmail.com> wrote:

On Thu, Nov 24, 2016 at 6:11 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Steffen,

the "read only" plugin is a Groovy script and can be found at [2].

With regards to REST API, we just added some simple rewrite rules. The missing bit was the state changes that could be triggered via SSH (review / verified by robots or Jenkins).

The read-only window last however for a couple of minutes, so the risk of losing some of them would not have been high.

Setting the projects with "read only" state would work as well, however you would need to record the status "pre-readonly" and resume it correctly afterwards.

[2] https://gerrit.googlesource.com/plugins/scripts/+/HEAD/admin/readonly-1.0.groovy

I've started working on making this into a standalone plugin.

So far I've uploaded 2 changes:

https://gerrit-review.googlesource.com/q/project:plugins%252Freadonly+status:open

The first simply mimics the functionality of the existing groovy plugin, using the commit validator to block all incoming commits.  The following change adds an HTTP filter that blocks POST/PUT/DELETE requests.

These changes are now merged, so with the read only plugin it's possible to prevent changes being pushed (either directly or for review) and prevent any REST calls other than GET.

 

The missing piece is still the fact that state can be changed by ssh commands.  I've been looking into the possibility of implementing a repository manager that opens the repositories read-only, but that doesn't seem to be very easy.

This is still the missing piece.  Making a read-only repository manager likely won't solve it either, since some of the ssh commands cause changes in the database rather than in the git repository.

If anyone has any ideas how to solve this, I would like to hear them :)

I've uploaded new SSH command interceptor extension point: [1] and

added HijackCommandInterceptor to readonly plugin: [2], here in action:

  $ ssh d gerrit version

  gerrit version (dev)

  $ ssh d gerrit plugin enable readonly

  $ ssh d gerrit plugin ls

  Name                           Version    Status   File

  -------------------------------------------------------------------------------

  readonly                       60dedcd-dirty ENABLED  readonly.jar

  $ ssh gerritd gerrit

  SSH subsystem is disabled

  $ ssh gerritd gerrit plugin rm readonly

  $ ssh gerritd gerrit version

  gerrit version (dev)

[1] https://gerrit-review.googlesource.com/162930

[2] https://gerrit-review.googlesource.com/162950

[Luca Milanesio]

That sounds *really* promising :-) Thanks David !!

Luca.

[David Pursehouse]

On Wed, Feb 28, 2018 at 5:28 PM David Ostrovsky <david.o...@gmail.com> wrote:

On Wednesday, February 28, 2018 at 6:01:04 AM UTC+1, David Pursehouse wrote:

On Tue, Feb 20, 2018 at 10:15 PM David Pursehouse <david.pu...@gmail.com> wrote:

On Thu, Nov 24, 2016 at 6:11 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Hi Steffen,

the "read only" plugin is a Groovy script and can be found at [2].

With regards to REST API, we just added some simple rewrite rules. The missing bit was the state changes that could be triggered via SSH (review / verified by robots or Jenkins).

The read-only window last however for a couple of minutes, so the risk of losing some of them would not have been high.

Setting the projects with "read only" state would work as well, however you would need to record the status "pre-readonly" and resume it correctly afterwards.

[2] https://gerrit.googlesource.com/plugins/scripts/+/HEAD/admin/readonly-1.0.groovy

I've started working on making this into a standalone plugin.

So far I've uploaded 2 changes:

https://gerrit-review.googlesource.com/q/project:plugins%252Freadonly+status:open

The first simply mimics the functionality of the existing groovy plugin, using the commit validator to block all incoming commits.  The following change adds an HTTP filter that blocks POST/PUT/DELETE requests.

These changes are now merged, so with the read only plugin it's possible to prevent changes being pushed (either directly or for review) and prevent any REST calls other than GET.

 

The missing piece is still the fact that state can be changed by ssh commands.  I've been looking into the possibility of implementing a repository manager that opens the repositories read-only, but that doesn't seem to be very easy.

This is still the missing piece.  Making a read-only repository manager likely won't solve it either, since some of the ssh commands cause changes in the database rather than in the git repository.

If anyone has any ideas how to solve this, I would like to hear them :)

I've uploaded new SSH command interceptor extension point: [1] and

added HijackCommandInterceptor to readonly plugin: [2], here in action:

This looks good, thanks for doing this.

The only downside, from my point of view, is that it requires the plugin to run against Gerrit master.  So far I was developing the plugin against 2.14.x.

 

  $ ssh d gerrit version

  gerrit version (dev)

  $ ssh d gerrit plugin enable readonly

  $ ssh d gerrit plugin ls

  Name                           Version    Status   File

  -------------------------------------------------------------------------------

  readonly                       60dedcd-dirty ENABLED  readonly.jar

  $ ssh gerritd gerrit

  SSH subsystem is disabled

  $ ssh gerritd gerrit plugin rm readonly

  $ ssh gerritd gerrit version

  gerrit version (dev)

[1] https://gerrit-review.googlesource.com/162930

[2] https://gerrit-review.googlesource.com/162950

--

[Luca Milanesio]

This is not the first time we introduce a new extension point in a stable branch: why don'y we cherry pick to stable-2.14?

As long as it does not break any compatibility and preserves maximum stability, I don't see any problem with it :-)

Additionally, if no plugins are using it, it should be transparent to current Gerrit setups, correct?

Luca.

[David Pursehouse]

On Wed, Feb 28, 2018 at 5:49 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

This is not the first time we introduce a new extension point in a stable branch: why don'y we cherry pick to stable-2.14?

As long as it does not break any compatibility and preserves maximum stability, I don't see any problem with it :-)

I was just about to suggest exactly the same...

 

Additionally, if no plugins are using it, it should be transparent to current Gerrit setups, correct?

Yes, I think so.

[Luca Milanesio]

Additionally, it would allow everyone in the community to perform Blue/Green deployments even during DB upgrades, which is very nice :-)

Luca.

[David Pursehouse]

On Wed, Feb 28, 2018 at 5:55 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

Additionally, it would allow everyone in the community to perform Blue/Green deployments even during DB upgrades, which is very nice :-)

Luca.

On 28 Feb 2018, at 08:52, David Pursehouse <david.pu...@gmail.com> wrote:

On Wed, Feb 28, 2018 at 5:49 PM Luca Milanesio <luca.mi...@gmail.com> wrote:

This is not the first time we introduce a new extension point in a stable branch: why don'y we cherry pick to stable-2.14?

As long as it does not break any compatibility and preserves maximum stability, I don't see any problem with it :-)

I was just about to suggest exactly the same...

I've moved the core gerrit change to stable-2.14 and patched it to make it compile (stable-2.14 doesn't have the source code path restructuring).

I've also fixed up the plugin change per review comments, and confirmed that it works as expected.

Thanks again, David Ostrovsky, for implementing this.