Uninterrupted LeoFS manager upgrades

[Peter Toth]

Hi all,

The following page on upgrading LeoFS http://leo-project.net/leofs/docs/admin_guide/admin_guide_10.html suggest to stop both managers.

When both managers are down for a longer period read/write operations start to fail.

As best practice, what is the correct way to upgrade managers without disruption?

Can this be avoided by just taking one manager offline at a time for upgrading?

Maybe the doc was intended this way and I just misrepresented the chart.

Thanks

Peter 

[yoshiyuki kanno]

Hi Peter,

> When both managers are down for a longer period read/write operations start to fail.

This should not happen if you would mean read/write operation for
GET/PUT/DELETEing the objects stored in LeoFS.

While managers stopped,
You can not do the operations related to the cluster topology like
- Attach/Detach/Rebalance storage nodes
- Join/Remove cluster for MDC
and operations related to the S3 information like
- get/add/delete user/endpoint/bucket(s)

Please let me know if operations(GET/PUT/DELETE to the objects start
to fail when both managers are down for a longer period.
This should be fixed.

> As best practice, what is the correct way to upgrade managers without disruption?
> Can this be avoided by just taking one manager offline at a time for upgrading?

Just in case, as I said at the above,
LeoFS can operate any workload coming from leo_gateway without
disruption while managers stopped.

Best,
Kanno.

> --
> You received this message because you are subscribed to the Google Groups
> "LeoProject.LeoFS" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leoproject_leo...@googlegroups.com.
> To post to this group, send email to leoproje...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/leoproject_leofs/aa0909df-449c-4abd-8a98-bf922dba528f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Yoshiyuki Kanno
LeoFS Committer(http://www.leofs.org)
--------------------------------------------------
Stoic Corp.
URL: http://www.stoic.co.jp/
E-mail: yoshiyu...@stoic.co.jp

[Peter Toth]

Hi Kanno,

Yes that is correct, if I shutdown both managers in the cluster, S3 PUT/GET operations start to fail.

Both public/private buckets become inaccessible after around 60 seconds.

If at least one manager is online all is OK.

Kind regards,

Peter 

[yoshiyuki kanno]

> Yes that is correct, if I shutdown both managers in the cluster, S3 PUT/GET operations start to fail.
> Both public/private buckets become inaccessible after around 60 seconds.
> If at least one manager is online all is OK.

My first guess is
https://github.com/leo-project/leofs/blob/1.3.2.1/apps/leo_gateway/priv/leo_gateway.conf#L85-L86
this configuration is set to 60 and
after 60 seconds passed, the operations start to fail due to failing
to retrieve the latest bucket information that is synced with
manager's one.
But this should not happen according the spec so there may be some bugs.
We are going to look into that.

Also just in case,
Could you share the settings on every nodes in each cluster also error
log files if presents?

Best,
Kanno.

> https://groups.google.com/d/msgid/leoproject_leofs/63a57baa-e3e5-4c2a-9a45-13d73e022ef7%40googlegroups.com.

[yoshiyuki kanno]

Hi Peter,

My first guess seems to be right.
Filed this issue on https://github.com/leo-project/leofs/issues/642
and will be fixed in the next release.

Thanks for your contribution.