What is the best time for Trac downtime?

[Erik Bray]

Hi all,

We're nearly ready to switch trac.sagemath.org over to its new server.
This will require roughly an hour for me to take down and dump/load
the database, as well as the Trac attachments (there are a lot!) and
to take down git.sagemath.org and transfer the git repository, then
bring everything back up again.

Will also need to coordinate with William on transferring the DNS.

I was thinking sometime over the weekend, but then a lot of Sage
developers are doing this on their spare time, which often comes on
weekends. Sage-devel is an international community so there's
probably no one best time.

Any thoughts on this?

Thanks,
Erik

[Dima Pasechnik]

IMHO the sooner the better, for Sage's trac is basically not working now...

Dima

--

---
You received this message because you are subscribed to the Google Groups "sagemath-admins" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sagemath-admi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Masson]

I agree with Dima: working infrastructure takes top priority.

Erik, on the new server I'm seeing this error on several tickets:

Trac detected an internal error:

OSError: [Errno 2] No such file or directory: '/tmp/tmp9sMYW3'

There was an internal error in Trac. It is recommended that you notify your local Trac administrator with the information needed to reproduce the issue.

To that end, you could  a ticket.

The action that triggered the error was:

GET: /ticket/20769

Is that because you haven't copied over all the data yet?

[Volker Braun]

In any case thats from our trac-git plugin not working, the merge of the branch should have been in that temporary directory

[Travis Scrimshaw]

Hey Erik,
   What will this mean for our git remote setup? Will ssh push/pull still be done via g...@trac.sagemath.org?

Also, IMO, the best time to do the transfer would probably be sometime around 7-8 AM France time on the weekend.

Best,
Travis

[Jeroen Demeyer]

On 2016-06-16 19:45, Erik Bray wrote:
> This will require roughly an hour

One hour is nothing, no reason to postpone it.

[Erik Bray]

On Fri, Jun 17, 2016 at 4:39 AM, Travis Scrimshaw <tsc...@ucdavis.edu> wrote:
> Hey Erik,
> What will this mean for our git remote setup? Will ssh push/pull still be
> done via g...@trac.sagemath.org?

Right. Nothing will change (modulo unexpectedly broken things).

[Erik Bray]

On Thu, Jun 16, 2016 at 9:47 PM, Paul Masson <paulm...@comcast.net> wrote:
> I agree with Dima: working infrastructure takes top priority.
>
> Erik, on the new server I'm seeing this error on several tickets:
>
> Trac detected an internal error:
>
> OSError: [Errno 2] No such file or directory: '/tmp/tmp9sMYW3'
>
> There was an internal error in Trac. It is recommended that you notify your
> local Trac administrator with the information needed to reproduce the issue.
>
> To that end, you could a ticket.
>
> The action that triggered the error was:
>
> GET: /ticket/20769

That's strange, thanks for pointing it out.

> Is that because you haven't copied over all the data yet?

The test server is currently running off a ~1 week old backup of the
data. I don't know why it would be puking some missing temp file
though, or even looking for one in the first place. Might be a bug in
the Sage Trac plugin.

> On Thursday, June 16, 2016 at 10:45:55 AM UTC-7, Erik Bray wrote:
>>
>> Hi all,
>>
>> We're nearly ready to switch trac.sagemath.org over to its new server.
>> This will require roughly an hour for me to take down and dump/load
>> the database, as well as the Trac attachments (there are a lot!) and
>> to take down git.sagemath.org and transfer the git repository, then
>> bring everything back up again.
>>
>> Will also need to coordinate with William on transferring the DNS.
>>
>> I was thinking sometime over the weekend, but then a lot of Sage
>> developers are doing this on their spare time, which often comes on
>> weekends. Sage-devel is an international community so there's
>> probably no one best time.
>>
>> Any thoughts on this?
>>
>> Thanks,
>> Erik
>

> --

> You received this message because you are subscribed to the Google Groups

> "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.

[Erik Bray]

On Fri, Jun 17, 2016 at 10:23 AM, Erik Bray <erik....@gmail.com> wrote:
> On Thu, Jun 16, 2016 at 9:47 PM, Paul Masson <paulm...@comcast.net> wrote:
>> I agree with Dima: working infrastructure takes top priority.
>>
>> Erik, on the new server I'm seeing this error on several tickets:
>>
>> Trac detected an internal error:
>>
>> OSError: [Errno 2] No such file or directory: '/tmp/tmp9sMYW3'
>>
>> There was an internal error in Trac. It is recommended that you notify your
>> local Trac administrator with the information needed to reproduce the issue.
>>
>> To that end, you could a ticket.
>>
>> The action that triggered the error was:
>>
>> GET: /ticket/20769
>
> That's strange, thanks for pointing it out.
>
>> Is that because you haven't copied over all the data yet?
>
> The test server is currently running off a ~1 week old backup of the
> data. I don't know why it would be puking some missing temp file
> though, or even looking for one in the first place. Might be a bug in
> the Sage Trac plugin.

Fixed--was just an issue with permissions on the repository. The web
server needs to be able to write to the repository in order to serve
merge commits--duh :)

[William Stein]

Hi,

(1) It is now newly possible for either me or Harald Schilly to do
anything related to the *.sagemath.org DNS setup.

(2) Mike Hansen is currently looking into migrating wiki.sagemath.org
to the same Google compute engine project...

- William

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Thierry]

On Sat, Jun 18, 2016 at 10:34:03AM -0700, William Stein wrote:
> Hi,
>
> (1) It is now newly possible for either me or Harald Schilly to do
> anything related to the *.sagemath.org DNS setup.
>
> (2) Mike Hansen is currently looking into migrating wiki.sagemath.org
> to the same Google compute engine project...

I guess it is needless to recall from a previous email that:

- all the archeological work is already done,
- config files are cleaned up,
- sysadmin choices (webserver, firewall,...) are compatible with asksage ones
for better cooperation with the collective which is daily working in
maintainig it,
- everythging is documented, so that any kind of migration or upgrade should
take less than 5 minutes (+ download time).

But it seems more interesting to let people duplicate work than letting some
sustainable collective to be built.

Btw, i am still waiting apologies for having been whistled like a dog [1].

Ciao,
A benevolent Sage worker

[1] http://article.gmane.org/gmane.comp.mathematics.sage.devel/86960

[William Stein]

On Saturday, June 18, 2016, Thierry <sage-goo...@lma.metelu.net> wrote:

On Sat, Jun 18, 2016 at 10:34:03AM -0700, William Stein wrote:
> Hi,
>
> (1) It is now newly possible for either me or Harald Schilly to do
> anything related to the *.sagemath.org DNS setup.
>
> (2) Mike Hansen is currently looking into migrating wiki.sagemath.org
> to the same Google compute engine project...

I guess it is needless to recall from a previous email that:

- all the archeological work is already done,
- config files are cleaned up,
- sysadmin choices (webserver, firewall,...) are compatible with asksage ones
  for better cooperation with the collective which is daily working in
  maintainig it,
- everythging is documented, so that any kind of migration or upgrade should
  take less than 5 minutes (+ download time).

I didn't realize all of the above was posted publicly somewhere. Can you please repost the link and save us some time.  Thanks!

Sent from my massive iPhone 6 plus.

[mmarco]

Did you recently enable cloudflare cache services?

I am using Tor to circunvent some pesky firewall, and when trying to push some changes, I get some git_trac.trac_error.TracConnectionError: Forbidden error.

Would it be possible to disable cloudflare? Or at least, to use the option to not blacklist Tor exit nodes, please?

[William Stein]

On Sun, Jun 19, 2016 at 4:09 AM, mmarco <mma...@unizar.es> wrote:
> Did you recently enable cloudflare cache services?
>
> I am using Tor to circunvent some pesky firewall, and when trying to push
> some changes, I get some git_trac.trac_error.TracConnectionError: Forbidden
> error.
>
> Would it be possible to disable cloudflare? Or at least, to use the option
> to not blacklist Tor exit nodes, please?

As far as I can tell I have now disabled cloudflare for everything
trac related. Does this help? Cloudflare is doing nothing but DNS.
trac.sagemath.org is currently 128.208.160.253

-- William

>
> El jueves, 16 de junio de 2016, 19:45:55 (UTC+2), Erik Bray escribió:
>>
>> Hi all,
>>
>> We're nearly ready to switch trac.sagemath.org over to its new server.
>> This will require roughly an hour for me to take down and dump/load
>> the database, as well as the Trac attachments (there are a lot!) and
>> to take down git.sagemath.org and transfer the git repository, then
>> bring everything back up again.
>>
>> Will also need to coordinate with William on transferring the DNS.
>>
>> I was thinking sometime over the weekend, but then a lot of Sage
>> developers are doing this on their spare time, which often comes on
>> weekends. Sage-devel is an international community so there's
>> probably no one best time.
>>
>> Any thoughts on this?
>>
>> Thanks,
>> Erik
>

[Andrey Novoseltsev]

On Sunday, 19 June 2016 20:19:50 UTC-6, William wrote:

On Sun, Jun 19, 2016 at 4:09 AM, mmarco <mma...@unizar.es> wrote:
> Did you recently enable cloudflare cache services?
>
> I am using Tor to circunvent some pesky firewall, and when trying to push
> some changes, I get some git_trac.trac_error.TracConnectionError: Forbidden
> error.
>
> Would it be possible to disable cloudflare? Or at least, to use the option
> to not blacklist Tor exit nodes, please?

As far as I can tell I have now disabled cloudflare for everything
trac related.  Does this help?   Cloudflare is doing nothing but DNS.
trac.sagemath.org is currently 128.208.160.253

 -- William

I could not do "git fetch trac" (timeout, network unreachable) but since I saw somewhere that it may work 1 out of 10 times I kept trying while doing other stuff. No luck. Tried again after receiving your message and it worked right away. That's from our department wired network at the University of Alberta, highly unlikely to be blacklisted somehow.

Andrey

[Dima Pasechnik]

fetch works for me now (via a domestic ISP).

but there is something funny with authentication - fetch via ssh does not ask for ssh passphrase...

[Erik Bray]

On Jun 18, 2016 10:37 PM, "William Stein" <wst...@gmail.com> wrote:
>
>
>
> On Saturday, June 18, 2016, Thierry <sage-goo...@lma.metelu.net> wrote:
>>
>> On Sat, Jun 18, 2016 at 10:34:03AM -0700, William Stein wrote:
>> > Hi,
>> >
>> > (1) It is now newly possible for either me or Harald Schilly to do
>> > anything related to the *.sagemath.org DNS setup.
>> >
>> > (2) Mike Hansen is currently looking into migrating wiki.sagemath.org
>> > to the same Google compute engine project...
>>
>> I guess it is needless to recall from a previous email that:
>>
>> - all the archeological work is already done,
>> - config files are cleaned up,
>> - sysadmin choices (webserver, firewall,...) are compatible with asksage ones
>>   for better cooperation with the collective which is daily working in
>>   maintainig it,
>> - everythging is documented, so that any kind of migration or upgrade should
>>   take less than 5 minutes (+ download time).
>
>
>
> I didn't realize all of the above was posted publicly somewhere. Can you please repost the link and save us some time.  Thanks!

I second that. Where us all of this documented?

I would like to also reiterate my suggestion for an "infrastructure" repository to track configuration files and documentation concerning Sage infrastructure. Currently there are several such repositories for bits and pieces of it spread around random places, with none of it at all comprehensive or documented.

In the process of rebuilding the trac/git server I started such a repository privately, but need some place to put it online where others can access it. That said I still think it should be private. This would be a necessary first step, IMO, to making better use of CM tools (chef/ansible/salt/etc.) to make this more sane in the future.

Would GitHub be a good place to put such a repository? Bitbucket? Anywhere as long as it's private and not actually on any of Sage's infrastructure itself. It would be great to get Thierry's documentation of ask and wiki integrated as well.

Best,
Erik

P.s. Let's please keep the person barbs out of this discussion or literally nothing will be accomplished.

[Harald Schilly]

On Mon, Jun 20, 2016 at 10:00 AM, Erik Bray <erik....@gmail.com> wrote:
> Would GitHub be a good place to put such a repository? Bitbucket? Anywhere
> as long as it's private and not actually on any of Sage's infrastructure
> itself.

I could start a private repo at github, where we have our sagemath
group. Just tell me a good name, maybe just "infrastructure" ?

-- h

[Harald Schilly]

On Mon, Jun 20, 2016 at 10:02 AM, Harald Schilly
<harald....@gmail.com> wrote:
> private repo at github

ok, wait, I thought that private repos at github are free now, but it
doesn't seem like that.

-- h

[Harald Schilly]

On Mon, Jun 20, 2016 at 10:02 AM, Harald Schilly
<harald....@gmail.com> wrote:

ok, I started

https://bitbucket.org/sagemath/infrastructure

and added you as a team member.

-- h

[Jeroen Demeyer]

On 2016-06-16 19:45, Erik Bray wrote:

> We're nearly ready to switch trac.sagemath.org over to its new server.

What's the status on this? Are the any obstructions remaining to do the
switch, or is it just a matter of finding time to actually do it?

[Erik Bray]

Mostly the latter.

I would like to understand what is currently the problem with trac/git in case it impacts the new server. But I suspect it's something in between the client and our apache server, such as a misconfigured proxy.

[mmarco]

It seems to be ok now. Thanks.

[Erik Bray]

On Mon, Jun 20, 2016 at 10:10 AM, Harald Schilly

Thanks Harald--

Would you like to coordinate a time to do the migration? Since we are
(I think?) in the same time zone that might be easiest.

Mostly I need to take the current {trac,git}.sagemath.org offline,
migrate the data, bring the new server online, do a few tests, and
then ping you to switch the DNS records over to point to the new IP.

[Harald Schilly]

On Mon, Jun 20, 2016 at 3:54 PM, Erik Bray <erik....@gmail.com> wrote:
>
> Would you like to coordinate a time to do the migration? Since we are
> (I think?) in the same time zone that might be easiest.

I'm UTC+2 (CEST), so sure, I'm happy to switch around DNS entries
during my day … and if it still happens to be night time for me, there
is also William at UTC-7 (PDT) to take over.

-- harald

[Erik Bray]

Is Wednesday morning OK? I can start the migration around 8:00 AM.
With luck have everything done by work time for many people.

I could do it even tomorrow, but I want to give everybody at least
24hrs notice, so Wednesday might be better.

[Harald Schilly]

On Mon, Jun 20, 2016 at 4:08 PM, Erik Bray <erik....@gmail.com> wrote:
> Is Wednesday morning OK?

yes, sounds good. just email me and wst...@gmail.com and one of us
will do it ASAP

-- harald

[William Stein]

Yep.

Also, I created

https://github.com/williamstein/sagemath-infrastructure

since I have been paying for github for years, so have private repos.
To me it's easier to only have to switch between two platforms
(github and trac), rather than three.

-- William

>
> -- harald