trac slow/unresponsive

[Ralf Stephan]

Pages need forever. Patchbots are stalling.

The ping is 200ms so it's probably not the net.

[Eric Gourgoulhon]

It's OK for me (from Paris). Do you still have any issue?

Eric.

[Ralf Stephan]

No, it seems to have been resolved for some time now.

On Mon, Jun 13, 2016 at 2:34 PM Eric Gourgoulhon <egourg...@gmail.com> wrote:

It's OK for me (from Paris). Do you still have any issue?

Eric.

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/XMvRoZvEVIQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

Currently, this is happening for me right now. Read-only access is still
mostly working, but changing anything takes forever.

[Paul Masson]

Similar for me in San Francisco, except can't make any changes. Server keeps timing out.

[William Stein]

Hi,

I just typed "service restart apache2" -- is it still slow? I did a
quick test and it seemed to be working now.

William

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William (http://wstein.org)

[Paul Masson]

No change for me. Still can't modify tickets. Server timeout.

[Jeroen Demeyer]

On 2016-06-15 20:59, William Stein wrote:
> Hi,
>
> I just typed "service restart apache2" -- is it still slow? I did a
> quick test and it seemed to be working now.

Still doesn't work for me.

[William Stein]

I can restart the machine if you guys want...

[Jeroen Demeyer]

On 2016-06-15 21:15, William Stein wrote:
> I can restart the machine if you guys want...

Maybe you should.

[Paul Masson]

Is there a firm plan in place to move this database to a hosting service? It's discouraging to spend time trying to help, only to run into machine issues.

[William Stein]

On Wed, Jun 15, 2016 at 4:45 PM, Paul Masson <paulm...@comcast.net> wrote:
> Is there a firm plan in place to move this database to a hosting service?
> It's discouraging to spend time trying to help, only to run into machine
> issues.
>

Yes. Everything is mostly setup and ready to go on Google Compute
engine, ... but Erik Bray I guess still hasn't given the green light.

You can test out the server he has staged here:

http://104.197.143.230/

I have also rebooted the live actual trac.sagemath.org server now.
Hopefully this will make it faster...

William

[Paul Masson]

No change for me. Still can't modify a ticket. Server not responding when submit change.

[Erik Bray]

On Jun 16, 2016 00:05, "William Stein" <wst...@gmail.com> wrote:
>
> On Wed, Jun 15, 2016 at 4:45 PM, Paul Masson <paulm...@comcast.net> wrote:
> > Is there a firm plan in place to move this database to a hosting service?
> > It's discouraging to spend time trying to help, only to run into machine
> > issues.
> >
>
> Yes.  Everything is mostly setup and ready to go on Google Compute
> engine, ... but Erik Bray I guess still hasn't given the green light.
>
> You can test out the server he has staged here:
>
>     http://104.197.143.230/
>
> I have also rebooted the live actual trac.sagemath.org server now.
> Hopefully this will make it faster...

Yup. I'm actively working on it. I'm just finishing up some updates to Sage's Trac plugin to fix some issues (particularly with the SSH key handling) that I wasn't satisfied with in the process of reconstructing the existing Trac + git setup.

Should have it done soon.

[Jeroen Demeyer]

On 2016-06-16 00:05, William Stein wrote:
> I have also rebooted the live actual trac.sagemath.org server now.
> Hopefully this will make it faster...

It still doesn't work. I am not able to change the description of ticket
#20686 for example.

[William Stein]

Well I have no ideas except: Erik, I'm standing by waiting to change
the DNS when you are ready!!!!!

[Jeroen Demeyer]

For what it's worth: it really seems to depend on the ticket number.
Most tickets work without problems, but #20686 for example is still broken.

[Ralf Stephan]

#17790 ticket description change does not work either, so no simple rules there.

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/XMvRoZvEVIQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

[Erik Bray]

Apologies for the server yo-yo-ing. I'm trying to debug what's going on here.

> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Erik Bray]

Bizarrely I was able to update this ticket fine over SSL, but not without SSL.

Wasn't someone talking about doing something with CloudFlare? Could
that be related?

[Travis Scrimshaw]

FYI - This is also broken on #20836.

Best,
Travis

[Erik Bray]

On Fri, Jun 17, 2016 at 4:45 PM, Travis Scrimshaw <tsc...@ucdavis.edu> wrote:
> FYI - This is also broken on #20836.

The issue here is with the web server, and not with Trac itself. If
there are problems posting to any particular ticket it's just, I
think, that their request headers are large enough that it takes too
long to read before dying in whatever way it's dying which I haven't
figured out.

[Volker Braun]

Possibly because trac static assets need to be served by the web server instead of trac's own handler? Otherwise trac is really slow, we had that before.

[Erik Bray]

On Jun 17, 2016 16:52, "Volker Braun" <vbrau...@gmail.com> wrote:
>
> Possibly because trac static assets need to be served by the web server instead of trac's own handler? Otherwise trac is really slow, we had that before.

No it has nothing to do with that.  Need to dig a little deeper, but I'm going home for the night. I put some debug code in the mod_wsgi script to see what's going on but it's not telling me much.  It blocks on reading environ['wsgi.input'] [1], which is a file-like handle to the form post data in the body of the request.

This happens regardless of whether Trac tries to read or my own code. The request body just never makes it, like Apache is just dropping the connection before the request body can be read. I wonder if we're being DoS'd (except GETs are working just fine).

http://wsgi.readthedocs.io/en/latest/definitions.html#envvar-wsgi.input

> On Friday, June 17, 2016 at 4:39:22 PM UTC+2, Erik Bray wrote:
>>
>> On Fri, Jun 17, 2016 at 1:52 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> > On 2016-06-16 09:09, Jeroen Demeyer wrote:
>> >>
>> >> On 2016-06-16 00:05, William Stein wrote:
>> >>>
>> >>> I have also rebooted the live actual trac.sagemath.org server now.
>> >>> Hopefully this will make it faster...
>> >>
>> >>
>> >> It still doesn't work. I am not able to change the description of ticket
>> >> #20686 for example.
>> >
>> >
>> > For what it's worth: it really seems to depend on the ticket number. Most
>> > tickets work without problems, but #20686 for example is still broken.
>>
>> Bizarrely I was able to update this ticket fine over SSL, but not without SSL.
>>
>> Wasn't someone talking about doing something with CloudFlare?  Could
>> that be related?
>

[Volker Braun]

On Friday, June 17, 2016 at 5:35:06 PM UTC+2, Erik Bray wrote: 

It blocks on reading environ['wsgi.input'] [1], which is a file-like handle to the form post data in the body of the request. This happens regardless of whether Trac tries to read or my own code. The request body just never makes it, like Apache is just dropping the connection before the request body can be read.

Sounds like the client is too slow, possibly due to packet loss / retransmission (we had that at UW at one point, easy to see in packet trace); 

Reading input is blocking. There are some speed limits depending on web server setup (e.g. mod_reqtimeout). That would explain why SSL works (data likely arrives in bigger chunks).

[Erik Bray]

On Jun 17, 2016 6:17 PM, "Volker Braun" <vbrau...@gmail.com> wrote:
>
> On Friday, June 17, 2016 at 5:35:06 PM UTC+2, Erik Bray wrote: 
>>
>> It blocks on reading environ['wsgi.input'] [1], which is a file-like handle to the form post data in the body of the request. This happens regardless of whether Trac tries to read or my own code. The request body just never makes it, like Apache is just dropping the connection before the request body can be read.
>
>
> Sounds like the client is too slow, possibly due to packet loss / retransmission (we had that at UW at one point, easy to see in packet trace); 

Maybe but that would seem to implicate something on UW's end if multiple people outside their network are having issues.

> Reading input is blocking. There are some speed limits depending on web server setup (e.g. mod_reqtimeout). That would explain why SSL works (data likely arrives in bigger chunks).

I played around quite a bit with the mod_reqtimeout settings but it didn't seem to make a difference.

[Jeroen Demeyer]

On 2016-06-17 16:39, Erik Bray wrote:
> On Fri, Jun 17, 2016 at 1:52 PM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
>> On 2016-06-16 09:09, Jeroen Demeyer wrote:
>>>
>>> On 2016-06-16 00:05, William Stein wrote:
>>>>
>>>> I have also rebooted the live actual trac.sagemath.org server now.
>>>> Hopefully this will make it faster...
>>>
>>>
>>> It still doesn't work. I am not able to change the description of ticket
>>> #20686 for example.
>>
>>
>> For what it's worth: it really seems to depend on the ticket number. Most
>> tickets work without problems, but #20686 for example is still broken.
>
> Bizarrely I was able to update this ticket fine over SSL, but not without SSL.

With SSL, I get a 500 Internal Server Error.

Without SSL, it's still broken.

[Ralf Stephan]

I just noticed that the #17790 problem is user-specific because tmonteil

could change the description yesterday while I can still not do so.

--
You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/XMvRoZvEVIQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.

[Erik Bray]

That's because I took down the SSL site before leaving work yesterday while just trying things. I guess I didn't put it back up again.

[Stephan Ehlen]

I'm having a lot of trouble pushing changes using git trac push - could this be related?

Using the web interface works without any problems for me - except that in Safari, I don't stay logged in at all but it works fine in Chrome.

But pushing just works every 10th time I try and otherwise gives me a timeout.

[William Stein]

On Sun, Jun 19, 2016 at 5:39 AM, Stephan Ehlen
<stephan...@gmail.com> wrote:
> I'm having a lot of trouble pushing changes using git trac push - could this
> be related?
> Using the web interface works without any problems for me - except that in
> Safari, I don't stay logged in at all but it works fine in Chrome.
> But pushing just works every 10th time I try and otherwise gives me a
> timeout.

I've disabled a few things that might be causing trouble; can you try again?

William

>
>
> On Saturday, June 18, 2016 at 3:25:40 AM UTC-4, Erik Bray wrote:
>>
>> On Jun 18, 2016 09:07, "Jeroen Demeyer" <jdem...@cage.ugent.be> wrote:
>> >
>> > On 2016-06-17 16:39, Erik Bray wrote:
>> >>
>> >> On Fri, Jun 17, 2016 at 1:52 PM, Jeroen Demeyer <jdem...@cage.ugent.be>
>> >> wrote:
>> >>>
>> >>> On 2016-06-16 09:09, Jeroen Demeyer wrote:
>> >>>>
>> >>>>
>> >>>> On 2016-06-16 00:05, William Stein wrote:
>> >>>>>
>> >>>>>
>> >>>>> I have also rebooted the live actual trac.sagemath.org server now.
>> >>>>> Hopefully this will make it faster...
>> >>>>
>> >>>>
>> >>>>
>> >>>> It still doesn't work. I am not able to change the description of
>> >>>> ticket
>> >>>> #20686 for example.
>> >>>
>> >>>
>> >>>
>> >>> For what it's worth: it really seems to depend on the ticket number.
>> >>> Most
>> >>> tickets work without problems, but #20686 for example is still broken.
>> >>
>> >>
>> >> Bizarrely I was able to update this ticket fine over SSL, but not
>> >> without SSL.
>> >
>> >
>> > With SSL, I get a 500 Internal Server Error.
>> >
>> > Without SSL, it's still broken.
>>
>> That's because I took down the SSL site before leaving work yesterday
>> while just trying things. I guess I didn't put it back up again.
>

[Ralf Stephan]

On Sat, Jun 18, 2016 at 9:20 AM Ralf Stephan <gtr...@gmail.com> wrote:

I just noticed that the #17790 problem is user-specific because tmonteil

could change the description yesterday while I can still not do so.

Cool, this seems fixed now with all tickets that I couldn't edit.

[stephan...@gmail.com]

Sorry for the delay, I’ve been traveling.
It did work immediately.

Thanks!

Stephan

> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/XMvRoZvEVIQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.