sagenb issues

[MichTex]

I've been trying to login to sagenb this morning. It took over a
minute for the server to respond when I first went to the login page.
After another long wait, it finally logged in. Then when I clicked on
the link to the worksheet that I wanted, there was a third long wait,
which finally produced a 503 error. Now all attempts to reach the
server simply produce the 'Service Temporarily Unavailable' message. I
know that traffic on sagenb has greatly increased in the last few
weeks, so perhaps the poor machine is maxed out in some way.

--Bill

[Jason Grout]

This is in the error log:

/sagenb/sage_install/sage-4.7.1/local/lib/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x56)[0x7fe74a7474f6]
/sagenb/sage_install/sage-4.7.1/local/lib/libpython2.6.so.1.0[0x7fe74a7813cd]
/lib/libpthread.so.0[0x7fe74a4563f7]
/lib/libc.so.6(clone+0x6d)[0x7fe749b3dbbd]

------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred in Sage.
This probably occurred because a *compiled* component of Sage has a bug
in it and is not properly wrapped with sig_on(), sig_off(). You might
want to run Sage under gdb with 'sage -gdb' to debug this.
Sage will now terminate.
------------------------------------------------------------------------
Segmentation fault

(full error log at http://pastebin.com/FQzxqsZg)

I'm restarting the server, as it says it is stopped currently.

Thanks,

Jason

[MichTex]

Thanks! It is back up, although it is still very slow. I'm guessing
that it has a heavy load already.

--Bill

> (full error log athttp://pastebin.com/FQzxqsZg)

[Jason Grout]

On 10/20/11 10:07 AM, MichTex wrote:
> Thanks! It is back up, although it is still very slow. I'm guessing
> that it has a heavy load already.
>

The load isn't particularly heavy---only about 8 people. It was
handling a much bigger load yesterday, for example. The server was even
slow serving up the webpage when I tried to get it from the computer the
server is running on, so I'm not sure what is going on. I restarted it
once more and it seems fine now. Let us know if it gets slow again.

Thanks,

Jason

[William Stein]

It got very slow again and I'm restarting it. I'm worried that has
something to do with what you (Jason) did, e.g., switching to twisted
11?

-- William

> Thanks,
>
> Jason
>
>
> --
> To post to this group, send email to sage-s...@googlegroups.com
> To unsubscribe from this group, send email to
> sage-support...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-support
> URL: http://www.sagemath.org
>

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[MichTex]

FWIW, I just tested again, and it seemed a lot snappier -- no delays
at all.
--Bill

On Oct 20, 2:02 pm, William Stein <wst...@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 8:31 AM, Jason Grout
>

[Jason Grout]

On 10/20/11 1:02 PM, William Stein wrote:
> On Thu, Oct 20, 2011 at 8:31 AM, Jason Grout
> <jason...@creativetrax.com> wrote:
>> On 10/20/11 10:07 AM, MichTex wrote:
>>>
>>> Thanks! It is back up, although it is still very slow. I'm guessing
>>> that it has a heavy load already.
>>>
>>
>> The load isn't particularly heavy---only about 8 people. It was handling a
>> much bigger load yesterday, for example. The server was even slow serving
>> up the webpage when I tried to get it from the computer the server is
>> running on, so I'm not sure what is going on. I restarted it once more and
>> it seems fine now. Let us know if it gets slow again.
>>
>
> It got very slow again and I'm restarting it. I'm worried that has
> something to do with what you (Jason) did, e.g., switching to twisted
> 11?

The recent changes certainly seem like the most likely culprit.
Switching back to the old versionis should be as easy as switching one
symbolic link to point to sage-4.7 instead of sage-4.7.1:

cd /sagenb/sage_install/
rm sage_sagenb
ln -s sage sage_sagenb

and then restarting sagenb.

Thanks,

Jason

[William Stein]

On Thu, Oct 20, 2011 at 11:39 AM, Jason Grout

If/when it gets slow again (today?) maybe we can look and see if we
have a clue why it is so slow.

[kcrisman]

demo.sagenb.org is not responding, or at any rate is responding very,
very slowly. This is bad, because I just sent students there (see
sage-devel thread) and now they aren't able to do HW :( I assume this
is related to the general sagenb issues?

[William Stein]

On Thursday, October 20, 2011, kcrisman <kcri...@gmail.com> wrote:
> demo.sagenb.org is not responding, or at any rate is responding very,
> very slowly.  This is bad, because I just sent students there (see

I will restart all the servers from my cell phone now, then revert Jason's change when I get home so this stops happening.  Clearly something he did must have major negative consequences.

> sage-devel thread) and now they aren't able to do HW :(  I assume this
> is related to the general sagenb issues?
>

[kcrisman]

On Oct 21, 12:14 am, William Stein <wst...@gmail.com> wrote:

> On Thursday, October 20, 2011, kcrisman <kcris...@gmail.com> wrote:
> > demo.sagenb.org is not responding, or at any rate is responding very,
> > very slowly.  This is bad, because I just sent students there (see
>
> I will restart all the servers from my cell phone now, then revert Jason's
> change when I get home so this stops happening.  Clearly something he did

Thank you so much, the server seems snappy now. I'll test it again in
class this morning with all (8) of us.

> must have major negative consequences.
>

And ones that only seem to manifest themselves with larger numbers of
users?

- kcrisman

[Jason Grout]

Then this points to the problem *not* being the changes I made. Nothing
I did should have affected demo.sagenb.org. That is running a
completely separate Sage installation with the notebook from July.

Thanks,

Jason

[Jason Grout]

On 10/20/11 11:14 PM, William Stein wrote:
>
>
> On Thursday, October 20, 2011, kcrisman <kcri...@gmail.com

> <mailto:kcri...@gmail.com>> wrote:
> > demo.sagenb.org <http://demo.sagenb.org> is not responding, or at any

> rate is responding very,
> > very slowly. This is bad, because I just sent students there (see
>
>
> I will restart all the servers from my cell phone now, then revert
> Jason's change when I get home so this stops happening. Clearly
> something he did must have major negative consequences.

There might have been a change in Rado's branch or the few bugfixes that
I made over the last weekend that slowed things down. But the fact that
demo.sagenb.org is having problems, which is running the codebase from
June (i.e., does not include any of the changes Rado or I made since
June), indicates that the problem is not in the changes Rado and I have
committed to the sage notebook since June.

Thanks,

Jason

[William Stein]

Interesting.  I've thus not reverted anything.

> Thanks,
>
> Jason

[kcrisman]

On Oct 21, 11:01 am, William Stein <wst...@gmail.com> wrote:
> On Friday, October 21, 2011, Jason Grout <jason-s...@creativetrax.com>

> wrote:> On 10/20/11 11:14 PM, William Stein wrote:
>

> >> On Thursday, October 20, 2011, kcrisman <kcris...@gmail.com

> >> <mailto:kcris...@gmail.com>> wrote:
> >>  > demo.sagenb.org <http://demo.sagenb.org> is not responding, or at any
> >> rate is responding very,
> >>  > very slowly.  This is bad, because I just sent students there (see
>
> >> I will restart all the servers from my cell phone now, then revert
> >> Jason's change when I get home so this stops happening.  Clearly
> >> something he did must have major negative consequences.
>
> > There might have been a change in Rado's branch or the few bugfixes that I
>
> made over the last weekend that slowed things down.  But the fact that
> demo.sagenb.org is having problems, which is running the codebase from June
> (i.e., does not include any of the changes Rado or I made since June),
> indicates that the problem is not in the changes Rado and I have committed
> to the sage notebook since June.
>

Ah, I disagree here. For instance, {demo,flask,www}.sagenb.org are
all once again misbehaving (really slow) as I type, whereas the one
for this summer's PREP (hosted not at UW) is working fine and quickly.

Now, how do I disagree? Not in that there is something wrong with the
code in demo! demo.sagenb.org being slow is a SYMPTOM of whatever
problem in the other ones is causing the trouble. After all,
sagenb.org was on flask (not the latest, though) just a week ago when
we had 300+ successful open worksheets at a time, right?

I have no idea, but a futile guess is that www.sagenb.org being on the
new codebase is causing the actual machine to grind to a halt (they
are all on the same server, correct?). I'm otherwise not having any
connectivity issues - only on the sagenb.org farm. Even sagemath.org
works fine.

- kcrisman

[William Stein]

On Thu, Oct 20, 2011 at 11:39 AM, Jason Grout

Instead I just commented these two lines out of the admin script:

#elif self.name == 'sage_notebook-sagenb':
# sage = 'sage-sagenb'

then did

sagenb@mod:~/servers$ ./admin --restart=sagenb

Instantly after doing this *demo* became much faster.
I noticed before doing this that there was a single sagenb server
Python process running in top using a lot of cpu..

-- William

[William Stein]

On Fri, Oct 21, 2011 at 8:15 AM, kcrisman <kcri...@gmail.com> wrote:
>
>
> On Oct 21, 11:01 am, William Stein <wst...@gmail.com> wrote:
>> On Friday, October 21, 2011, Jason Grout <jason-s...@creativetrax.com>
>> wrote:> On 10/20/11 11:14 PM, William Stein wrote:
>>
>> >> On Thursday, October 20, 2011, kcrisman <kcris...@gmail.com
>> >> <mailto:kcris...@gmail.com>> wrote:
>> >>  > demo.sagenb.org <http://demo.sagenb.org> is not responding, or at any
>> >> rate is responding very,
>> >>  > very slowly.  This is bad, because I just sent students there (see
>>
>> >> I will restart all the servers from my cell phone now, then revert
>> >> Jason's change when I get home so this stops happening.  Clearly
>> >> something he did must have major negative consequences.
>>
>> > There might have been a change in Rado's branch or the few bugfixes that I
>>
>> made over the last weekend that slowed things down.  But the fact that
>> demo.sagenb.org is having problems, which is running the codebase from June
>> (i.e., does not include any of the changes Rado or I made since June),
>> indicates that the problem is not in the changes Rado and I have committed
>> to the sage notebook since June.
>>
>
> Ah, I disagree here.  For instance, {demo,flask,www}.sagenb.org are
> all once again misbehaving (really slow) as I type, whereas the one
> for this summer's PREP (hosted not at UW) is working fine and quickly.
>
> Now, how do I disagree?  Not in that there is something wrong with the
> code in demo!  demo.sagenb.org being slow is a SYMPTOM of whatever
> problem in the other ones is causing the trouble.

Sadly, I think you're right.

> After all,
> sagenb.org was on flask (not the latest, though) just a week ago when
> we had 300+ successful open worksheets at a time, right?

Yes, that was pretty normal. It hasn't got very high recently, I
think, because of the speed regression.

We'll see what happens today: I just reverted back to how things were
a week ago.

>
> I have no idea, but a futile guess is that www.sagenb.org being on the
> new codebase is causing the actual machine to grind to a halt (they
> are all on the same server, correct?).   I'm otherwise not having any
> connectivity issues - only on the sagenb.org farm.  Even sagemath.org
> works fine.
>
> - kcrisman
>

[Jason Grout]

Yes, that would work too.

> then did
>
> sagenb@mod:~/servers$ ./admin --restart=sagenb
>
> Instantly after doing this *demo* became much faster.
> I noticed before doing this that there was a single sagenb server
> Python process running in top using a lot of cpu..

Okay, then it certainly does point to kcrisman's conclusion. I'll see
if I can diagnose the problem, but probably won't have time to do so
until early next week.

Thanks,

Jason

[Jason Grout]

On 10/21/11 10:32 AM, William Stein wrote:

> Instantly after doing this *demo* became much faster.
> I noticed before doing this that there was a single sagenb server
> Python process running in top using a lot of cpu..

For future reference, it would have been *great* to get a stack trace of
the currently executing code for the runaway process. I followed the
instructions in a stack overflow answer [1] and can now do, as the
sagenb user on mod:

gdb -p <PID of sagenb process>
pystack

and get a stack frame listing of a running process.

As I have time, I'll look into where there might be a performance
regression, but there have been a lot of code changes between June and
now, and we haven't seen the performance regression on test.sagenb.org.

Is there a possibility of running the new codebase for sagenb.org again
and getting the stack trace when the process is using that much CPU?

Thanks,

Jason

[1]
http://stackoverflow.com/questions/132058/getting-stack-trace-from-a-running-python-application/147114#147114

[kcrisman]

> As I have time, I'll look into where there might be a performance
> regression, but there have been a lot of code changes between June and
> now, and we haven't seen the performance regression on test.sagenb.org.

Well, there are not very many users on test.sagenb.org. The problem
seems to only manifest itself when the new code is used on a really
heavily used server.

And to be fair, most of the new code is probably awesome! Do you
have a sense that any of the new material is a likely culprit? (For
instance, the new code re-enabling "viewable" sharing?)

> Is there a possibility of running the new codebase for sagenb.org again
> and getting the stack trace when the process is using that much CPU?

Excellent idea. But maybe send out an email to sage-support or sage-
edu first to let people know, since there is a likely slowdown
involved!

- kcrisman

[William Stein]

On Tue, Oct 25, 2011 at 9:50 AM, Jason Grout
<jason...@creativetrax.com> wrote:
> On 10/21/11 10:32 AM, William Stein wrote:
>
>> Instantly after doing this *demo* became much faster.
>> I noticed before doing this that there was a single sagenb server
>> Python process running in top using a lot of cpu..
>
> For future reference, it would have been *great* to get a stack trace of the
> currently executing code for the runaway process.  I followed the
> instructions in a stack overflow answer [1] and can now do, as the sagenb
> user on mod:
>
> gdb -p <PID of sagenb process>
> pystack
>
> and get a stack frame listing of a running process.
>
> As I have time, I'll look into where there might be a performance
> regression, but there have been a lot of code changes between June and now,
> and we haven't seen the performance regression on test.sagenb.org.
>
> Is there a possibility of running the new codebase for sagenb.org again and
> getting the stack trace when the process is using that much CPU?
>

I'm personally not super comfortable with this, especially if I have
to have to be the one responsible for fixing things when they
inevitably break and cause trouble. From the second I switched back
to the old version until now, I haven't had any complaints about any
*.sagenb.org's not working, and haven't had to restart them, etc. So
I'm only OK with this if you're 100% on deck, and I don't have to do
anything, and you switch it all back as soon as you can.

Of course, the right thing to do is write code to simulate a heavy
load, and run it against test.sagenb.org (or something like that).

-- William

> Thanks,
>
> Jason
>
>
> [1]
> http://stackoverflow.com/questions/132058/getting-stack-trace-from-a-running-python-application/147114#147114
>
>
>

[William Stein]

(at least that is, this quarter -- I plan to work a lot on the
notebook next quarter)

[Jason Grout]

On 10/25/11 12:54 PM, William Stein wrote:

> I'm personally not super comfortable with this, especially if I have
> to have to be the one responsible for fixing things when they
> inevitably break and cause trouble. From the second I switched back
> to the old version until now, I haven't had any complaints about any
> *.sagenb.org's not working, and haven't had to restart them, etc. So
> I'm only OK with this if you're 100% on deck, and I don't have to do
> anything, and you switch it all back as soon as you can.

I agree in that I'm not super comfortable with this either. And I agree
that the right solution is to write stress-testing code. Let's see:
Rado had something that was stress-testing things back in March. Rado:
if you have time, can you try hammering test.sagenb.org with that test
suite?

I won't do the sagenb.org test until I have time to deal with whatever
comes up from it.

Thanks,

Jason

[MichTex]

I have been using sagenb without difficulty now for several days. I've
been using it again this morning, but just now I started getting the
'could not connect to www.sagenb.org' message when I tried to save my
worksheet. I tried opening another window to sagenb and got the same
result. After a couple of minutes, the server finally responded to my
first request. I then tried re-running the computation in the last
cell that I had been working on. It did run, although very slowly. I
have tried a couple of other things, and they were all very slow as
well. I don't think it is a connection problem from my end.

--Bill Cavnar

[William Stein]

Of possible relevance is that the machine that sagenb is running on is
heavily loaded right now, due to me upgrading the system-wide Sage
install on that machine, and building in parallel. This should be
finished in a few minutes.

William

William

[leif]

On 26 Okt., 17:47, William Stein <wst...@gmail.com> wrote:
> Of possible relevance is that the machine that sagenb is running on is
> heavily loaded right now, due to me upgrading the system-wide Sage
> install on that machine, and building in parallel.

FWIW, slightly off-topic:

Setting MAKE to "make -jN" when (re-)building Sage doesn't limit the
*total* number of build jobs to N; it's usually better to also use '-
lM' [1], where M is a floating-point number, telling 'make' to not
spawn further jobs if the (total) system load exceeds M. This is
especially useful if you already have other programs / processes
running which significantly consume CPU time and / or RAM, or
shouldn't suffer from too high load.

(RAM is certainly not an issue on mod, but If you choose N too large
without limiting the max. sysload, you may also quickly run out of
physical memory, such that the machine starts swapping, which of
course slows down the build process or may even "freeze" your machine
[at least for a while].)

[I'm not sure whether that's mentioned anywhere in the Sage
documentation, but rather doubt it is.]


-leif

[1] http://www.gnu.org/software/make/manual/make.html#Options-Summary

[Volker Braun]

On Thursday, October 27, 2011 11:01:19 AM UTC-4, leif wrote:

On 26 Okt., 17:47, William Stein <wst...@gmail.com> wrote:
Setting MAKE to "make -jN" when (re-)building Sage doesn't limit the
*total* number of build jobs to N;

It does limit the total number of build jobs within the current make. Of course it doesn't take into account make jobs that are run by different users, that run on different computers, that were invoked at a different time, etc.

[Mark Florisson]

On Oct 26, 4:47 pm, William Stein <wst...@gmail.com> wrote:

> On Wed, Oct 26, 2011 at 8:44 AM, MichTex <bill.cav...@gmail.com> wrote:
> > I have been using sagenb without difficulty now for several days. I've
> > been using it again this morning, but just now I started getting the

> > 'could not connect towww.sagenb.org'message when I tried to save my

> > worksheet. I tried opening another window to sagenb and got the same
> > result. After a couple of minutes, the server finally responded to my
> > first request. I then tried re-running the computation in the last
> > cell that I had been working on. It did run, although very slowly. I
> > have tried a couple of other things, and they were all very slow as
> > well. I don't think it is a connection problem from my end.
>
> > --Bill Cavnar
>
> Of possible relevance is that the machine that sagenb is running on is
> heavily loaded right now, due to me upgrading the system-wide Sage
> install on that machine, and building in parallel.   This should be
> finished in a few minutes.
>
> William
>
> William
>
>
>
> > --
> > To post to this group, send email to sage-s...@googlegroups.com
> > To unsubscribe from this group, send email to sage-support...@googlegroups.com

> > For more options, visit this group athttp://groups.google.com/group/sage-support

> > URL:http://www.sagemath.org
>
> --
> William Stein
> Professor of Mathematics
> University of Washingtonhttp://wstein.org

I'm currently getting the following message:

"
Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /.

Reason: Error reading from remote server
"

when trying to log in. Sometimes I get the same message just by going
to sagenb.org.

Mark

[William Stein]

That's probably because there are over 500 simultaneous open worksheets right now.   I've reset the servers but it will probably get heavily loaded again soon enough.   Obviously we need to start planning for the next level in scalability...

> Mark

>
> --
> To post to this group, send email to sage-s...@googlegroups.com
> To unsubscribe from this group, send email to sage-support...@googlegroups.com

> For more options, visit this group at http://groups.google.com/group/sage-support

[mark florisson]

Great, thanks. I just wanted to download my worksheets which I need
for tomorrow :)

[Jason Grout]

On 10/27/11 12:56 PM, William Stein wrote:
>
> That's probably because there are over 500 simultaneous open worksheets
> right now. I've reset the servers but it will probably get heavily
> loaded again soon enough. Obviously we need to start planning for the
> next level in scalability...

I wonder if switching from the simple twistd server to, say, nginx
serving flask, would help things in the short term. I notice that the
sagenb.org process (given by the PID listed in the ./admin --status
command) hovers around 50-70% CPU utilization. That seems like a lot.

Thanks,

Jason

[kcrisman]

> That's probably because there are over 500 simultaneous open worksheets
> right now.   I've reset the servers but it will probably get heavily loaded
> again soon enough.   Obviously we need to start planning for the next level
> in scalability...

I guess that's a good problem to have...

I wonder if anyone noticed whether the other *.sagenb.org servers
experienced slowdowns then. In the past (not counting Jason's upgrade
attempt), slowdowns on sagenb.org did not seem to affect the others'
performance.

- kcrisman

[leif]

On 27 Okt., 19:12, Volker Braun <vbraun.n...@gmail.com> wrote:
> On Thursday, October 27, 2011 11:01:19 AM UTC-4, leif wrote:
> > Setting MAKE to "make -jN" when (re-)building Sage doesn't limit the
> > *total* number of build jobs to N;
>
> It does limit the total number of build jobs within the current make.

Provided the communication to the jobserver (through inherited file
descriptors, whose numbers are passed in MAKEFLAGS) isn't broken,
which currently apparently isn't always the case during the build.

> Of
> course it doesn't take into account make jobs that are run by different
> users, that run on different computers, that were invoked at a different
> time, etc.

:-) I thought the number referred to all simultaneous 'make' jobs in
the Milky Way.


-leif

[Vinay Wagh]

I am not sure whether my "problem" has anything to do with the topic of this thread...

I want to define a Power Series Ring over QQ. The following code works perfectly fine on my local computer (Ubuntu 11.10, Sage Version 4.7.1, Release Date: 2011-08-11), whereas if I give the same on sagenb.org it gives me an error.

I am defining the ring via:
R.<x,y,z> = PowerSeriesRing(QQ);

The error on sagenb:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "_sage_input_5.py", line 10, in <module>
    exec compile(u'open("___code___.py","w").write("# -*- coding: utf-8 -*-\\n" + _support_.preparse_worksheet_cell(base64.b64decode("Ui48eCx5LHo+ID0gUG93ZXJTZXJpZXNSaW5nKFFRKQpS"),globals())+"\\n"); execfile(os.path.abspath("___code___.py"))
  File "", line 1, in <module>
    
  File "/tmp/tmpHLLnj7/___code___.py", line 2, in <module>
    R = PowerSeriesRing(QQ, names=('x', 'y', 'z',)); (x, y, z,) = R._first_ngens(3)
  File "/sagenb/sage_install/sage-4.7/local/lib/python2.6/site-packages/sage/rings/power_series_ring.py", line 183, in PowerSeriesRing
    name = gens.normalize_names(1, name)
  File "parent_gens.pyx", line 213, in sage.structure.parent_gens.normalize_names (sage/structure/parent_gens.c:2311)
IndexError: the number of names must equal the number of generators

My observation is if I fedine a Power Series Ring in 1 variable, then sagenb accepts, but it seems it doesnt like multiple variables. In fact none of the example in the documentation work for that matter. Please let me know if there is some other issue involved...

Thanks and regards

-- VInay

[Jason Grout]

On 11/2/11 10:15 AM, Vinay Wagh wrote:
> I am not sure whether my "problem" has anything to do with the topic of
> this thread...
>
> I want to define a Power Series Ring over QQ. The following code works
> perfectly fine on my local computer (Ubuntu 11.10, Sage Version 4.7.1,
> Release Date: 2011-08-11), whereas if I give the same on sagenb.org

> <http://sagenb.org> it gives me an error.

sagenb.org is running Sage 4.7. My guess is that is the problem here.

Thanks,

Jason

[Vinay Wagh]

I also suspected the same, but the documentation doesnot say anywhere that the 4.7 does not support multi-var power series... (Or at least I couldnt find!)

Thanks and regards
-- Vinay
 

[William Stein]

Sage has multivariate power series now! Cool. I remember people
talking about implementing them periodically for 6 years, and never
doing it. I'm glad it's done now.

Note that http://test.sagenb.org/ has a newer version of sage.

-- William

>
> Thanks and regards
> -- Vinay
>
>

> --
> To post to this group, send email to sage-s...@googlegroups.com
> To unsubscribe from this group, send email to
> sage-support...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-support
> URL: http://www.sagemath.org
>

--

[Vinay Wagh]

Oh! I wasn't aware that multivariable power series is a "new" addition...

Thanks anyway...

-- VInay