Those cookies again...

[kcrisman]

I don't know that we've had as many people complaining about cookies
recently. Maybe this has been fixed.

Unfortunately, we just upgraded our campus server to 4.3 (the latest
VMWare image that we hadn't heard problems about) and apparently it is
hexed by the cookie issue (namely, that it wants you to delete Sage
cookies before it actually lets you log in).

This is particularly vexing because I can't get it to let me log in at
all. I can even get rid of all Sage cookies, restart the browser
(Safari), go to the notebook server, then delete the one cookie I get
there, and STILL it doesn't let me log in.

Sysadmin has found possible workaround of deleting history of the
browser. This is fine in a lab, but potentially very crippling for
those of us who rely on auto-completion of often-visited sites.
Sysadmin is also very unlikely to try 4.5.2 VMWare image after recent
reports of it not being so hot, though I think those may have been
exaggerated - and anyhow he has a lot to do with the start of classes.

Is it just conceivable that there is something we can configure with
the server and/or some patch we can apply from within the VMWare image
to fix this? Again, this is 4.3, my apologies. I am really hesitant
to use this in class when I can't even make it work on my own computer
properly. I think I now start to understand the arguments about
rather having one version that works rather than constant upgrades...

Thanks,
- kcrisman

[Dr. David Kirkby]

On 08/22/10 03:12 AM, kcrisman wrote:
> I don't know that we've had as many people complaining about cookies
> recently. Maybe this has been fixed.
>
> Unfortunately, we just upgraded our campus server to 4.3 (the latest
> VMWare image that we hadn't heard problems about) and apparently it is
> hexed by the cookie issue (namely, that it wants you to delete Sage
> cookies before it actually lets you log in).

> This is particularly vexing because I can't get it to let me log in at
> all. I can even get rid of all Sage cookies, restart the browser
> (Safari), go to the notebook server, then delete the one cookie I get
> there, and STILL it doesn't let me log in.

I've seen that problem too. I can't say for sure whether it has been fixed, but
I have some comments about this further down.

> Sysadmin has found possible workaround of deleting history of the
> browser. This is fine in a lab, but potentially very crippling for
> those of us who rely on auto-completion of often-visited sites.
> Sysadmin is also very unlikely to try 4.5.2 VMWare image after recent
> reports of it not being so hot, though I think those may have been
> exaggerated - and anyhow he has a lot to do with the start of classes.

I expect Sage upgrades will slip further down your system admin's priority list
if they are causing him problems.

> I am really hesitant
> to use this in class when I can't even make it work on my own computer
> properly.

I don't blame you.

> I think I now start to understand the arguments about
> rather having one version that works rather than constant upgrades...
>
> Thanks,
> - kcrisman
>

I think Peter Jeremy summed up the problem quite well when he said this on a
trac ticket

http://trac.sagemath.org/sage_trac/ticket/6456#comment:67

=================== From Peter Jeremy ===========================
I am very concerned at this "release it now, we'll make it work later" mentality.
================================================================

In my opinion, (and one I think that is shared by Peter too), Sage needs to
devote *far* more time to testing, and a lot less time to adding features, if
it's ever to become a viable alternative to the 4 M's.

At the most basic level, the notebook does not even produce valid HTML. The
login page has errors, which one discovers when one searches with the W3C
validator.

http://validator.w3.org/check?uri=http%3A%2F%2Fwww.sagenb.org%2F&charset=%28detect+automatically%29&doctype=Inline&group=0

I note two of the errors are:

======================================================================
# Error Line 91, Column 31: The for attribute of the label element must refer
to a form control.

<label for="email">Username</label>

✉
# Error Line 96, Column 34: The for attribute of the label element must refer to
a form control.

<label for="password">Password</label>
========================================================================

I wonder if those errors have anything to do with logging in?

The only possible way Sage might get less buggy, is for more people with similar
views to me, make them known to William. *Perhaps*, if he realises people like
you are reluctant to use Sage for classes because of the bug rates, he might do
something to address the quality control issues.

One of the release mangers for 4.5.3 has said the first release candidate for
4.5.3 will be available on Monday and he hopes to release 4.5.3 on Friday.
That's simply insufficient time for testing in my personal personal opinion.

I'd like to see regular "bug-fix-only" releases, where no new features are
added, but only code that addresses known bugs is incorporated.

Whilst Brooks claims in his book

http://en.wikipedia.org/wiki/The_Mythical_Man-Month#The_tendency_towards_irreducible_number_of_errors

that

===================================================================
in a suitably complex system there is a certain irreducible number of errors.
Any attempt to fix observed errors tends to result in the introduction of other
errors
===================================================================

I think Sage is a long way from that point.

Sage is certainly "suitably complex", but I don't think it's reached the point
where attempts to fix bugs will not reduce the total number of bugs. I think
with some effort, and a change of attitude, the number of bugs in Sage could be
reduced, but this would be at the expense of adding new features. It might even
lose some developers, who can't tolerate such a change of attitude.

Just my 2 pennies

Dave

[Mike Witt]

For whatever it's worth I'd like to say that I emphatically agree
that more attention to fixing bugs (presumably at the expense of
adding features) would make Sage *much* more viable from my point
of view. My point of view being as:

(1) Not a developer, but simply a user.

(2) Not a mathematician, but someone who is (late in the day :-)
slowly making my way through the undergrad math/physics
sequence.

(3) Someone who has tried unsuccessfully to get classmates and
instructors interested in Sage as an alternative to certain
other "M"s.

(4) Someone who, not being particularly brilliant with this
stuff, probably represents more of what a "typical" user
would look like if Sage ever attained widespread use.

Having said this, I can't help but wonder what possible
motivation there could be, among developers, to do something
like a bug fix release? It sounds like of boring ...

-Mike

[Jeff Post]

On Sunday 22 August 2010 08:07, Mike Witt wrote:
>
> Having said this, I can't help but wonder what possible
> motivation there could be, among developers, to do something
> like a bug fix release?
>

Professionalism?

Jeff

[Simon King]

Pride in their work?

Simon

[Dr. David Kirkby]

Mike,

Making bug-fix releases is an essential part of professional software
development. Jeff is right - it is the professional thing to do. Unfortunately,
most Sage developers do not have a background in software engineering, so do not
appreciate that.

As for motivation, these two links might give you some thoughts. There's some
very useful responses on the first link.

http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=17798

http://www.python.org/dev/peps/pep-0006/

Dave

[Mike Witt]

I may not have been thinking too clearly when I wrote that
last paragraph. I was, after all, trying to argue *in favor*
of bug fixes :-)

-Mike

[Dr. David Kirkby]

Unfortunately, I think a lot of Sage developers would be unhappy about bug-fix
releases. William is far from keen on the idea and I think there are others who
share his view.

Dave

[Nathann Cohen]

Hello !

> Making bug-fix releases is an essential part of professional software
> development. Jeff is right - it is the professional thing to do. Unfortunately,
> most Sage developers do not have a background in software engineering, so do not
> appreciate that.

I know I am just handing the stick, but I think some choices in
software development - like this one -- should not be considered as
lack of "knowledge/skill/talent" but just.... taste ?

-1 to bug-fix only releases. The "professionalism" may be improved,
but at the cost of fun. :-D

Nathann

(P.S.: You can now use the stick)

[Simon King]

Hi All!

Shouldn't this discussion better go to sage-devel?

On 22 Aug., 22:01, "Dr. David Kirkby" <david.kir...@onetel.net> wrote:
> ...
> http://www.python.org/dev/peps/pep-0006/

Quoting from this source:
"In general, only the N-1 release will be under active maintenance at
any time. That is, during Python 2.4's development, Python 2.3
gets
bugfix releases."

I see that simultaneously, there is work on Python 2.6.6 final (to
appear tomorrow) and Python 3.2 alpha2 (to appear September 5th).

My impression is that the Sage development process is quite far from
that way of thinking.

First of all, is there in Sage the concept of "maintenance" of a
previous version? Usually, if people hit a problem with, say,
sage-4.4, then they are told that they use a bronze age version and
should upgrade to the *latest* version sage-4.5.2. According to the
above quote, one would rather say "upgrade to bugfix release
sage-4.4.3".

So, in a way, only one Sage version is actively maintained at a time,
namely always the latest version.

If I remember correctly, there recently was a thread (sage-devel?)
about the role of milestones in the Sage trac. My impression is that
people virtually always choose the earliest possibility in the
"milestone" menue. Having bugfix releases would require a change of
attitude. People should tick 4.5.3 *only* if they have a bugfix.
Otherwise, they should tick 4.6 or 5.0. But then it may even be worth
to change the milestone menu. Perhaps:
"bugfix only" (without mentioning a number, as this will always go
into the earliest possible release),
"minor addition: 4.5.3" (There is no change in existing code and
little new code, so, it may be safe to consider early inclusion - but
care has to be taken, as it is more than a bugfix)
"minor addition: 4.5.4" (dito)
"major addition: 4.6" (There is much testing required, as there is
change in old code or there is much new code)
"critical change: 5.x" (in contrast to an addition, a change might
not be fully compatible with Sage-4.x and will thus not go in before
version 5.0)

At this point, a question on the current milestones: What is the
meaning of the milestones "sage-invalid/duplicate/wontfix" and "sage-
i18n"? The former looks more like a resolution (not a milestone) of a
ticket, and the latter is totally obscure to me.

Cheers,
Simon

[Simon King]

PS:

On 23 Aug., 12:55, Simon King <simon.k...@nuigalway.ie> wrote:
> ...

> My impression is that the Sage development process is quite far from
> that way of thinking.

... or perhaps it is not so much the way of thinking?
I would expect that Python has a lot more person power than Sage. How
many people do release management for Python?

[kcrisman]

Wow, I really didn't expect to open this discussion with that post.

> I expect Sage upgrades will slip further down your system admin's priority list
> if they are causing him problems.

Though he's actually quite Sage-friendly, and sounds like he'll do
it. The only issue was

> > I am really hesitant
> > to use this in class when I can't even make it work on my own computer
> > properly.
>
> I don't blame you.

Though again, I am personally quite likely to just find a way about
it, since I've invested so much in Sage.

> =================== From Peter Jeremy ===========================
> I am very concerned at this "release it now, we'll make it work later" mentality.
> ===============================================================

> The only possible way Sage might get less buggy, is for more people with similar
> views to me, make them known to William. *Perhaps*, if he realises people like
> you are reluctant to use Sage for classes because of the bug rates, he might do
> something to address the quality control issues.
>

Well, in general it seems to me that most Sage bugs come from things/
functionality that didn't exist before, and once they exist people
want to start using them. But unlike a commercial system, the only
realistic way we have to look for these bugs is for people to use the
system. I just don't see how else to do it; I don't think most Sage
developers necessarily use the latest bleeding-edge release for day-to-
day stuff. For instance, I have a modified R optional package
installed on one of my Sage installs, and since getting that to play
nice on every new Sage would be tedious and boring, I just keep
sage-4.4.4 on my laptop for research only. Similarly for classroom use
- one usually uses the same server all semester, so there isn't
opportunity to see *new* bugs.

> ===================================================================
> in a suitably complex system there is a certain irreducible number of errors.
> Any attempt to fix observed errors tends to result in the introduction of other
> errors
> ===================================================================

I think that while there are certainly some bugs that really don't
satisfy this, there are plenty of bugfixes that introduce new errors
in Sage - it is easily that complex. However, these usually result as
a result of uncovering some hitherto non-existent (or possibly non-
wrapped) functionality which turns out to have bugs, which will only
be uncovered by "real" users.

So actually, I find the release early and often mantra to be helpful,
because it helps make it better; finding edge cases is much less
likely to happen without a full-time team of dedicated staff
otherwise. I think that even some randomized test suite is not so
likely to find things unless it starts just entering random valid Sage
commands, even ones that don't make sense to a "real" user.

And I don't think there is a lack of attention to fixing bugs; every
release is a bug-fix release. It is also a new-functionality release
- which yes, introduces bugs, but not ones that would have even been
visible before.

- kcrisman

[Alex Ghitza]

On Sat, 21 Aug 2010 19:12:49 -0700 (PDT), kcrisman <kcri...@gmail.com> wrote:
> Sysadmin has found possible workaround of deleting history of the
> browser. This is fine in a lab, but potentially very crippling for
> those of us who rely on auto-completion of often-visited sites.
> Sysadmin is also very unlikely to try 4.5.2 VMWare image after recent
> reports of it not being so hot, though I think those may have been
> exaggerated - and anyhow he has a lot to do with the start of classes.

I'd say the only way you will know whether 4.5.2 works for you is to try
it out. The vmware image gets downloaded *a lot* (200 times in the
first 5 days), and as far as I know there were only two problems
reported with it:
* unable to access the notebook from the host machine on Windows
(this works for me on Mac OS X, and I don't have a Windows machine to
test on -- someone suggested a workaround on the list but we haven't
heard the original poster confirm that it fixed his problem)
* request for having R installed with image support; I will look into
this for 4.5.3, but if this is likely to be important to you, you can
just install the ubuntu libraries in your copy of the virtual machine

So, I'm not saying that the 4.5.2 vmware image is perfect, but I'm not
convinced it's any worse than 4.3, and once again, you can find out
whether it works for you by trying it out. Try it out on your computer
first and see if the cookies issue is fixed, and then see if your system
administrator is willing to give it a shot.

Best,
Alex

--
Alex Ghitza -- http://aghitza.org/
Lecturer in Mathematics -- The University of Melbourne -- Australia

[Dr. David Kirkby]

On 08/23/10 04:20 PM, kcrisman wrote:

> Well, in general it seems to me that most Sage bugs come from things/
> functionality that didn't exist before, and once they exist people
> want to start using them.

Well, there are an alful lot of open-bugs in trac. Some have been open a very
long time. They are assigned to person X (for example I get the Solaris ones),
but person X does not work on them, but spends time writing new code.

> But unlike a commercial system, the only
> realistic way we have to look for these bugs is for people to use the
> system. I just don't see how else to do it;

There are several packages in Sage which have test suites that we could run from
spkg-check. But people have not added the spkg-check file to Sage, so we can not
run the tests.

http://trac.sagemath.org/sage_trac/ticket/9767 # cliquer
http://trac.sagemath.org/sage_trac/ticket/9613 # linbox
http://trac.sagemath.org/sage_trac/ticket/9311 # ratpoints

and *many* others. Of the 100 or so standard packages in sage, only around 20
have the ability to run any self-tests of the packages when they are built. In
some cases, those test suites are *far* more comprehensive than the ones in
Sage. (I believe Pari is an exception, where the Sage doctests are more
comprehensive than the Pari tests).

If you look at the first and second columns of this ticket:

http://trac.sagemath.org/sage_trac/ticket/9281

you will see those where there is an spkg-check file, and there where there is
not one. Not every package in sage has the ability to do any self checks during
the build process, so we will never be able to get that list complete, but there
are a lot missing.

Dave

[kcrisman]

I've put this on sage-devel where it belongs.

On Aug 24, 5:14 am, "Dr. David Kirkby" <david.kir...@onetel.net>
wrote:

> On 08/23/10 04:20 PM, kcrisman wrote:
>
> > Well, in general it seems to me that most Sage bugs come from things/
> > functionality that didn't exist before, and once they exist people
> > want to start using them.
>
> Well, there are an alful lot of open-bugs in trac. Some have been open a very
> long time. They are assigned to person X (for example I get the Solaris ones),
> but person X does not work on them, but spends time writing new code.

"Assigned to" just means that there is some system default for where a
given category of ticket goes. It does not mean "homework for or you
can't work on Sage any more". But I agree that this is very
confusing. Perhaps someone who has permission to make such changes on
Trac could make that more obvious.

> > But unlike a commercial system, the only
> > realistic way we have to look for these bugs is for people to use the
> > system.  I just don't see how else to do it;
>
> There are several packages in Sage which have test suites that we could run from
> spkg-check. But people have not added the spkg-check file to Sage, so we can not
> run the tests.
>
> http://trac.sagemath.org/sage_trac/ticket/9767# cliquerhttp://trac.sagemath.org/sage_trac/ticket/9613# linboxhttp://trac.sagemath.org/sage_trac/ticket/9311# ratpoints

That would be great. I think this is somewhat orthogonal to fixing
bugs in the Sage library, though; obviously we want to make sure
upstream is good, but again this is a different thing. I guess one
could add this as a reqt. to any upgrades of spkgs.

> and *many* others. Of the 100 or so standard packages in sage, only around 20
> have the ability to run any self-tests of the packages when they are built. In
> some cases, those test suites are *far* more comprehensive than the ones in
> Sage. (I believe Pari is an exception, where the Sage doctests are more
> comprehensive than the Pari tests).
>
> If you look at the first and second columns of this ticket:
>
> http://trac.sagemath.org/sage_trac/ticket/9281
>
> you will see those where there is an spkg-check file, and there where there is
> not one. Not every package in sage has the ability to do any self checks during
> the build process, so we will never be able to get that list complete, but there
> are a lot missing.

That's a great ticket.

Anyway, I think (as you have correctly noted before) we have a bit of
a culture clash between software engineering and mathematics.
However, as far as I can tell, the only way to solve it is to vastly
increase the user base until enough of them become developers that the
load of these things does not fall on just a few people, nearly *all*
of whom (including you, and you have done enormous high-quality work)
are doing it on a volunteer basis.

So we will add what makes Sage better for us. This is definitely not
ideal project management in the sort of setting you are talking about,
but the alternative is Sage staying completely stagnant, I think.
It's a matter of motivating the troops in terms of things like
documentation, testing, etc. And I, for one, would rather have Sage
add lots of useful content for my courses than have it pass every spkg
test - not that those are bad, far from it!

It's a long evolution from "William's private replacement for Magma
written on top of Python" to "highest-quality possible replacement for
M*", and we aren't there yet. But I think if you look at how things
have changed over the last three or four years, the release process
(for instance) that Mitesh et al. are currently doing is vastly
different from the one long ago; five years from now it could be
nearly up to your standards, I hope - because you are right, they are
the right ones to have.

Just have patience with those of us who aren't from a software
background - and trust that we are trying hard to internalize your
lessons, but that we have more immediate needs to fill as well for our
next course or paper. I think that just as Minh's messages about
documentation are slowly taking hold in the whole ecosystem, so are
yours about software engineering.

- kcrisman

[Jason Grout]

On 8/21/10 9:12 PM, kcrisman wrote:
> I don't know that we've had as many people complaining about cookies
> recently. Maybe this has been fixed.
>
> Unfortunately, we just upgraded our campus server to 4.3 (the latest
> VMWare image that we hadn't heard problems about) and apparently it is
> hexed by the cookie issue (namely, that it wants you to delete Sage
> cookies before it actually lets you log in).

I just had this problem on Safari, and the only way I fixed it was to go
to safari security preferences and change the cookie handling from
accepting cookies "only from sites I visited" to "always".

Weird, and it seems like a bug that should be tracked down.

Thanks,

Jason

[kcrisman]

+1

I'm glad someone else has seen this recently! It's definitely only in
Safari.

- kcrisman

[Tim Joseph Dumol]

Hi,

I've created a ticket addressing this issue at (#9822
http://trac.sagemath.org/sage_trac/ticket/9822) and a patch in an
attempt to fix it. I'm not sure whether it works though, as I don't
have access to Safari. Can either of you try it out?

> --
> To post to this group, send email to sage-s...@googlegroups.com
> To unsubscribe from this group, send email to sage-support...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/sage-support
> URL: http://www.sagemath.org
>

--
Tim Joseph Dumol <tim (at) timdumol (dot) com>
http://timdumol.com

[Jason Grout]

One of my students had it in internet explorer today.

I wonder if part of the problem had to do with them creating accounts
directly before trying to log in.

Jason

[Dr. David Kirkby]

On 08/28/10 05:49 AM, Jason Grout wrote:
> On 8/27/10 8:03 AM, kcrisman wrote:
>>
>>
>> On Aug 27, 7:37 am, Jason Grout<jason-s...@creativetrax.com> wrote:
>>> On 8/21/10 9:12 PM, kcrisman wrote:
>>>
>>>> I don't know that we've had as many people complaining about cookies
>>>> recently. Maybe this has been fixed.
>>>
>>>> Unfortunately, we just upgraded our campus server to 4.3 (the latest
>>>> VMWare image that we hadn't heard problems about) and apparently it is
>>>> hexed by the cookie issue (namely, that it wants you to delete Sage
>>>> cookies before it actually lets you log in).
>>>
>>> I just had this problem on Safari, and the only way I fixed it was to go
>>> to safari security preferences and change the cookie handling from
>>> accepting cookies "only from sites I visited" to "always".
>>>
>>> Weird, and it seems like a bug that should be tracked down.
>>
>> +1
>>
>> I'm glad someone else has seen this recently! It's definitely only in
>> Safari.

I've seen this problem, though not necessarily on the latest version using
Firefox. So the problem is not unique. I'm pretty sure I've seen it on

http://t2nb.math.washington.edu:8000/

since I updated that to Sage 4.5.1, but I can't be 100% sure on that.

> One of my students had it in internet explorer today.
>

To my way of thinking, a developer familiar with the notebook should look first
determine why the login page is not producing valid HTML. The W3C validator
clearly shows this, and two of the three errors are directly on the fields about
username and password.

If the code sent to the browser is not valid HTML, the behavior of the browser
is undefined.

One might make changes to the cookie code which fixes this for some browsers or
even all browsers now, but it's anyones guess what will happen long-term if the
sections that requires one to log in are sending invalid data to the browser. Is
it any big surprise the cookies are not working correctly in this case?

> I wonder if part of the problem had to do with them creating accounts
> directly before trying to log in.

It's really anyone's guess. But clearly people should be able to create accounts
before or after they try to log in.

> Jason

Dave

[Tim Joseph Dumol]

The invalid HTML is orthogonal to this issue and is merely a typo. The
<label> tag merely affects form display semantics and does not have
anything to do regarding cookies.

Frankly, even http://google.com has invalid HTML:
http://validator.w3.org/check?uri=www.google.com&charset=(detect+automatically)&doctype=Inline&group=0

A few more:

* CNN.com http://validator.w3.org/check?uri=http://www.cnn.com&charset=(detect+automatically)&doctype=Inline&group=0&user-agent=W3C_Validator/1.1

* NSF: http://validator.w3.org/check?uri=http://www.nsf.gov/&charset=(detect+automatically)&doctype=Inline&group=0&user-agent=W3C_Validator/1.1

* Harvard: http://validator.w3.org/check?uri=http://harvard.edu/&charset=(detect+automatically)&doctype=Inline&group=0&user-agent=W3C_Validator/1.1

And so on.

I do not say that writing invalid HTML should be accepted as the norm,
but it is pointless to blame the cookie issue on it.

>
>> I wonder if part of the problem had to do with them creating accounts
>> directly before trying to log in.
>
> It's really anyone's guess. But clearly people should be able to create
> accounts before or after they try to log in.

And people are indeed able to do so.

>
>> Jason
>
> Dave

[Dr. David Kirkby]

On 08/28/10 12:27 PM, Tim Joseph Dumol wrote:

> The invalid HTML is orthogonal to this issue and is merely a typo. The
> <label> tag merely affects form display semantics and does not have
> anything to do regarding cookies.
>
> Frankly, even http://google.com has invalid HTML:
> http://validator.w3.org/check?uri=www.google.com&charset=(detect+automatically)&doctype=Inline&group=0

Yes I know. But Google is not presenting a problem.

> I do not say that writing invalid HTML should be accepted as the norm,
> but it is pointless to blame the cookie issue on it.

I was not blaming the HTML on it. I'm merely proposing that it might be the
cause and it would certainly be the first thing I would look at fixing. It would
rule out that possibility.

Dave

[Tim Joseph Dumol]

A fix is presented at http://trac.sagemath.org/sage_trac/ticket/9832.

[Jason Grout]

On 8/28/10 6:27 AM, Tim Joseph Dumol wrote:
> On Sat, Aug 28, 2010 at 5:48 PM, Dr. David Kirkby

>> It's really anyone's guess. But clearly people should be able to create
>> accounts before or after they try to log in.
>
> And people are indeed able to do so.
>

Yes, I should point out that an overwhelming majority of the people had
no problem yesterday logging in.

Jason