Fwd: [sage-support] Mysterious behaviour of q_eigenform... Bug?

[John Cremona]

---------- Forwarded message ----------
From: Misja <m.ste...@gmail.com>
Date: 28 April 2016 at 15:09
Subject: [sage-support] Mysterious behaviour of q_eigenform... Bug?
To: sage-support <sage-s...@googlegroups.com>


When understand the specific reason why my code is not working
properly, I managed to pin it down to the following mysterious
behaviour of q_eigenform.

First run the following code in sage.

G=DirichletGroup(80);
chi=G[22];
D=ModularSymbols(chi,2,-1).cuspidal_subspace().new_subspace().decomposition();
for f in D:
elt=f.q_eigenform(10,'alpha')[3];
N=elt.parent().absolute_field('a');
fact=N.factor(2);
for P,e in fact:
res_field=N.residue_field(P);
print res_field(elt);


It will print

0
0
0
0

which, I think, is the 'right' answer.


Now close your sage session entirely. Open a new session and then run
the following *silly* code:

G=DirichletGroup(80);
for chi in G:
D=ModularSymbols(chi,2,-1).cuspidal_subspace().new_subspace().decomposition();
for f in D:
elt=f.q_eigenform(10,'alpha')[3];
if not elt.parent()==QQ:
K=elt.parent().absolute_field('b');
if chi==G[22]:
fact=K.factor(2);
for P,e in fact:
res_field=K.residue_field(P);
print res_field(elt);


It will print:

0
0
1
0

As far as I understand the theory, this cannot happen. If you let sage
print the alpha^3 coefficient of you see that in both cases it picks a
different q_eigenform in f, the Galois conjugacy class of newforms.
Although this can be a bit annoying, in theory it is fine. But I am
pretty sure that when your reduce this coefficient modulo some prime
P, any two elements of the same Galois conjugacy class can differ at
most by some automorphism of the residue field (and obviously 1 and 0
do not satisfy this criterion).


To make matters worse: if you run a single sage session and you run
the 'good' code first and the 'bad' code second, then suddenly the
'bad' code is fixed and printing only 0s. If you run the 'bad' code
first and the 'good' code second, then they are both 'bad' and the
'good' code suddenly prints 0,0,1,0 as well.

By trying I found out that this is because somehow q_eigenform picks
the same q_eigenform as whichever code that ran first and somehow
these choices are not compatible! I don't know the inner workings of
q_eigenform, but this behaviour seems strange to me.

Can anyone explain what is going on here? Is it a bug?

Thanks!

--
You received this message because you are subscribed to the Google
Groups "sage-support" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sage-support...@googlegroups.com.
To post to this group, send email to sage-s...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-support.
For more options, visit https://groups.google.com/d/optout.

[Kevin Buzzard]

Here's a shorter version of the code which I think already exhibits a problem. I should say that I'm currently only just learning sage so I might be victim to a gotcha. But here we go:

G=DirichletGroup(80);

for chi in G: 

    D=ModularSymbols(chi,2,-1).cuspidal_subspace().new_subspace().decomposition()

    for f in D: 

        elt=f.q_eigenform(10,'alpha')[3]; 

        print(elt.is_integral())

If I cut and paste this into a sage session (either 6.4.1 or 7.0 on Ubuntu 14.04) I get mostly true's but a few false's. My understanding is that I'm computing cuspidal new eigenforms here and checking to see that the coefficient of q^3 is an algebraic integer in each case. It sometimes isn't. What are we doing wrong?

Kevin

[David Loeffler]

Hi Kevin and Misja,

This has something to do with the fraught issue of *variable names*. Kevin's code prints a mixture of True's and False's; but, on the other hand, if you run the code

sage: G=DirichletGroup(80);
sage: for chi in G:

        D=ModularSymbols(chi,2,-1).cuspidal_subspace().new_subspace().decomposition()
        for f in D:

                elt=f.q_eigenform(10,'alpha' + str(randint(1,1000)))[3];
                print(elt.is_integral())

then you get all True's (at least, with rather high probability!). (Or, more civilised: replace str(randint(1, 1000)) with str(f.factor_number()), which is just an arbitrary ordering on the decomposition factors of a given Hecke module.)

There's a natural response that this is absurd -- how can the name given to a variable affect its behaviour, in any sane environment? if I remember correctly, the reason for this is that Sage's coercion system has to have a way of knowing when two "number field" Sage objects with the same defining polynomial are supposed to represent the same field (so there is a canonical map between them) and when they're supposed to represent different, non-canonically-isomorphic fields. Sage's solution is to assume that if two number field objects have the same defining polynomial *and the same generator name*, then they're identical. This leads to weird behaviour when you create a bunch of number fields one after the other and call the generators "alpha" in each case.

So now you know why the default behaviour of the "Newforms" constructor is to append an integer to the variable names -- if you do "Newforms(DirichletGroup(80), names='alpha')" you'll get coefficient fields with generators alpha0, alpha1, etc (not all alpha). I'll leave it to other, more expert users to comment on why the collision of names causes the exact bug above, but hopefully this should give you the means to work around it.

David

--
You received this message because you are subscribed to the Google Groups "sage-nt" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-nt+u...@googlegroups.com.
To post to this group, send email to sag...@googlegroups.com.

Visit this group at https://groups.google.com/group/sage-nt.

[Misja Steinmetz]

Wow, David, this is a great response! Indeed, it seems quite absurd to me that picking the same name for a generator can produce such strange results, but at least the fix isn't too hard. It seems to have fixed my problems for now :-).

Thanks!

Misja

--
You received this message because you are subscribed to a topic in the Google Groups "sage-nt" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-nt/NM7bbCgefdo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sage-nt+u...@googlegroups.com.

[David Loeffler]

On 4 May 2016 at 10:25, Misja Steinmetz <m.ste...@gmail.com> wrote:

It seems to have fixed my problems for now :-)

I'm glad to hear it! As a general rule, if you hit a Sage problem that seems to depend mysteriously on what's previously been executed in the session, then it's likely to be something to do with the coercion framework.

David

[Peter Bruin]

Hi all,

David is certainly correct that there is a problem with number fields
and coercion. The following example runs into the same (or at least a
similar) bug:

k.<zeta10> = CyclotomicField(10)
R.<x> = PolynomialRing(k)
f0 = x^2 + (zeta10 + 1)*x + zeta10^2 - 2*zeta10 + 1
K.<alpha> = k.extension(f0)
f1 = f0.base_extend(K)
print f0
assert(f0(alpha) == 0)
print f1
assert(f1(alpha) == 0)
# del K, alpha, f0, f1; import gc; gc.collect()
g0 = x^2 + (zeta10^3 + 1)*x - 2*zeta10^3 - zeta10 + 1
K.<alpha> = k.extension(g0)
g1 = g0.base_extend(K)
print g0
assert(g0(alpha) == 0)
print g1
assert(g1(alpha) == 0)

In at least Sage 7.2.beta5 and 7.2.beta6, the last assertion fails.
Uncommenting the line starting with "del ..." makes the problem go away
(in a fresh Sage session).

Peter

[Kevin Buzzard]

In my student's original code, he didn't even _need_ access to the
variable name that he gave when running q_eigenform; he just needed to
compute the valuation of a specific element of the number field he was
creating, at a prime ideal he was about to create by factoring a
rational prime number. If creating variable names for number fields is
so fraught with difficulty, is there any way of avoiding this
completely? My memory was that in magma you could just build a number
field from a polynomial and if you didn't give it a variable name
which was to correspond to a root of the polynomial then magma would
just name it something like $.1 and not worry about it any more. This
is the situation I think we're in here -- but in sage
f.q_eigenform(10,'') and f.q_eigenform(10) both throw ValueErrors. I
am now terrified to write any code containing loops and q_eigenform!
Even running the same code twice in a session sounds to me to be
dangerous.

Kevin

> --
> You received this message because you are subscribed to a topic in the Google Groups "sage-nt" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-nt/NM7bbCgefdo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-nt+u...@googlegroups.com.
> To post to this group, send an email to sag...@googlegroups.com.

[Peter Bruin]

Hello,

Kevin Buzzard <kevin.m...@gmail.com> wrote:

> In my student's original code, he didn't even _need_ access to the
> variable name that he gave when running q_eigenform; he just needed to
> compute the valuation of a specific element of the number field he was
> creating, at a prime ideal he was about to create by factoring a
> rational prime number. If creating variable names for number fields is
> so fraught with difficulty, is there any way of avoiding this
> completely? My memory was that in magma you could just build a number
> field from a polynomial and if you didn't give it a variable name
> which was to correspond to a root of the polynomial then magma would
> just name it something like $.1 and not worry about it any more.

Creating number fields without a variable name is not possible in Sage.
There is nothing fundamentally against creating two number fields with
the same variable name; this _should_ not cause any problems. However,
distinguishing two relative number fields with the same variable name is
apparently not done correctly when creating polynomial rings:

sage: k.<zeta10> = CyclotomicField(10)
sage: R.<x> = PolynomialRing(k)
sage: K.<alpha> = k.extension(x^2 + (zeta10 + 1)*x + zeta10^2 - 2*zeta10 + 1)
sage: L.<alpha> = k.extension(x^2 + (zeta10^3 + 1)*x - 2*zeta10^3 - zeta10 + 1)
sage: K is L
False
sage: K.absolute_polynomial() == L.absolute_polynomial()
True
sage: A = PolynomialRing(K, 'x')
sage: B = PolynomialRing(L, 'x')
sage: A is B
True

It looks like PolynomialRing constructor only compares the _absolute_
defining polynomials and the variable names of K and L, but this is not
enough since K and L have different _relative_ defining polynomials.
I suspect that this is the underlying problem in both the q_eigenform
bug and an older bug report about roots of polynomials over relative
number fields <http://trac.sagemath.org/ticket/18942>.

> This is the situation I think we're in here -- but in sage
> f.q_eigenform(10,'') and f.q_eigenform(10) both throw ValueErrors. I
> am now terrified to write any code containing loops and q_eigenform!
> Even running the same code twice in a session sounds to me to be
> dangerous.

Indeed, if it is indeed the case that this bug occurs whenever you have
two different relative number fields constructed in different ways but
with the same absolute defining polynomial and the same variable name,
then this situation is likely to come up when you run through a family
of Hecke eigenvalue fields.

Peter

[Kevin Buzzard]

> I suspect that this is the underlying problem in both the q_eigenform
> bug and an older bug report about roots of polynomials over relative
> number fields <http://trac.sagemath.org/ticket/18942>.

That bug has been open 10 months, and silently breaks code done in
loops where different relative number fields are initialised with the
same generator variable? Yikes.

I see that in the original bug report the author was as completely
bewildered as Misja was.

Kevin

[Kiran Kedlaya]

I have a patch up on ticket 18942 which addresses this issue. The problem seems to have been that the hash of a number field, which is used to do the cache lookup, was defined using the absolute defining polynomial rather than the relative one. (It remains to verify that fixing this didn't break anything else...)

[Misja Steinmetz]

I am new to Sage (and http://trac.sagemath.org/), so this may be a silly question. When/in which version of sage can I expect ticket 18942 to be resolved? What confuses me is that the ticket is closed and the milestone is set to sage-7.2, but on my laptop (with sage 7.2 and ubuntu 14.04) the issue seems to still be there.

[Dima Pasechnik]

On Tuesday, May 17, 2016 at 12:53:30 PM UTC+1, Misja Steinmetz wrote:

I am new to Sage (and http://trac.sagemath.org/), so this may be a silly question. When/in which version of sage can I expect ticket 18942 to be resolved? What confuses me is that the ticket is closed and the milestone is set to sage-7.2, but on my laptop (with sage 7.2 and ubuntu 14.04) the issue seems to still be there.

they forgot to bump up the milestone to 7.3 (fixed now). So, no, it's not in 7.2, but you could get this update for your system, if you need it...