git from trac.sagemath.org out of order ?

[Emmanuel Charpentier]

Dear list,

The trac repository seems no longer able to answer my "fetch" requests.

My status :

git status
Sur la branche develop
Votre branche est à jour avec 'origin/develop'.
Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)

    src/sage_setup/autogen/pari/timestamp

aucune modification ajoutée à la validation mais des fichiers non suivis sont présents (utilisez "git add" pour les suivre)

My config :

cat /usr/local/sage-6.8/.git/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
[remote "origin"]
    url = http://trac.sagemath.org/sage.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master
[branch "develop"]
    remote = origin
    merge = refs/heads/develop
[remote "trac"]
    url = git://trac.sagemath.org/sage.git
    fetch = +refs/heads/*:refs/remotes/trac/*
    pushurl = g...@trac.sagemath.org:sage.git
[trac]
    username = XXXXXXX
    password = XXXXXXXXXXXXXXXXXXX
[branch "t/18229/R_upgrade_with_new_patch"]
    remote = trac
    merge = refs/heads/u/leif/R_upgrade_with_new_patch

Symptom : "git fetch" simply hangs. Never did that on me on this machine, as far as I can remember...

Any idea ?

[Emmanuel Charpentier]

Update : I just got :

error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

As far as I know, HTTP error code 500 is ... internal server error. Ouch !

This smells...

HTH,
--
Emmanuel Charpentier

[Volker Braun]

Transient error or port 22 blocked, I guess. Works for me:

$ git trac fetch 18229

remote branch: f451e9a5227d73fc5ab8719857a4d17aae129eb8

From git://trac.sagemath.org/sage

 * branch            f451e9a5227d73fc5ab8719857a4d17aae129eb8 -> FETCH_HEAD

[Emmanuel Charpentier]

It still (2015-06-12,06:50 CET) fails for me. Not *that* transient, it seems...

And, yes, I can ssh from this machine to The Rest of The World...

HTH,
--
Emmanuel Charpentier

[Volker Braun]

You are more likely to get help if you post the failing command with output.

[Emmanuel Charpentier]

Same problem on a different pachine with a different network connection :

git fetch
fatal: unable to connect to trac.sagemath.org:
trac.sagemath.org[0: 128.208.178.249]: errno=Connexion terminée par expiration du délai d'attente

HTH,
--
Emmanuel Charpentier

[Volker Braun]

git fetch -v

[Emmanuel Charpentier]

A couple more datapoints :

Machine 1, setup 1 : home machine connected via a (slow but usually reliable) ADSL land line : fails since yesterday, last checked this mornong (about 08:00 CET).
Machine 2, setup 2 : work machine connected via the "corporate" network of my employer. Fails since this mornong (about 09:00 CET), just failed again (about 10:00 CET)
Machine 3 : notebook, connected via cellphone (yes...) : git fetch just succeeded (about 09:50 CET).

I can ssh a fourth machine from these three machines.

HTH,
--
Emmanuel Charpentier

[Emmanuel Charpentier]

On machine 2, setup 2, same old same...

git fetch -v
fatal: unable to connect to trac.sagemath.org:

[Nathann Cohen]

Desperately trying to push something, the function hangs.  I also cannot get to load the ticket I want to update [1] :-P

Nathann

[1] http://trac.sagemath.org/ticket/18681

[Volker Braun]

I've seen it now, too. The connection is pretty flaky even when it works, e.g. massive packet loss (look at the sequence numbers):

$ ping trac.sagemath.org

PING trac.sagemath.org (128.208.178.249): 56 data bytes

64 bytes from 128.208.178.249: seq=6 ttl=46 time=512.834 ms

64 bytes from 128.208.178.249: seq=7 ttl=46 time=514.608 ms

64 bytes from 128.208.178.249: seq=9 ttl=46 time=475.129 ms

64 bytes from 128.208.178.249: seq=15 ttl=46 time=512.305 ms

64 bytes from 128.208.178.249: seq=27 ttl=46 time=478.100 ms

64 bytes from 128.208.178.249: seq=29 ttl=46 time=476.673 ms

Tracing it, the problem seems to originate at the Telia -> Level3 boundary. So there is nothing we can do, just wait for the big guys to fix it.

[Emmanuel Charpentier]

Thanks !

Should we sacrifice a goat ? (Or a manager...)

--
Emmanuel Charpentier

[Nathann Cohen]

> Should we sacrifice a goat ? (Or a manager...)

Things seems normal again here. If you don't think that you will eat
that whole goat by yourself, you can count me in.

Nathann

[Volker Braun]

Apparently Telekom Malaysia (https://twitter.com/TMCorp/status/609167065300271104) broke the internet for a short while until the Tier 1 providers cut them off...

[Emmanuel Charpentier]

Still fails for me.

On second thought, a live goat as some real uses (goat cheese ... mmmm ...).

--
Emmanuel Charpentier

[Nathann Cohen]

> Apparently Telekom Malaysia
> (https://twitter.com/TMCorp/status/609167065300271104) broke the internet
> for a short while until the Tier 1 providers cut them off...

Hey man, can you share the wisdom!?

1) How did you figure that out?
2) How do you learn about internet's organization (Tier 1, Level 3, Telia?...)

Nathann

[Volker Braun]

1) Go to google, type in "level3", click on search tools -> past hour

2) https://en.wikipedia.org/wiki/Tier_1_network

[Volker Braun]

And of course

$ traceroute trac.sagemath.org

traceroute to trac.sagemath.org (128.208.178.249), 30 hops max, 38 byte packets

[...]

 6  ip5886ca49.dynamic.kabel-deutschland.de (88.134.202.73)  8.599 ms  8.090 ms  8.055 ms

 7  bei-b1-link.telia.net (80.239.132.237)  16.656 ms  8.077 ms  7.952 ms

 8  ffm-bb2-link.telia.net (80.91.254.228)  21.138 ms  ffm-bb2-link.telia.net (62.115.112.155)  21.678 ms  ffm-bb2-link.telia.net (62.115.112.157)  23.770 ms

 9  ffm-b10-link.telia.net (62.115.137.237)  22.439 ms  ffm-b10-link.telia.net (62.115.137.171)  23.021 ms  ffm-b10-link.telia.net (62.115.137.241)  24.582 ms

10  *  *  *

11  *  *  *

12  *  ae-22-52.car2.Seattle1.Level3.net (4.69.147.164)  437.995 ms  *

13  *  *  *

14  *  *  *

15  209.124.188.133 (209.124.188.133)  515.040 ms  *  *

16  *  *  *

17  *  *  *

18  *  *  smc-dc1-249.math.washington.edu (128.208.178.249)  392.572 ms

[Nathann Cohen]

Yo !

> 1) Go to google, type in "level3", click on search tools -> past hour

Come on man, how do you 'guess' level3?

> 2) https://en.wikipedia.org/wiki/Tier_1_network

I was doing that, but again "how do you guess that name"? Is there
some book explaining all this?

Nathann

[Nathann Cohen]

> And of course
>
> $ traceroute trac.sagemath.org

Yeah. Well at that time mine involved different operators. Cogentco or
something.

Nathann

[Volker Braun]

On Friday, June 12, 2015 at 2:25:11 PM UTC+2, Nathann Cohen wrote:

> 1) Go to google, type in "level3", click on search tools -> past hour
Come on man, how do you 'guess' level3?

Was pretty obvious from the traceroute output...

[Emmanuel Charpentier]

Things seem still flaky now (22:42 CET) : git fetch still hangs. But git fetch -v  gives some signs of life before hanging :

git fetch -v
POST git-upload-pack (gzip 12481 to 6253 bytes)
POST git-upload-pack (gzip 13031 to 6539 bytes)
POST git-upload-pack (gzip 14581 to 7322 bytes)
POST git-upload-pack (gzip 17231 to 8652 bytes)
POST git-upload-pack (gzip 22731 to 11443 bytes)
POST git-upload-pack (gzip 33681 to 17013 bytes)
POST git-upload-pack (gzip 56631 to 28499 bytes)
POST git-upload-pack (gzip 103581 to 51743 bytes)
POST git-upload-pack (gzip 148331 to 73896 bytes)
POST git-upload-pack (gzip 199231 to 99039 bytes)
POST git-upload-pack (gzip 228286 to 113433 bytes)

error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

Is there anything I can do at my end to fix this ? Or pray for Malaysian to recover reason (an possibly sacrificing another manager) ?

--
Emmanuel Charpentier

[Volker Braun]

The server communication works, this is unrelated

[Volker Braun]

I tried cloning just the repo and then fetching all of trac and it worked. My git version seems to be more recent though, the output is all different. Have you tried increasing the post buffer? 

git config --global http.postBuffer 524288000

git fetch -v

[Emmanuel Charpentier]

[ I didn't know that option. Interesting... ] Anyway, same difference :

charpent@asus16-ec:/usr/local/sage-6.8$ git config  http.postBuffer 524288000charpent@asus16-ec:/usr/local/sage-6.8$ git fetch -v
POST git-upload-pack (gzip 12681 to 6392 bytes)
POST git-upload-pack (gzip 13231 to 6679 bytes)
POST git-upload-pack (gzip 14781 to 7465 bytes)
POST git-upload-pack (gzip 17431 to 8795 bytes)
POST git-upload-pack (gzip 22931 to 11585 bytes)
POST git-upload-pack (gzip 33881 to 17159 bytes)
POST git-upload-pack (gzip 56831 to 28643 bytes)
POST git-upload-pack (gzip 103781 to 51882 bytes)
POST git-upload-pack (gzip 148531 to 74038 bytes)
POST git-upload-pack (gzip 199431 to 99200 bytes)
POST git-upload-pack (gzip 228586 to 113622 bytes)

error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

I am at loss : It's not on my side (I even went to te extremity of rebooting my machine).

[Volker Braun]

Which version of git are you using? Have you tried the git version that is included in Sage? 

sage -f git

sage -git fetch -v

[Emmanuel Charpentier]

[ got a famillial NMI. Sorry for the delay... ]

Dear Volker,

First of all, thank you for your attention to my petty troubles !

Le samedi 13 juin 2015 12:05:57 UTC+2, Volker Braun a écrit :

Which version of git are you using? Have you tried the git version that is included in Sage? 

From another directory :

charpent@asus16-ec:~/Perso/Free$ git --version
git version 2.1.4

After installing Sage's git, same difference (again...) :

charpent@asus16-ec:/usr/local/sage-6.8$ ./sage -git --version
git version 2.3.0
charpent@asus16-ec:/usr/local/sage-6.8$ ./sage -git fetch -v
POST git-upload-pack (gzip 12731 to 6418 bytes)
POST git-upload-pack (gzip 13281 to 6704 bytes)
POST git-upload-pack (gzip 14831 to 7489 bytes)
POST git-upload-pack (gzip 17481 to 8821 bytes)
POST git-upload-pack (gzip 22981 to 11608 bytes)
POST git-upload-pack (gzip 33931 to 17183 bytes)
POST git-upload-pack (gzip 56881 to 28664 bytes)
POST git-upload-pack (gzip 103831 to 51903 bytes)
POST git-upload-pack (gzip 148581 to 74060 bytes)
POST git-upload-pack (gzip 199481 to 99218 bytes)
POST git-upload-pack (gzip 228636 to 113642 bytes)

error: RPC failed; result=22, HTTP code = 500
fatal: The remote end hung up unexpectedly

HTH,

--
Emmanuel Charpentier at the end of his rope...

[Volker Braun]

Can you PM me your IP number, then I'll grep through the log...

[Emmanuel Charpentier]

I do not know how to find your email address, and I don't know how to PM you friom Google's Groups. I have to search for this...

--
Emmanuel Charpentier

.

[Volker Braun]

There seems to be a 5-minute timeout, is your internet connection really slow?

[13/Jun/2015:07:57:35 -0700] "GET /sage.git/info/refs?service=git-upload-pack HTTP/1.1" 200 431350

[13/Jun/2015:07:57:37 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 1125

[13/Jun/2015:07:57:38 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 1912

[13/Jun/2015:07:57:38 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 3630

[13/Jun/2015:07:57:39 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 6635

[13/Jun/2015:07:57:39 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 12723

[13/Jun/2015:07:57:40 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 22931

[13/Jun/2015:07:57:41 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 50308

[13/Jun/2015:07:57:43 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 100612

[13/Jun/2015:07:57:45 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 160013

[13/Jun/2015:07:57:48 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 200 222889

[Sat Jun 13 08:02:55 2015] [error] [client x.x.x.x] (70007)The timeout specified has expired: Error reading request entity data

[13/Jun/2015:07:57:52 -0700] "POST /sage.git/git-upload-pack HTTP/1.1" 500 638

[Emmanuel Charpentier]

Le samedi 13 juin 2015 21:13:50 UTC+2, Volker Braun a écrit :

There seems to be a 5-minute timeout, is your internet connection really slow?

Not *THAT* slow... I'm afraid that it might be a routing problem...

--
Emmanuel Charpentier

[Volker Braun]

Actually, the log shows that the first couple of uploads are within 20 seconds and then nothing. You probably have a "transparent" proxy in-between. Can you try to switch the url to force transport over ssh:

[remote "origin"]

    url = g...@trac.sagemath.org:sage.git

    fetch = +refs/heads/*:refs/remotes/origin/*

[remote "trac"]

    url = g...@trac.sagemath.org:sage.git

    fetch = +refs/heads/*:refs/remotes/trac/*

[Emmanuel Charpentier]

That did it ! Yes !

Thank you very much !

I think that I'll have a word with my ISP : such a "transparent" proxy shouldn't be on my connection. What do you think ?

Thanks again !

--
Emmanuel Charpentier (now on to #18691...)

[Emmanuel Charpentier]

Aaaarghh ! I have a patch to propose for trac#18691, which I got by git trac checkout, but i find myself unable to push my patch. OI probably charpent@asus16-ec:/usr/local/sage-6.8$ git remote -v
origin    g...@trac.sagemath.org:sage.git (fetch)
origin    g...@trac.sagemath.org:sage.git (push)
trac    g...@trac.sagemath.org:sage.git (fetch)
trac    g...@trac.sagemath.org:sage.git (push)
messed up my config, which is :

charpent@asus16-ec:/usr/local/sage-6.8$ git remote -v
origin    git@trac.sagemath.org:sage.git (fetch)
origin    git@trac.sagemath.org:sage.git (push)
trac    git@trac.sagemath.org:sage.git (fetch)
trac    git@trac.sagemath.org:sage.git (push)

When I try to push, I get :

charpent@asus16-ec:/usr/local/sage-6.8$ ./sage -git push --set-upstream origin t/18691/trac__17572_breaks_the_installation_of_several_r_packages_
Décompte des objets: 5, fait.
Delta compression using up to 8 threads.
Compression des objets: 100% (5/5), fait.
Écriture des objets: 100% (5/5), 489 bytes | 0 bytes/s, fait.
Total 5 (delta 4), reused 0 (delta 0)
remote: FATAL: W refs/heads/t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ sage charpent DENIED by fallthru
remote: error: hook declined to update refs/heads/t/18691/trac__17572_breaks_the_installation_of_several_r_packages_
To g...@trac.sagemath.org:sage.git
 ! [remote rejected] t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ -> t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ (hook declined)
error: impossible de pousser des références vers 'g...@trac.sagemath.org:sage.git'


The "origin" remote seems suspicious. When I try to replace it by" trac", I get the same error :

charpent@asus16-ec:/usr/local/sage-6.8$ ./sage -git push --set-upstream trac t/18691/trac__17572_breaks_the_installation_of_several_r_packages_
Décompte des objets: 5, fait.
Delta compression using up to 8 threads.
Compression des objets: 100% (5/5), fait.
Écriture des objets: 100% (5/5), 489 bytes | 0 bytes/s, fait.
Total 5 (delta 4), reused 0 (delta 0)
remote: FATAL: W refs/heads/t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ sage charpent DENIED by fallthru
remote: error: hook declined to update refs/heads/t/18691/trac__17572_breaks_the_installation_of_several_r_packages_
To g...@trac.sagemath.org:sage.git
 ! [remote rejected] t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ -> t/18691/trac__17572_breaks_the_installation_of_several_r_packages_ (hook declined)
error: impossible de pousser des références vers 'g...@trac.sagemath.org:sage.git'


The Developer's guide has confused rather than enlightened me.

Any suggestion ?

--
Emmanuel Charpentier

[Volker Braun]

You can only push to branches named u/username/branch or public/branch. If you use "git trac" then it does it for you, otherwise you need to specify the branch by hand.

[Emmanuel Charpentier]

Thank you again, Volker ! But re-reading *trac* docs gave me the solution : I was asininely trying to "git push "something I got by "git trac checkput". "git trac push" worked the first time...

Thanks again !

--
Emmanuel Charpentier