Trac slow

[Erik Bray]

Just posting to say I've noticed and am investigating.

It seems that somebody is performing a scrape of the front page which
includes a list of users on the site [1] each of which is a link to a
search query for all tickets and wiki pages associated with that user.
That's what's causing the slowdown.

The IP address seems to belong to Opera (maybe it's a VPN?)
http://tools.tracemyip.org/lookup/107.167.122.77


[1] https://trac.sagemath.org/#AccountNamesMappedtoRealNames

[Dima Pasechnik]

On Thursday, April 20, 2017 at 3:38:12 PM UTC+1, Erik Bray wrote:

Just posting to say I've noticed and am investigating.

It seems that somebody is performing a scrape of the front page which
includes a list of users on the site [1] each of which is a link to a
search query for all tickets and wiki pages associated with that user.
That's what's causing the slowdown.

The IP address seems to belong to Opera (maybe it's a VPN?)  

http://tools.tracemyip.org/lookup/107.167.122.77

note that the map on the right shows that the connection originates in China.

Thus yes, it's a VPN. Actually, is sagemath.org blocked in China? I don't know.

 

[1] https://trac.sagemath.org/#AccountNamesMappedtoRealNames

[Erik Bray]

Strange, well, I created a firewall rule blocking that IP for now and
everything seems to be fine.

[John H Palmieri]

Since some time last night, I have been unable to use "git trac checkout TICKET":

$ git trac checkout 22764
Loading ticket #22764...
Checking out Trac #22764 remote branch fb90d5e3667237f2d9fc7ba982b466bc52802b1d -> local branch t/22764/fb90d5e3667237f2d9fc7ba982b466bc52802b1d...
Traceback (most recent call last):
  File "/usr/local/bin/git-trac", line 18, in <module>
    cmdline.launch()
  File "/Library/Python/2.7/site-packages/git_trac/cmdline.py", line 215, in launch
    app.checkout(args.ticket_or_branch, args.branch_name)
  File "/Library/Python/2.7/site-packages/git_trac/app.py", line 116, in checkout
    self._checkout_ticket(int(ticket_or_branch), branch_name)
  File "/Library/Python/2.7/site-packages/git_trac/app.py", line 144, in _checkout_ticket
    self.repo.checkout_new_branch(ticket.branch, branch)
  File "/Library/Python/2.7/site-packages/git_trac/git_repository.py", line 136, in checkout_new_branch
    self.git.fetch('trac', remote)
  File "/Library/Python/2.7/site-packages/git_trac/git_interface.py", line 341, in meth
    return self.execute(git_cmd, *args, **kwds)
  File "/Library/Python/2.7/site-packages/git_trac/git_interface.py", line 328, in execute
    popen_stderr=subprocess.PIPE)
  File "/Library/Python/2.7/site-packages/git_trac/git_interface.py", line 263, in _run
    raise GitError(result)
git_trac.git_error.GitError: git returned with non-zero exit code (1) when executing "git fetch trac fb90d5e3667237f2d9fc7ba982b466bc52802b1d"

Is this related?

--
John

[Daniel Krenn]

On 2017-04-20 17:09, John H Palmieri wrote:
> Since some time last night, I have been unable to use "git trac checkout
> TICKET":

This (other ticket) worked for me about an hour ago.

Daniel

[John H Palmieri]

Yes, you're right. It's just this ticket causing problems.

  John
 

Daniel

[Dima Pasechnik]

Does this mean that we need some robots.txt somewhere, perhaps after some restructuring,

which would protect expensive resources from this sort of overload?

[Thierry]

Hi,

On Thu, Apr 20, 2017 at 12:10:42PM -0700, Dima Pasechnik wrote:
> >
> > Strange, well, I created a firewall rule blocking that IP for now and
> > everything seems to be fine.
> >
>
> Does this mean that we need some robots.txt somewhere, perhaps after some
> restructuring,
> which would protect expensive resources from this sort of overload?

If the "culprit" host does not announce itself as a friendly bot (look at
the webserver access logs), i doubt it will kindly follow what is
suggested in robots.txt. Firewalling is the way to go, it is at the lowest
level, hence much lighter than application-level solutions (.htaccess or
whatever).

Ciao,
Thierry

[Erik Bray]

There already is a robots.txt and this host was not respecting it.

[Michael Orlitzky]

On 04/21/2017 05:30 AM, Erik Bray wrote:
>>
>> Does this mean that we need some robots.txt somewhere, perhaps after some
>> restructuring,
>> which would protect expensive resources from this sort of overload?
>
> There already is a robots.txt and this host was not respecting it.
>

If this becomes a bigger problem, one solution is to make sure the main
anonymous trac pages are cached plain HTML files, and to severely limit
e.g. the search function (with a CAPTCHA, rate limit, etc.)

One motivated person can still slow you down, but they'll have to at
least try -- I think most of these bot attacks are only accidentally a
denial of service.

[Anne Schilling]

I just tried

git fetch origin
fatal: unable to connect to trac.sagemath.org:
trac.sagemath.org[0: 104.197.143.230]: errno=Operation timed out

Is this related?

Anne

[Dima Pasechnik]

On Friday, April 28, 2017 at 4:59:48 AM UTC+1, Anne Schilling wrote:

I just tried

git fetch origin
fatal: unable to connect to trac.sagemath.org:
trac.sagemath.org[0: 104.197.143.230]: errno=Operation timed out

worked for me now (~3 hours later).

[Erik Bray]

Did it ever starting working for you again, Anne? I'm not seeing any
problems here.

For what it's worth, before anyone else asks, the original problem
that this thread was about had to do with a badly behaving web scraper
that was causing a DOS on the Trac website itself--this would not
impact git access over SSH since that doesn't involve the web server.

(Also, unrelated, but consider using git.sagemath.org for accessing
the git repository, as opposed to trac.sagemath.org. While both
addresses currently point to the same server, that may not always be
the case.)

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Kwankyu Lee]

Hi,

For the remote origin, the usual configuration is 

[remote "origin"]

url = git://github.com/sagemath/sage.git

fetch = +refs/heads/*:refs/remotes/origin/* 

If the same fetch setting is used for the remote trac, then the git client would try to download all heads from the trac. 

I confess that I mistakenly committed this "crime" recently :-( and fixed it as soon as I recognized it.

Might this be a cause of the recent trac slowdown? By the way, I did not experience it myself... I am not sure if this is related and/or makes sense...

[Erik Bray]

On Fri, Apr 28, 2017 at 12:49 PM, Kwankyu Lee <ekwa...@gmail.com> wrote:
> Hi,
>
> For the remote origin, the usual configuration is
>
> [remote "origin"]
> url = git://github.com/sagemath/sage.git
> fetch = +refs/heads/*:refs/remotes/origin/*
>
> If the same fetch setting is used for the remote trac, then the git client
> would try to download all heads from the trac.
>
> I confess that I mistakenly committed this "crime" recently :-( and fixed it
> as soon as I recognized it.
>
> Might this be a cause of the recent trac slowdown? By the way, I did not
> experience it myself... I am not sure if this is related and/or makes
> sense...

No. To be more explicit, the Trac slowdowns have exactly nothing to do
with git...

[Anne Schilling]

I am sorry if I posted in the wrong thread (should I open a new one?).

But for me this still does not work. I cannot even ping trac:

ping sine.math.ucdavis.edu
PING sine.math.ucdavis.edu (169.237.99.79): 56 data bytes
64 bytes from 169.237.99.79: icmp_seq=0 ttl=57 time=13.852 ms
64 bytes from 169.237.99.79: icmp_seq=1 ttl=57 time=16.273 ms
....

anne$ ping trac.sagemath.org
PING trac.sagemath.org (104.197.143.230): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
...

anne$ ping git.sagemath.org
PING trac.sagemath.org (104.197.143.230): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
...

Should I be using the github url 

url = git://github.com/sagemath/sage.git

This is not what my current set-up does (but it has always worked before):

anne$ git remote -v
origin	git://trac.sagemath.org/sage.git (fetch)
origin	git://trac.sagemath.org/sage.git (push)
trac	git://trac.sagemath.org/sage.git (fetch)
trac	g...@trac.sagemath.org:sage.git (push)

Best,

Anne

[Volker Braun]

Did you run that from within UC Davis? There is quite a number of academic networks that subscribe to the "if we filter out ICMP at the network border we can't be hacked" fallacy. A better test would be:

$ ssh g...@trac.sagemath.org

PTY allocation request failed on channel 0

hello vbraun, this is git@trac running gitolite3 3.5.3.1-2 (Debian) on git 1.9.1

 R W gitolite-admin

 R W sage

Connection to trac.sagemath.org closed.

PS: the github url is for releases, so if you want to work on tickets you need to be able to connect to trac...

[Anne Schilling]

Thanks! I actually fixed my problem now. There was something broken with my openssl installation. Sorry for the noise!

Best,

Anne