Dealing with command spam?
Brian Rak
will cause very slow responses to all other connected users. If I
block the spamming user, everything returns to normal. Any ideas as
to why this is happening and what I can do to prevent it? I don't
really want to have to rate limit commands.
To give you an idea of numbers, this user is spamming SIZE at around
10-15 commands per second. The issue will appear with just this user
and one other user connected, so it doesn't appear to only happen on
highly loaded servers.
When the command is being spammed, even attempting a directory listing
has about a 50% chance of timing out before you get a response.
Giampaolo Rodolà
I think there are softwares to handle such kind of issues at network level, but that's all I know, since I've never used one.
Jay Loden
> We're running into an issue where one user spamming a command (SIZE),
> will cause very slow responses to all other connected users. If I
> block the spamming user, everything returns to normal. Any ideas as
> to why this is happening and what I can do to prevent it? I don't
> really want to have to rate limit commands.
Just to clarify...
The reason this is happening is because pyftpdlib is asynchronous, not
multi-threaded. That means to handle multiple requests it has to kick
off several tasks, then come back around and check to see if each of
them has finished. If so, we return results to the user or move on to
another request.
In this case, the user is spamming SIZE requests which are preventing
the asynchronous model from working properly as it's spending too much
time handling the spammed commands to process other requests. To add
insult to injury, I believe SIZE is one of the slowest running
commands so it's exacerbating the symptoms. The only ways I can think
of to resolve this in pyftpdlib would be to rate-limit the user or the
commands, ban the user (temporarily or permanently).
-Jay
Brian Rak
see if I can make size run a bit faster, a quick look at strace shows
it lstat's the entire directory tree up to the file a few times for
each request.
As a workaround, I lowered serve_forever's timeout to 0.01s, and now
see drastically improved responsiveness. I'm not entirely sure why
that would help if the cause is indeed a slow running size like you
indicated.
I should be able to get rate limits working, but I was hoping to not
have to do that. Unfortunately, banning clients for spamming commands
really wouldn't work out too great. They aren't maliciously spamming
size commands, they are just using some poorly coded software that's
trying to "tail" a log file via FTP.
Giampaolo Rodolà
nature of pyftpdlib represents a problem only when functions taking a
long time to return are involved (e.g. a single db query completing
after 2 seconds).
Here's a simple benchmark script:
import ftplib, time
f = ftplib.FTP()
f.connect('localhost', 2121)
f.login()
f.sendcmd('type i')
stop_at = time.time() + 1
x = 0
while 1:
if time.time() >= stop_at:
break
f.size('/foo.py')
x += 1
print x
On my linux box (3.1 Ghz dual core CPU) pyftpdlib is able to handle
3630 SIZE requests per-second.
The fact that the OP says that 10-15 SIZE requests per second are
enough to hog server resources makes me think that it's likely that
one of such "long running calls" are somewhat involved, in which case
the problem should be fixed at another level (e.g by making the call
run asynchrounously or by using a thread or a sub-process as a last
resort).
Maybe I can elaborate more if the OP provides more info.
How is SIZE exactly implemented? Does it make a query against a db?
Regards,
Giampaolo
2011/7/7 Jay Loden <jlo...@gmail.com>:
> --
> You received this message because you are subscribed to the "Python FTP server library" project group:
> http://code.google.com/p/pyftpdlib/
> To post to this group, send email to pyft...@googlegroups.com
> To unsubscribe from this group, send email to pyftpdlib-...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/pyftpdlib
Giampaolo Rodolà
Giampaolo Rodolà
Not properly tested but it should do the work.
Maybe it makes sense to add this in the demo directory.
from pyftpdlib.ftpserver import FTPHandler, FTPServer,
DummyAuthorizer, CallLater
class AntiFloodHandler(FTPHandler):
"""
A handler which bans an IP if it sends more than x number of commands
per seconds.
"""
cmds_per_second = 50
ban_for = 60 * 60 # 1 hour
banned_ips = []
def __init__(self, *args, **kwargs):
FTPHandler.__init__(self, *args, **kwargs)
self.processed_cmds = 0
self.pcmds_callback = CallLater(1, self.check_processed_cmds)
def handle(self):
# called when client connects.
if self.remote_ip in self.banned_ips:
self.respond('550 you are banned')
self.close()
else:
FTPHandler.handle(self)
def check_processed_cmds(self):
# called every second; checks for the number of commands
# sent in the last second.
if self.processed_cmds > self.cmds_per_second:
self.ban(self.remote_ip)
else:
self.processed_cmds = 0
def process_command(self, *args, **kwargs):
# increase counter for every received command
self.processed_cmds += 1
FTPHandler.process_command(self, *args, **kwargs)
def ban(self, ip):
# ban ip and schedule next un-ban
if ip not in self.banned_ips:
self.log('banned IP %s for command flooding' % ip)
CallLater(self.ban_for, self.unban, ip)
self.respond('550 you are banned for %s seconds' % self.ban_for)
self.close()
self.banned_ips.append(ip)
def unban(self, ip):
# unban ip
try:
self.banned_ips.remove(ip)
except ValueError:
pass
else:
self.log('unbanning IP %s' % ip)
def close(self):
FTPHandler.close(self)
if not self.pcmds_callback.cancelled:
self.pcmds_callback.cancel()
def main():
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', '.', perm='elradfmw')
authorizer.add_anonymous('.')
handler = AntiFloodHandler
handler.authorizer = authorizer
ftpd = FTPServer(('', 2121), handler)
ftpd.serve_forever(timeout=1)
if __name__ == '__main__':
main()
Brian Rak
file on disk. We aren't using a custom filesystem or anything, just a
custom authorizor (to read proftpd-style configs), and a custom
handler (to use a different SSL cert per-ip, and a different config
file path per-ip).
While you are running your benchmark script, did you try accessing the
FTP server from a normal client? The user running the SIZE requests
is getting their responses back pretty much instantly, so it's not
really the number of SIZE requests that can be handled per second
that's the issue.
Giampaolo Rodolà
> SIZE in this case is just the normal, default, grab the size of the
> file on disk. We aren't using a custom filesystem or anything, just a
> custom authorizor (to read proftpd-style configs), and a custom
> handler (to use a different SSL cert per-ip, and a different config
> file path per-ip).
>
> While you are running your benchmark script, did you try accessing the
> FTP server from a normal client? The user running the SIZE requests
> is getting their responses back pretty much instantly, so it's not
> really the number of SIZE requests that can be handled per second
> that's the issue.
I've tried right now by using a script and a graphical client (FileZilla).
Here's the script:
import ftplib
f = ftplib.FTP()
f.connect('localhost', 2121)
f.login()
f.sendcmd('type i')
while 1:
f.size('/foo.py')
While this is running I'm able to navigate the filesystem with
FileZilla without noticing slowdowns.
This is 2 clients connected simultaneously.
How many clients are connected when you notice the slowdown? What are
they doing exactly? Maybe the slowdown if not related with SIZE at
all.
As I said, if there are no blocking functions in your code pyftpdlib
should scale just fine, even with many clients.
Even if there's a client actively flooding your server that shouldn't
make a big difference.
If you're still sure SIZE is the cause of the problem try to run the
benchmark script I pasted above and paste the result.
Also, I find strange that lowering serve_forever() timeout results in
a drastical speed up.
That shouldn't happen.
Regards,
Giampaolo