Paramiko EOFErrors start to occur after a while

[Tino de Bruijn]

Hi,

My setup is the following: I have two webservers that serve all dynamic request. I have a seperate static file server (that also has some celery queue workers running). When images are uploaded by users, those images are uploaded to the webservers, and processed async by a queue. In the job they are transfered to the static file server and resized. For the admin however, I want the transfering and resizing done right then, so that when admins post content, it is immediatly visible. Admins post different content than users, so this is implemented by using a different field on the models. The admin images are transfered to the static file server directly by using sftpstorage from django-storages.

The problem is that paramiko (used by django-storages for the sftp implementation) starts to throw EOFErrors. This doesn't always happen. When the mod_wsgi instance is fresh, everything works, but after a while these errors start occurring. Touching the wsgi.py file makes everything work again.

I have build a retry loop (5 times) to try to circumvent network glitches, but it consistently fails. If it needs to retry it'll keep failing.

I have two questions:

1) Does anybody have an idea why this could be happening?

2) Is this the best way to implement this strategy, or would you suggest a different technique?

Thanks,

Tino

[Kurtis Mullins]

I messed with the SFTP implementatoin a while back. I noticed it was pretty tough and, if I'm not mistaken, a bit out-dated. We ended up moving to Amazon S3 so it wasn't a big deal.

Anyways, maybe you could try using another file system? SSH isn't really an ideal solution for this sort of a problem -- even though it works as a great starting point.

I'm not sure if that helps much but good luck to you!

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/UXX_dJrQJOwJ.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Tino de Bruijn]

Hi,

Unfortunatly we can't move to Amazon S3. So what other file system or transfer method would you recommend? We had an NFS in place, but getting it to play nice with permissions was quite a hassle.

How do other people do this?

Tino

On Wednesday, May 2, 2012 1:25:52 AM UTC+2, Kurtis wrote:

I messed with the SFTP implementatoin a while back. I noticed it was pretty tough and, if I'm not mistaken, a bit out-dated. We ended up moving to Amazon S3 so it wasn't a big deal.

Anyways, maybe you could try using another file system? SSH isn't really an ideal solution for this sort of a problem -- even though it works as a great starting point.

I'm not sure if that helps much but good luck to you!

On Tue, May 1, 2012 at 11:49 AM, Tino de Bruijn <tinodb> wrote:

Hi,

My setup is the following: I have two webservers that serve all dynamic request. I have a seperate static file server (that also has some celery queue workers running). When images are uploaded by users, those images are uploaded to the webservers, and processed async by a queue. In the job they are transfered to the static file server and resized. For the admin however, I want the transfering and resizing done right then, so that when admins post content, it is immediatly visible. Admins post different content than users, so this is implemented by using a different field on the models. The admin images are transfered to the static file server directly by using sftpstorage from django-storages.

The problem is that paramiko (used by django-storages for the sftp implementation) starts to throw EOFErrors. This doesn't always happen. When the mod_wsgi instance is fresh, everything works, but after a while these errors start occurring. Touching the wsgi.py file makes everything work again.

I have build a retry loop (5 times) to try to circumvent network glitches, but it consistently fails. If it needs to retry it'll keep failing.

I have two questions:

1) Does anybody have an idea why this could be happening?

2) Is this the best way to implement this strategy, or would you suggest a different technique?

Thanks,

Tino

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/UXX_dJrQJOwJ.
To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

[Phang Mulianto]

Hi,

i also have the script to sftp using python...run with cron every 30 mins,  and suddently since last week, the eol errors come out..

i think this problem bcoz the sftp server..

are you trying rsync to transfer between host ?

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/20YX_C3Oo3AJ.

To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[Kurtis Mullins]

Hey,

I'd take a look at these options if you want to stick with Django-Storages ( http://django-storages.readthedocs.org/en/latest/index.html  listed on the top of the page).

Otherwise, I wonder if you could use the hadoop fileystem or some other, similar distributed file system built for high performance computing? I agree though with your problems using NFS -- I've had a ton of trouble with it in the past. It's not pleasant to work with at all.