A subprocess call fails, but only under Django

[Shawn Milochik]

Hi everybody. I have a weird problem. I have a small script that does a subprocess call. It works. It works when run via iPython. It blows up when run in manage.py shell or under Django with OSError: [Errno 12] Cannot allocate memory. Does anyone have any idea on how to fix this?

It's not doing anything that uses much memory -- even if I just make a subprocess call to the "ls" command it happens -- but only under Django.

I've done a bunch of Google searches and checked with the IRC channel, but so far nothing.

Thanks,

Shawn

[Russell Keith-Magee]

On Wed, Mar 12, 2014 at 2:06 AM, Shawn Milochik <shawn...@gmail.com> wrote:

Hi everybody. I have a weird problem. I have a small script that does a subprocess call. It works. It works when run via iPython. It blows up when run in manage.py shell or under Django with OSError: [Errno 12] Cannot allocate memory. Does anyone have any idea on how to fix this?

It's not doing anything that uses much memory -- even if I just make a subprocess call to the "ls" command it happens -- but only under Django.

I agree - this sounds like a doozy.

 

Can you clarify what you mean by "under Django"? 

Also - when you're running './manage.py shell' - is this activating iPython, or is it starting a "plain" shell? Does the script work under a plain shell by itself?

(Mostly asking vague questions because I'm just as confused as to why you're getting OSErrors; hoping a pattern might emerge if we test it in different places…)

Yours,

Russ Magee %-)

[Shawn Milochik]

Hi Russ, thanks for the reply.

What I mean by "under Django" is if I call it from within a view or with "manage.py shell" I get the problem. On the system in question, "manage.py shell" does in fact invoke iPython. However, invoking iPython manually (without manage.py) works. Executing the script stand-alone works.

I tried to circumvent the entire problem by adding an "if __name__ == '__main__'" block to the script and having it take a command-line argument via argparse and print the desired output. Then I did a subprocess.check_output call on that instead of importing the module. This worked from "manage.py shell," but not within a view in Django -- the same OS error 12.

The error happens in a call to os.fork within the subprocess.call or subprocess.check_output methods (I tried both). The traceback is here: http://bpaste.net/show/NRYtgvzoqyOBawl6XoxY/

I'm going to have to keep poking at it, but I was really hoping someone had encountered this before. I did see some cases via Google on forums and StackOverflow where people reported the problem, but there were no answers other than directly addressing the amount of memory required, but I'm pretty sure this script isn't consuming much memory. Even when it's not running ls, it's just doing a subprocess call to gpg to sign and encrypt a file. It takes well under one second.

Thanks for looking at it.

Shawn

[Drew Ferguson]

Hi

Could your problem be some SELINUX issue?

Perhaps disable SELINUX temporarily and see if the problem persists.

On Tue, 11 Mar 2014 20:07:48 -0400
Shawn Milochik <shawn...@gmail.com> wrote:

> Hi Russ, thanks for the reply.
>
> What I mean by "under Django" is if I call it from within a view or with
> "manage.py shell" I get the problem. On the system in question,
> "manage.py shell" does in fact invoke iPython. However, invoking iPython
> manually (without manage.py) works. Executing the script stand-alone
> works.
>
> I tried to circumvent the entire problem by adding an "if __name__ ==
> '__main__'" block to the script and having it take a command-line
> argument via argparse and print the desired output. Then I did a

> subprocess.check_output call on *that* instead of importing the module.

> This worked from "manage.py shell," but not within a view in Django --
> the same OS error 12.
>
> The error happens in a call to os.fork within the subprocess.call or
> subprocess.check_output methods (I tried both). The traceback is here:
> http://bpaste.net/show/NRYtgvzoqyOBawl6XoxY/
>
> I'm going to have to keep poking at it, but I was really hoping someone
> had encountered this before. I did see some cases via Google on forums
> and StackOverflow where people reported the problem, but there were no
> answers other than directly addressing the amount of memory required,
> but I'm pretty sure this script isn't consuming much memory. Even when
> it's not running ls, it's just doing a subprocess call to gpg to sign
> and encrypt a file. It takes well under one second.
>
> Thanks for looking at it.
> Shawn
>

--
Drew Ferguson

[Nick Santos]

Hey Shawn,

What does your web stack and environment look like? If it's failing during a fork with an out of memory, that makes me wonder if the host process for django is consuming a chunk of memory for some reason, and when it gets forked, it'll get replicated over (which supposedly would fail, even with copy on write, with certain OS settings - I'm far from an expert in this area). Have you tested this code on another machine to see if it exhibits the same behavior?

-Nick

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAOzwKwFtAFocN0riTpViRhwRxTuvGR4dX6T53ToQdwJHZAr7bQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Shawn Milochik]

On Tue, Mar 11, 2014 at 9:06 PM, Drew Ferguson <dr...@afccommercial.co.uk> wrote:

Hi

Could your problem be some SELINUX issue?

Perhaps disable SELINUX temporarily and see if the problem persists.

How would selinux cause the problem to only happen under certain conditions? I don't think it's running. Based on this page[1] it doesn't appear that it is the default on Ubuntu, and sestatus doesn't appear to be available on Ubuntu server or in the apt-get repository.  

[1] https://wiki.ubuntu.com/SELinux

[Shawn Milochik]

On Tue, Mar 11, 2014 at 9:24 PM, Nick Santos <ultr...@gmail.com> wrote:

Hey Shawn,

What does your web stack and environment look like? If it's failing during a fork with an out of memory, that makes me wonder if the host process for django is consuming a chunk of memory for some reason, and when it gets forked, it'll get replicated over (which supposedly would fail, even with copy on write, with certain OS settings - I'm far from an expert in this area). Have you tested this code on another machine to see if it exhibits the same behavior?

-Nick

Nick,

Thanks for the ideas. It was being run with gunicorn. I'll try with other options, such as runserver and try to get it working on a separate machine. The reason I've only been testing it on one server is that it's the one set up with the gpg keychain our single-sign-on needs.

Shawn 

[Shawn Milochik]

On Tue, Mar 11, 2014 at 8:07 PM, Shawn Milochik <shawn...@gmail.com> wrote:

I tried to circumvent the entire problem by adding an "if __name__ == '__main__'" block to the script and having it take a command-line argument via argparse and print the desired output. Then I did a subprocess.check_output call on that instead of importing the module. This worked from "manage.py shell," but not within a view in Django -- the same OS error 12.

TL;DR; 

It's working now, but I don't know why. Thanks for looking, nothing to see here.

I'm bored, tell me all about it:

I just tried this again with runserver and gunicorn, and this workaround was working in both cases. Then I tried again importing the module directly and it continued to work. It's looking like there was some other problem and perhaps the "Django connection" was a red herring. I really don't think there's less load on this machine now compared to earlier in the day.

This is really puzzling, because was able to replicate the problem consistently and repeatedly earlier in the day. 

Thanks to Russ, Drew, and Nick for taking the time to listen to my woes and try to help.