Re: Python - CGI-BIN - Apache Timeout Problem

[Chris Rebert]

On Fri, Mar 2, 2012 at 12:09 PM, Sean Cavanaugh (scavanau)
<scav...@cisco.com> wrote:
<snip>
> THE PROBLEM:
>
> When I execute the scripts from the command line (#python main.py) it
> generates it fine (albeit slowly), it prints all the html code out including
> the script.  The ‘core’ part of the script dumbed down to the lowest level
> is->
>
>         proc = subprocess.Popen(['/usr/local/bin/python', 'tests.py'],
> stdout=subprocess.PIPE)
>         output = proc.stdout.read()

Note the red warning box about possible deadlock with .stdout.read()
and friends:
http://docs.python.org/library/subprocess.html#popen-objects

>         print output
>         proc.stdout.close()

As the docs advise, try using .communicate()
[http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
] instead:
proc = subprocess.Popen(…)
out, err = proc.communicate()
print out

> When I open main.py and execute the script it just hangs… it seems to
> execute the script (I see pcap fires on the interface that I am testing on
> the firewall) but its not executing correctly… or loading the entire
> webpage…the webpage keeps chugging along and eventually gives me an error
> timeout.

The hanging makes me suspect that the aforementioned deadlock is occurring.

Cheers,
Chris
--
http://chrisrebert.com

[Sean Cavanaugh (scavanau)]

Hey Chris,

Thanks for your quick reply! I switched my code to->

proc = subprocess.Popen(['/usr/local/bin/python', 'tests.py'], stdout=subprocess.PIPE)

out, err = proc.communicate()
print out

proc.stdout.close()

It still dead locked. Interestingly enough When I did a #python tests.py on the command line even that was taking awhile to print out to the command line so I decided to restart my webserver... wow from mucking before something must have been running in the background still. I got the script down to 2 seconds or so...

Now it still works but faster when I do #python main.py it generates all the text to the command line but the website still hangs when I go to http://webserver/main.py... I am not sure what is going wrong... no error in the /var/log except for the eventual timeout after a couple minutes goes by.

-S

[Sean Cavanaugh (scavanau)]

Hey All,

So maybe this part is important (after doing some troubleshooting) hopefully not everyone has beers in hand already since its Friday :-)

The way the code works if you want to send through the firewall (i.e. SERVER->FIREWALL->SERVER) I split the process into two threads. One is listening on the egress, then I send on the ingress. The main waits until the thread finishes (it times out after 2 seconds). I had to do this b/c scapy (the library I used) won't let me send pcaps while I receive them. This lets me see packets on both sides (i.e. did that sort of internet traffic go through the firewall). The reason I think this could be a problem is when I ran a test where I sent to the firewall (rather than through it) and my program does not need to thread the webserver works fine......

Suggestions anyone?

[Sean Cavanaugh (scavanau)]

Thanks Chris,

I isolated it using logging

import logging
logging.basicConfig(filename="test3.log", level=logging.INFO)
then logging.info('sniffer got to point A')

and going through my code until I isolated the problem to a function with scapy called sniff. For some reason this function operates very weirdly on FreeBSD9. I have already submitted an email to the scapy mailing list. Going to try to replicate this on Fedora but I doubt I will see this problem. Even from the command line the sniff() function is not working correctly on FreeBSD 9.