Looking for an http proxy server written in Python

Chris Gray

unread,

Nov 18, 2003, 10:10:03 AM11/18/03

to

Is there such a beast? In particular, I'm looking for a
production-quality proxy server fully compliant with HTTP/1.1 written in
Python.

If there isn't, is there anything that comes close?

Thanks for any help you can throw my way,
Chris Gray

"Okey dokey.. free my mind. Right, no problem, free my mind, free my mind,
no problem, right... " -The Matrix

Alan Kennedy

unread,

Nov 18, 2003, 10:20:25 AM11/18/03

to

[Chris Gray]

> I'm looking for a
> production-quality proxy server fully compliant with HTTP/1.1 written in
> Python.
>
> If there isn't, is there anything that comes close?

That depends on what you want the proxy to do.

Check the following page for a list of python http proxies.

http://xhaus.com/alan/python/proxies.html

If there's nothing there that catches your eye, then give a clearer
indication of what you need the proxy to do.

regards,

--
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/mailto/alan

Dave Brueck

unread,

Nov 18, 2003, 10:26:54 AM11/18/03

to python-list

Chris wrote:
> Is there such a beast? In particular, I'm looking for a
> production-quality proxy server fully compliant with HTTP/1.1 written in
> Python.

Forward or reverse proxy?

> If there isn't, is there anything that comes close?

Alan Kennedy has this page:

http://xhaus.com/alan/python/proxies.html

Chris Gray

unread,

Nov 18, 2003, 11:16:51 AM11/18/03

to Alan Kennedy

Thanks, Alan,

Your resource page is very helpful and your "Why" section explains exactly
why I'm interested in this. At first blush it looks like httpMonitor is
most in line with what I'm planning, namely, the ability to do arbitrary
processing on requests and responses.

The problem I've run across so far is transparency and performance. I've
written my own proxy server and experimented with WebDebug and found that
some things slow down considerably and other things don't seem to pass
through at all. I want, as much as possible, for the proxy to pass
requests and responses through in a way that makes no difference to the
client, but gives me a hook to hang selective logging, storage, and other
processing on.

That's why I specified production-quality and HTTP/1.1 compliance. If I
could just find a proxy server that forwarded requests and responses
seamlessly, I'd be happy.

Thanks again,
Chris

"Okey dokey.. free my mind. Right, no problem, free my mind, free my mind,
no problem, right... " -The Matrix

Paul Rubin

unread,

Nov 18, 2003, 2:08:15 PM11/18/03

to

Chris Gray <cpg...@library.uwaterloo.ca> writes:
> Is there such a beast? In particular, I'm looking for a
> production-quality proxy server fully compliant with HTTP/1.1 written in
> Python.

I sometimes use Amit Patel's proxy3 for web surfing, but I don't know
if I'd call it production quality because it's so slow. I don't know
if anything can be done about that, because Python itself is pretty
slow. You may need to bite the bullet and use Squid or the Apache
proxy module or something like that. I've had reasonable success
hacking various html rewriting features into the Apache module when
such things were needed.

John Abel

unread,

Nov 18, 2003, 10:35:32 PM11/18/03

to Chris Gray, pytho...@python.org

You might want to have a look at Willow from www.digitallumber.com.

Regards

John

Chris Gray

unread,

Nov 19, 2003, 9:29:12 AM11/19/03

to

On Tue, 18 Nov 2003, it was written:

>
> I sometimes use Amit Patel's proxy3 for web surfing, but I don't know
> if I'd call it production quality because it's so slow. I don't know
> if anything can be done about that, because Python itself is pretty
> slow. You may need to bite the bullet and use Squid or the Apache
> proxy module or something like that. I've had reasonable success
> hacking various html rewriting features into the Apache module when
> such things were needed.

I may have a look at Squid and mod_proxy (and Jigsaw to boot) at some
point. I'm not as worried about speed as I am about choking on requests
that can't be handled. I'm not trying to do the processing in real time;
that is, the results of the processing don't have to be available to the
client in real time.

Thanks for the feedback,
Chris

Chris Gray

unread,

Nov 19, 2003, 9:32:28 AM11/19/03

to

On Wed, 19 Nov 2003, John Abel wrote:

> You might want to have a look at Willow from www.digitallumber.com.
>
> Regards

Thanks, I hadn't run across this before. Looks interesting.

Cheers,
Chris

Alan Kennedy

unread,

Nov 19, 2003, 4:01:16 PM11/19/03

to

Chris Gray wrote:
> Your resource page is very helpful and your "Why" section explains
> exactly why I'm interested in this. At first blush it looks like
> httpMonitor is most in line with what I'm planning, namely, the
> ability to do arbitrary processing on requests and responses.

I've tried to make the page a little more useful now, by adding
freshness dates, and a better overview of the features offered by the
various proxies.

http://www.xhaus.com/alan/python/proxies.html

> That's why I specified production-quality and HTTP/1.1 compliance. If I
> could just find a proxy server that forwarded requests and responses
> seamlessly, I'd be happy.

FWIW, I mostly run a very simple proxy, derived from Suzuki Hisao's
Tiny proxy, to do protocol and resource logging, which I need when I'm
doing functional and load testing. I store the output in python
syntax, to be fed straight back into a python program. One of these
days I might clean it up a little and release it.

If you have any comments about the page, I'd be happy to hear them.

One of these days, I'm going to a full-on software review of the
proxies for a variety of different purposes. Content filtering and
blocking seems to be much in demand.