Use aiohttp client with multiple interfaces

[akhom...@gmail.com]

Good day.

Is it possible to perform aiohttp client calls from different network interfaces?

What should I do? Create multiple connectors?

Thank you!

[Alexander Shorin]

Hi,

On Wed, Dec 2, 2015 at 7:13 PM, <akhom...@gmail.com> wrote:
> Is it possible to perform aiohttp client calls from different network
> interfaces?
> What should I do? Create multiple connectors?

General solution is to use 0.0.0.0 bind address for ipv4 or :: for ipv6.

--
,,,^..^,,,

[Andrew Svetlov]

There is no way to bind socket address via aiohttp client API.
TCPConnector._create_connection() may pass local_addr parameter into
loop.create_connection() call.
local_addr may be specified in TCPCOnnector.__init__ as keyword-only argument.

The patch looks pretty simple. Is there a volunteer for the issue?

[Al Johri]

Hi Andrew,

If I understand the problem correctly, we're essentially trying to enable using a SOCKS proxy with aiohttp? If this is correct, I'm also very interested in this. I got a basic working example using TOR with asyncio.open_connection (wrapper for create_connection) and PySocks.

This example assumes tor is running on 127.0.0.1:9050.

I'd love if you could point me in the right direction for implementing this in aiohttp. Thanks!

import asyncio

import urllib.parse

import sys

import socket

import socks

socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9050, True)

@asyncio.coroutine

def print_http_headers(url):

    url = urllib.parse.urlsplit(url)

    s = socks.socksocket()

    if url.scheme == 'https':

        s.connect((url.hostname, 443))

        connect = asyncio.open_connection(server_hostname='', ssl=True, sock=s)

    else:

        s.connect((url.hostname, 80))

        connect = asyncio.open_connection(sock=s)

    reader, writer = yield from connect

    query = ('GET {path} HTTP/1.0\r\n'

             'Host: {hostname}\r\n'

             '\r\n').format(path=url.path or '/', hostname=url.hostname)

    writer.write(query.encode('latin-1'))

    while True:

        line = yield from reader.readline()

        if not line:

            break

        line = line.decode('latin1').rstrip()

        if line:

            print('HTTP header> %s' % line)

    # Ignore the body, close the socket

    writer.close()

url = sys.argv[1]

loop = asyncio.get_event_loop()

task = asyncio.ensure_future(print_http_headers(url))

loop.run_until_complete(task)

loop.close()

[Andrew Svetlov]

> On Thursday, December 3, 2015 at 3:18:36 AM UTC+2, Al Johri wrote:
> Hi Andrew,

> If I understand the problem correctly, we're essentially trying to enable using a SOCKS proxy with aiohttp?

No, it's not about SOCKS but just binding specified network address to local socket.


In you example `s.connect((url.hostname, 80))` is blocking operation which is not good.
Also as I see PySocks is not proved to work in nonblocking mode, thus some calls may fail eventually leading to unpredictable behavior.

 

[Al Johri]

https://github.com/nibrag/aiosocks from Nail Ibragimov seems like it may help.

[Al Johri]

Yup, it worked! Big thanks to Nail Ibragimov. https://github.com/AlJohri/python-tor-examples

Andrew, it may be worth adding this connector class directly into aiohttp to improve the API for working with socks5 proxies.

What do you think?

[Andrew Svetlov]

On Sunday, October 23, 2016 at 9:40:08 PM UTC+3, Al Johri wrote:

Yup, it worked! Big thanks to Nail Ibragimov. https://github.com/AlJohri/python-tor-examples

Andrew, it may be worth adding this connector class directly into aiohttp to improve the API for working with socks5 proxies.

What do you think?

I don't want to maintain SOCKS, sorry. Better to keep it in separate library.
But I'm ok with mentioning the library in aiohttp documentation.
The Pull Request is welcome as usual.