Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
How would I do cross domain with Pyramid and Nginx
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
  15 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Mark Huang  
View profile  
 More options Jun 14 2012, 2:58 am
From: Mark Huang <zhengha...@gmail.com>
Date: Wed, 13 Jun 2012 23:58:26 -0700 (PDT)
Local: Thurs, Jun 14 2012 2:58 am
Subject: How would I do cross domain with Pyramid and Nginx

Hi,

I'm not sure if this question belongs here because it touches Pyramid as
well as Nginx.

I have two web applications, one called rhino another called mantis
(rhino.abc.com & mantis.abc.com).  Mantis is some sort of web service
without any page templates; it converts data sent from Rhino into PDFs or
Excel documents.  I've done something similar in the past; sending data
cross domain using JSONP, but then I switched to using Apache ProxyPass,
unfortunately, I had a server specialist to set that up for me.    I want
to use Nginx's proxy_pass directive but I'm not too sure how to set it up
so that Pyramid works with it.

Scenario use case:

User clicks on a button in the UI.  This triggers an ajax request to the
backend of Rhino.  Rhino pulls and processes data and returns a data
structure as a response to the ajax request.  Here comes the tricky part
(unsure part): I send another ajax request to
rhino.abc.com/pdf/reports/get_doc with the response data. Nginx
"understands" the /pdf/reports/get_doc path and uses proxy pass to
"redirect" that to mantis as mantis.ebalu.com/pdf/reports/get_doc.

My question here:  Is there a way for Pyramid to immediately send the
processed data to Mantis without having to respond to the ajax request and
then making a secondary ajax request to Mantis?

My mantis nginx configuration isn't doing anything special here, but my
rhino nginx configuration is as follows:

upstream rhino.ebalu.com {
    server 127.0.0.3:6545 fail_timeout=0;

}

server {
    client_max_body_size 4G;
    server_name rhino.abc.com;
    keepalive_timeout 8;
    root /web/data/prod/rhino/rhino;

    error_log    /web/data/logs/rhino/error.log;
    access_log   /web/data/logs/rhino/access.log;

    location = /favicon.ico {
      return 204;
      access_log     off;
      log_not_found  off;
    }

    location / {
        proxy_set_header X-Forwarded_For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        if (!-f $request_filename) {
            proxy_pass http://rhino.abc.com;
        }
    }

    location ^~ /pdf/ {
        #rewrite ^/pdf/(.*) /pdf/$1 break;              <------What do I do
here?
        *proxy_pass http://mantis.abc.com/;*
    }

}

Can someone please help me out?  

Regards,
Mark Huang


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jun 14 2012, 7:17 am
From: Mark Huang <zhengha...@gmail.com>
Date: Thu, 14 Jun 2012 04:17:38 -0700 (PDT)
Local: Thurs, Jun 14 2012 7:17 am
Subject: Re: How would I do cross domain with Pyramid and Nginx

I figured out the way to configure the nginx, so that part is done.  My
question still remains.  Is it possible to "forward" some data cross-domain
and receive soemthing back in response?

I noticed the Response object has a body attribute.  Is this used to pass
data?

Regards,
Mark Huang


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andi  
View profile  
 More options Jun 14 2012, 8:15 am
From: Andi <andi.ba...@googlemail.com>
Date: Thu, 14 Jun 2012 14:15:15 +0200
Local: Thurs, Jun 14 2012 8:15 am
Subject: Re: How would I do cross domain with Pyramid and Nginx

hi,

you question is a little hard to understand for me. but it seems you want to connect the response of a to A request to B.

using an ajax call crossdomain is easy if you own both domains. look for cors (cross origin resource sharing).  there's also a great js lib (which uses that) out there, dont remember the name now.

but i think ajax is off topic here as you want to transport data. this should not be done via the client for security aspects.

a useful solution would be in
my opinion to:

- request A through nginx
- A processes the request and responds, adding a "x-accel-redirect" header with the nginx location for B
- nginx uses the response and forwards to B in a new request for B
- B processes whatever it wants, changes the response if it wants and gives back to the client

everything is in your backend, youre not blocking anywhere, no subrequests, etc.

we used this technique in a heavyloaded application, all fine with it. but its crucial to have a good testing setup to build it reliable. check out doctests and lovely.testlayers for building a sandbox to test this stuff.

cheers, andi

(sent right out of my head)

On 14.06.2012, at 13:17, Mark Huang <zhengha...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Jonathan Vanasco  
View profile  
 More options Jun 14 2012, 9:08 am
From: Jonathan Vanasco <jonat...@findmeon.com>
Date: Thu, 14 Jun 2012 06:08:37 -0700 (PDT)
Local: Thurs, Jun 14 2012 9:08 am
Subject: Re: How would I do cross domain with Pyramid and Nginx
i started typing this before i saw andi's repsonse.

If both these domains are on the same machine...

nginx has a facility to allow one request to "authorize" a second
request.  it's the x-accel / x-sendfile modules.

http://wiki.nginx.org/X-accel

if they're on different machines :

- you can proxypass the redirect to another server, but you'd probably
want to add in some sort of authticket type header, so that you ensure
the redirects/proxies were from a server you control

- you could do a wildcard cookie that anything on an abc domain can
read, and then stash values there.  ie: server one sets a cookie that
says "send me this file" , and server 2 reads that cookie and serves
that specific file.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andi Balke  
View profile  
 More options Jun 14 2012, 10:13 am
From: Andi Balke <andi.ba...@googlemail.com>
Date: Thu, 14 Jun 2012 16:13:53 +0200
Local: Thurs, Jun 14 2012 10:13 am
Subject: Re: How would I do cross domain with Pyramid and Nginx

thanks jonathan, good idea ;)

i remember one thing to add to our answers.

> - you can proxypass the redirect to another server, but you'd probably
> want to add in some sort of authticket type header, so that you ensure
> the redirects/proxies were from a server you control

i think its notable that the nginx should be the one and only reverse proxy for both domains. no matter if A and B are hosted on the same machine. nginx as an entrypoint matters.

when you think about forwarding to a public endpoint, e.g. B is *not* behind the same reverseproxy (nginx), you should think in the same way as if you request from the client to B. because you won't know if A is the only machine, accessing B and if you don't know this you have to validate anything the same way as if the client accesses B. i dont think its a good solution then to use the nginx setup.

in case you have two public domains which are not behind the same proxy, you could check something like that one: http://packages.python.org/itsdangerous/

andi


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jun 14 2012, 11:51 am
From: Mark Huang <zhengha...@gmail.com>
Date: Thu, 14 Jun 2012 08:51:48 -0700 (PDT)
Local: Thurs, Jun 14 2012 11:51 am
Subject: Re: How would I do cross domain with Pyramid and Nginx

Hi Andi and Jonathan,

Both of you guys are spot on.  Yes, I want to connect the response of A to
the request of B.  

Currently I managed to perform a rewrite+proxy pass where making a request
to http://A.com/pdf/generate_document would forward the request to
http://B.com/pdf/generate_document.    This was all done in NGINX.  The
problem with this is that, only the function (at B) is called, but no data
is being passed along.

Your suggestions of using the X-ACCEL headers, is that done in the server
configuration of A or is that done in B OR would that be done by altering
the Response object?

Regards,
Mark Huang


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jun 14 2012, 12:36 pm
From: Mark Huang <zhengha...@gmail.com>
Date: Thu, 14 Jun 2012 09:36:28 -0700 (PDT)
Local: Thurs, Jun 14 2012 12:36 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

So I went to do some self study and realized that Pyramid response object
did not support such a header natively.  There seemed to be a package
called wsgithumbs <http://packages.python.org/wsgithumb/index.html> that
did something like this, only thing is, the documentation was rather poor.  

Andi, could you illustrate what your setup was on Pyramid to do this
X-Accel thingy?  


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gael Pasgrimaud  
View profile  
 More options Jun 14 2012, 12:40 pm
From: Gael Pasgrimaud <g...@gawel.org>
Date: Thu, 14 Jun 2012 18:40:23 +0200
Local: Thurs, Jun 14 2012 12:40 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx
Hi,

On 14/06/2012 18:36, Mark Huang wrote:

> So I went to do some self study and realized that Pyramid response
> object did not support such a header natively. There seemed to be a
> package called wsgithumbs
> <http://packages.python.org/wsgithumb/index.html> that did something
> like this, only thing is, the documentation was rather poor.

Contributions are welcome ;)

I recently moved the code to the pyramid-collective

https://github.com/pyramid-collective/wsgithumb

Cheers,

Gael


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Michael Merickel  
View profile  
 More options Jun 14 2012, 12:51 pm
From: Michael Merickel <mmeri...@gmail.com>
Date: Thu, 14 Jun 2012 11:51:41 -0500
Local: Thurs, Jun 14 2012 12:51 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

On Thu, Jun 14, 2012 at 11:36 AM, Mark Huang <zhengha...@gmail.com> wrote:
> Andi, could you illustrate what your setup was on Pyramid to do this X-Accel
> thingy?

The idea is to send a response that has the X-Accel header.. nginx
sees that header in the response and sends to the real client a
response without that header but containing the file content in the
body.

def my_view(request):
    response = Response()
    response.content_type = 'some-pdf-type'
    response.headers['X-Accel'] = '/path/to/file'
    return response

You then need to setup nginx as per its docs to support watching for
that header.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gael Pasgrimaud  
View profile   Translate to Translated (View Original)
 More options Jun 14 2012, 12:58 pm
From: Gael Pasgrimaud <g...@gawel.org>
Date: Thu, 14 Jun 2012 18:58:49 +0200
Local: Thurs, Jun 14 2012 12:58 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx
On 14/06/2012 18:51, Michael Merickel wrote:

That's what wsgithumb do. Except that you can configure it to not use
nginx during development (eg when the accel_header param is None)

See http://packages.python.org/wsgithumb/api.html#serving-files


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jun 14 2012, 3:04 pm
From: Mark Huang <zhengha...@gmail.com>
Date: Thu, 14 Jun 2012 12:04:58 -0700 (PDT)
Local: Thurs, Jun 14 2012 3:04 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

Thanks Michael for your response once again (here in google groups as well
as stackoverflow, I really appreciate the help).

With your example would I be able to "send" some preprocessed data to the
real client?  The real client requires some data that is processed on my
end to generate the PDF.  Is that done using the Response object's body
attribute?

Example.

def my_view(request):
    response = Response()
    *response.body = {"title": "Orders", "table": [ ["Red", 1, 2, 1, 1, 2],
['Blue', 3, 4, 3,1] ] }*
    response.content_type = 'some-pdf-type'
    response.headers['X-Accel'] = '/path/to/file'
    return response

Then over in the real client, I can read the data using the Request
object's body attribute like req.body?

One last thing to clarify reagarding the path to a file that I set for
response.headers['X-Accel']:

What if the path is not pointing to a file?  Because I'm just passing some
data over to the real client to process and return me a PDF file.  So
taking the example here<http://kovyrin.net/2010/07/24/nginx-fu-x-accel-redirect-remote/>as inspiration:

response.headers['X-Accel-Redirect'] = '/api/pdf/generate_doc'

And then in my nginx configuration, I add a location block like so:

location ~* /api/(.*) {
    set $download_uri $1;
    set $download_url http://<the_real_client>.com/$download_uri;
    proxy_set_header Host <the_real_client>.com
    proxy_pass $download_url

}

Then over in the real client I will receive the request and be able to read
the contents of the Response from A as req.body.

Did I get this part right?

I'm sorry for being such a noob.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jun 17 2012, 1:49 pm
From: Mark Huang <zhengha...@gmail.com>
Date: Sun, 17 Jun 2012 10:49:10 -0700 (PDT)
Local: Sun, Jun 17 2012 1:49 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

Just wanted to confirm one thing before this is closed.  How do I transfer
the data that I processed via the response?  

Is it using the response.body to add the data in there?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andi Balke  
View profile  
 More options Jun 17 2012, 6:23 pm
From: Andi Balke <andi.ba...@googlemail.com>
Date: Mon, 18 Jun 2012 00:23:10 +0200
Local: Sun, Jun 17 2012 6:23 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

hi mark,

we used this setup in an authentication proxy, nginx + tornado as upstream. all non-public requests went to tornado first, this one did an ``x-accel-redirect`` response with information put in the responseheaders. nginx took the responsecode to distiguish between authorized and unauthorized. additional headers where used to apply information about a user to the real backend service.

two things to note:

- in our case this tornado was a simple http upstream pool, no uwsgi as in your case. i think that should be managable with a uwsgi location somehow too.
- we had no need to pass any responsebody to the second request. and the more i think about it, it makes no sense to have a body passed on a redirect in general. so sorry for the confusion, if this hint was wrong (still i didn't try yet).

you can do something more advanced - not to say crazy ;) - and use the nginx lua module. with that i'm sure it works. i built some request splitter for a friend which works well. looks somehow like that (you could do ``two sequences ``capture`` instead of the  ``capture_multi``)::

    location /multi {
        default_type 'text/html';

        # echo does not work at all if ``content_by_lua`` is used
        # echo "start lua";
        content_by_lua '
            local res_one, res_two = ngx.location.capture_multi {
                { "/one" },
                { "/two" }
            }

            if res_one.status == 200
                    and res_two.status == 200 then
                -- ``ngx.header`` must be set before print
                ngx.header["X-Status"] = "Lua"
                ngx.print(res_one.body.." ... "..res_two.body)
                -- ngx.exit(ngx.OK)
                -- return ngx.redirect("/cache");
                return
            end

            -- this one does not work: ``ngx.exit(ngx.ERROR)`
            -- whereas setting the status does...
            ngx.status = ngx.HTTP_INTERNAL_SERVER_ERROR
            ngx.print("error "..ngx.time())
            ';
    }

    location /one {
        default_type 'text/plain';
        echo "one";
    }

    location /two {
        default_type 'text/plain';
        echo "two";
    }

according to http://wiki.nginx.org/HttpLuaModule this is still non-blocking: "Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code executed using this module can be 100% non-blocking on network traffic as long as the Nginx API for Lua provided by this module is used to handle requests to upstream services..."

here is also the relevant part from the buildout, i remember this was a bit bitchy to get it working. atm under lion it failed to compile, but its a start::

[lua]
recipe = hexagonit.recipe.cmmi
url = http://www.lua.org/ftp/lua-5.1.4.tar.gz
; url = http://www.lua.org/ftp/lua-all.tar.gz
strip-top-level-dir=true
configure-command = /usr/bin/true
make-options = INSTALL_TOP=${buildout:directory}/parts/lua ${os:make_opt}

[lua_jit]
recipe = hexagonit.recipe.cmmi
url = http://luajit.org/download/LuaJIT-2.0.0-beta9.tar.gz
strip-top-level-dir=true
configure-command = /usr/bin/true
make-options = PREFIX=${lua:location}

[nginx_lua_module]
recipe = hexagonit.recipe.download
# url = https://github.com/chaoslawful/lua-nginx-module/tarball/v0.1.6rc4
# url = https://download.github.com/saga-lua-nginx-module-v0.0.1rc8-0-g0ef14a...
url = https://download.github.com/chaoslawful-lua-nginx-module-v0.1.6rc10-0...
strip-top-level-dir=true

[nginx]
recipe = hexagonit.recipe.cmmi
url = http://sysoev.ru/nginx/nginx-0.9.4.tar.gz
strip-top-level-dir=true
configure-options = --with-debug
                --add-module=${upstream_fair:destination}
                --add-module=${headers_more_module:destination}
                --add-module=${echo_module:destination}
                --add-module=${eval_module:destination}
                --add-module=${nginx_devel_kit:destination}
                --add-module=${nginx_lua_module:destination}
                --with-cc-opt="-D NGX_HAVE_CASELESS_FILESYSTEM=0"
                --with-http_ssl_module
                --with-http_stub_status_module
                --http-proxy-temp-path=${buildout:directory}/var/nginx/cache/client_body_te mp
cache_size = 64
dep-lua_jit = ${lua_jit:location}
dep-lua-nginx = ${nginx_lua_module:url}
environment =
    LUA_LIB=${lua:location}/lib
    LUA_INC=${lua:location}/include/luajit-2.0

cheers, andi

On 17.06.2012, at 19:49, Mark Huang wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Andi Balke  
View profile  
 More options Jun 17 2012, 6:36 pm
From: Andi Balke <andi.ba...@googlemail.com>
Date: Mon, 18 Jun 2012 00:36:57 +0200
Local: Sun, Jun 17 2012 6:36 pm
Subject: Re: How would I do cross domain with Pyramid and Nginx

but thinking again about that: why don't you just put some information to a queue, e.g. rabbitmq, from app A and app B consumes this? its much easier :)

On 18.06.2012, at 00:23, Andi Balke wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Mark Huang  
View profile  
 More options Jul 4 2012, 1:18 am
From: Mark Huang <zhengha...@gmail.com>
Date: Tue, 3 Jul 2012 22:18:30 -0700 (PDT)
Local: Wed, Jul 4 2012 1:18 am
Subject: Re: How would I do cross domain with Pyramid and Nginx

Hi,

Thanks everyone for your inputs.  It was very useful.  Have been busy the
past couple of weeks (even on weekends....it sucks) and only got to
replying this week.  My solution to this problem was to simply make a curl
request from my backend to the other application.  

Thanks Andi for your explanation.  Yes, I do intend to use RabbitMQ
eventually once the load starts to increase, but for now, a curl is all I
need.  

I was just over-complicating things.

Regards,
Mark Huang


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »