Accessing the typhoonae's datastore through remote_api
37 views
Skip to first unread message
Maarten Siersema
Jun 8, 2011, 9:49:51 AM6/8/11
to typhoonae
Hi,
In the production roadmap I saw that there's support for the app
engine remote_api, but I can't seem to find any documentation on how
to use it within Typhoonae.
In some of our backend services we currently use the remote_api to
retrieve data from the app engine datastore. This way we can easily
access and manipulate the data which is stored on Google. We use the
remote_api_stub module from the google.appengine.ext.remote_api
package as followed:
from google.appengine.ext.remote_api import remote_api_stub
remote_api_stub.ConfigureRemoteDatastore('gae-app-id', '/
remote_api', ('username', 'password'), 'gae-app-id.appspot.com')
For typhoonae we want to use the same interface so we don't need to
change any of our backend code to communicate with typhoonae's Mongo
database. We've added the remote_api to the buildin section of the
app.yaml like:
builtins:
- remote_api: on
This creates the following handler:
handlers:
- login: admin
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
secure: optional
url: /remote_api
When we try to access the remote api through the registered handler at
http://app-id.ourapp.com/remote_api we see the following response:
HTTPError: HTTP Error 302: Requires login
If I go to the same url in my browser I see a redirect page with the
text "You're logged in as admin@typhoonae! This is a demo login
handler. Continue". When I click continue a typhoonae_login cookie is
set. After a refresh I see "This request did not contain a necessary
header".
Is there a way that we can skip this login procedure so that it
corresponds with the app engine code? I've manually removed the login:
admin section from the app.yaml but that doesn't seem to be working. I
also can't find any details on the credentials (admin@typhoonae??)
which are needed for the ConfigureRemoteDatastore method. Does anybody
know how to get the remote_api to work for typhoonae's database?
regards,
Maarten
In the production roadmap I saw that there's support for the app
engine remote_api, but I can't seem to find any documentation on how
to use it within Typhoonae.
In some of our backend services we currently use the remote_api to
retrieve data from the app engine datastore. This way we can easily
access and manipulate the data which is stored on Google. We use the
remote_api_stub module from the google.appengine.ext.remote_api
package as followed:
from google.appengine.ext.remote_api import remote_api_stub
remote_api_stub.ConfigureRemoteDatastore('gae-app-id', '/
remote_api', ('username', 'password'), 'gae-app-id.appspot.com')
For typhoonae we want to use the same interface so we don't need to
change any of our backend code to communicate with typhoonae's Mongo
database. We've added the remote_api to the buildin section of the
app.yaml like:
builtins:
- remote_api: on
This creates the following handler:
handlers:
- login: admin
script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
secure: optional
url: /remote_api
When we try to access the remote api through the registered handler at
http://app-id.ourapp.com/remote_api we see the following response:
HTTPError: HTTP Error 302: Requires login
If I go to the same url in my browser I see a redirect page with the
text "You're logged in as admin@typhoonae! This is a demo login
handler. Continue". When I click continue a typhoonae_login cookie is
set. After a refresh I see "This request did not contain a necessary
header".
Is there a way that we can skip this login procedure so that it
corresponds with the app engine code? I've manually removed the login:
admin section from the app.yaml but that doesn't seem to be working. I
also can't find any details on the credentials (admin@typhoonae??)
which are needed for the ConfigureRemoteDatastore method. Does anybody
know how to get the remote_api to work for typhoonae's database?
regards,
Maarten
Tobias
Jun 8, 2011, 10:49:00 AM6/8/11
to typhoonae
Hi Maarten - Thanks a lot for giving TyphoonAE a go!
We have limited support for the Remote API. Due to the fact that we
only provide a dummy login handler out of the box, accessing the
Remote API only works from the same host where your application is
served. A common workaround for that is tunneling through ssh.
Furthermore you have to remove the login: admin option from the
configuration in the handlers section of your app.yaml file.
BTW, are you using the 0.2.0 buildout download or have you installed
TyphoonAE directly from the mercurial repository?
Please don't hesitate to ask if you face any problems with TyphoonAE!
Cheers,
Tobias
> When we try to access the remote api through the registered handler athttp://app-id.ourapp.com/remote_apiwe see the following response:
We have limited support for the Remote API. Due to the fact that we
only provide a dummy login handler out of the box, accessing the
Remote API only works from the same host where your application is
served. A common workaround for that is tunneling through ssh.
Furthermore you have to remove the login: admin option from the
configuration in the handlers section of your app.yaml file.
BTW, are you using the 0.2.0 buildout download or have you installed
TyphoonAE directly from the mercurial repository?
Please don't hesitate to ask if you face any problems with TyphoonAE!
Cheers,
Tobias
Maarten Siersema
Jun 8, 2011, 11:44:14 AM6/8/11
to typhoonae
Hi Tobias,
Thank you for the fast reply.
At the moment I'm using the Amazon AMI with an update on and a
buildout from the mercurial repository.
I guess a ssh tunnel would work, but how would I connect from a local
system to remote_api? Is tunneling port 80 and creating a local host
entry sufficient?
KR,
Maarten
> > When we try to access the remote api through the registered handler athttp://app-id.ourapp.com/remote_apiwesee the following response:
Thank you for the fast reply.
At the moment I'm using the Amazon AMI with an update on and a
buildout from the mercurial repository.
I guess a ssh tunnel would work, but how would I connect from a local
system to remote_api? Is tunneling port 80 and creating a local host
entry sufficient?
KR,
Maarten
Tobias
Jun 8, 2011, 2:39:02 PM6/8/11
to typhoonae
Hey Maarten,
Have you already tried something like this?
$ ssh -i <keypair> -L 8080:localhost:80 username@ec2-xxx-xxx-xxx-
xx.compute-1.amazonaws.com
Then you should be able to connect to the Remote API, for instance by
using the remote_api_shell.py program from the SDK as follows:
$ remote_api_shell.py -s localhost:8080 -p /remote_api [app-id]
BTW, slightly different use-case, but http://hybrid.typhoonae.org
directly connects to the GAE Datastore through the Remote API while
the app runs on TyphoonAE (on a XEN instance in Germany);
http://typhoonae-hybrid.appspot.com forms the counterpart.
And please let me know if you have any further issues! It always helps
to improve TyphoonAE.
Cheers,
Tobias
> > > When we try to access the remote api through the registered handler athttp://app-id.ourapp.com/remote_apiweseethe following response:
Have you already tried something like this?
$ ssh -i <keypair> -L 8080:localhost:80 username@ec2-xxx-xxx-xxx-
xx.compute-1.amazonaws.com
Then you should be able to connect to the Remote API, for instance by
using the remote_api_shell.py program from the SDK as follows:
$ remote_api_shell.py -s localhost:8080 -p /remote_api [app-id]
BTW, slightly different use-case, but http://hybrid.typhoonae.org
directly connects to the GAE Datastore through the Remote API while
the app runs on TyphoonAE (on a XEN instance in Germany);
http://typhoonae-hybrid.appspot.com forms the counterpart.
And please let me know if you have any further issues! It always helps
to improve TyphoonAE.
Cheers,
Tobias
Maarten Siersema
Jun 10, 2011, 8:38:17 AM6/10/11
to typhoonae
Hi Tobias,
I've just tried tunneling to the Amazon image, but it looks like the
remote_api_shell redirects to app engine instead of the typhoonae
deployment. Even when I try to access the remote_api locally from the
same image on which it's deployed it keeps requesting for app engine
credentials. When I enter bogus information like 'admin' for both
email and password I even get a captcha from Google. I've removed the
"login: admin" setting from the app.yaml like you said.
When I tunnel i can access my app through something like http://apphostname.dev:8080
when I altered my hosts file. So when I try to access the remote_api I
do something like this:
$ remote_api_shell.py -s apphostname:8080 -p /_ah/remote_api app_id
which results in a 401 after asking credentials for email and
password.
urllib2.HTTPError: HTTP Error 401: Unauthorized
The app is btw accessible on Amazon and working fine. I can even
access remote_api url through: http://apphostname.dev:8080/_ah/remote_api
which results in the following response without the dummy login
procedure we saw earlier:
This request did not contain a necessary header
So it looks like the remote_api doesn't work like expected. It only
redirects to the app engine datastore and not to the typhoonae's own
datastore.
KR,
Maarten
On Jun 8, 8:39 pm, Tobias <tobias.rodae...@googlemail.com> wrote:
> Hey Maarten,
>
> Have you already tried something like this?
>
> $ ssh -i <keypair> -L 8080:localhost:80 username@ec2-xxx-xxx-xxx-
> xx.compute-1.amazonaws.com
>
> Then you should be able to connect to the Remote API, for instance by
> using the remote_api_shell.py program from the SDK as follows:
>
> $ remote_api_shell.py -s localhost:8080 -p /remote_api [app-id]
>
> BTW, slightly different use-case, buthttp://hybrid.typhoonae.org
> > > > When we try to access the remote api through the registered handler athttp://app-id.ourapp.com/remote_apiweseethefollowing response:
I've just tried tunneling to the Amazon image, but it looks like the
remote_api_shell redirects to app engine instead of the typhoonae
deployment. Even when I try to access the remote_api locally from the
same image on which it's deployed it keeps requesting for app engine
credentials. When I enter bogus information like 'admin' for both
email and password I even get a captcha from Google. I've removed the
"login: admin" setting from the app.yaml like you said.
When I tunnel i can access my app through something like http://apphostname.dev:8080
when I altered my hosts file. So when I try to access the remote_api I
do something like this:
$ remote_api_shell.py -s apphostname:8080 -p /_ah/remote_api app_id
which results in a 401 after asking credentials for email and
password.
urllib2.HTTPError: HTTP Error 401: Unauthorized
The app is btw accessible on Amazon and working fine. I can even
access remote_api url through: http://apphostname.dev:8080/_ah/remote_api
which results in the following response without the dummy login
procedure we saw earlier:
This request did not contain a necessary header
redirects to the app engine datastore and not to the typhoonae's own
datastore.
KR,
Maarten
On Jun 8, 8:39 pm, Tobias <tobias.rodae...@googlemail.com> wrote:
> Hey Maarten,
>
> Have you already tried something like this?
>
> $ ssh -i <keypair> -L 8080:localhost:80 username@ec2-xxx-xxx-xxx-
> xx.compute-1.amazonaws.com
>
> Then you should be able to connect to the Remote API, for instance by
> using the remote_api_shell.py program from the SDK as follows:
>
> $ remote_api_shell.py -s localhost:8080 -p /remote_api [app-id]
>
> directly connects to the GAE Datastore through the Remote API while
> the app runs on TyphoonAE (on a XEN instance in Germany);http://typhoonae-hybrid.appspot.comforms the counterpart.
Tobias
Jun 10, 2011, 10:14:40 AM6/10/11
to typhoonae
Hey Maarten,
Given that you've tunneled the remote HTTP port to localhost:8080 use -
s localhost:8080 instead. The main reason for this workaround is in
the google.appengine.tools.remote_rpc.AbstractRpcServer implementation
http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/appengine_rpc.py#162
where the authentication method depends on whether the server address
is localhost or some Google (hosted) domain.
Hope this helps,
Tobias
PS: You're also very welcome to join #typhoonae on irc.freenode.net
for discussing issues and ideas around TAE.
> > the app runs on TyphoonAE (on a XEN instance in Germany);http://typhoonae-hybrid.appspot.comformsthe counterpart.
Given that you've tunneled the remote HTTP port to localhost:8080 use -
s localhost:8080 instead. The main reason for this workaround is in
the google.appengine.tools.remote_rpc.AbstractRpcServer implementation
http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/tools/appengine_rpc.py#162
where the authentication method depends on whether the server address
is localhost or some Google (hosted) domain.
Hope this helps,
Tobias
PS: You're also very welcome to join #typhoonae on irc.freenode.net
for discussing issues and ideas around TAE.
> I've just tried tunneling to the Amazon image, but it looks like the
> remote_api_shell redirects to app engine instead of the typhoonae
> deployment. Even when I try to access the remote_api locally from the
> same image on which it's deployed it keeps requesting for app engine
> credentials. When I enter bogus information like 'admin' for both
> email and password I even get a captcha from Google. I've removed the
> "login: admin" setting from the app.yaml like you said.
>
> When I tunnel i can access my app through something likehttp://apphostname.dev:8080
> remote_api_shell redirects to app engine instead of the typhoonae
> deployment. Even when I try to access the remote_api locally from the
> same image on which it's deployed it keeps requesting for app engine
> credentials. When I enter bogus information like 'admin' for both
> email and password I even get a captcha from Google. I've removed the
> "login: admin" setting from the app.yaml like you said.
>
Maarten Siersema
Jun 14, 2011, 8:37:48 AM6/14/11
to typhoonae
Hey Tobias,
That doesn't seem to be working too :( When I tunnel to 8080 and try
make a connection through the remote_api_shell.py like
$ remote_api_shell.py -s localhost:8080 -p /_ah/remote_api kalydo-
console-v4
I'm still getting 'connection refused' after getting prompted for an
email and password. After entering bogus information or just hitting
enter two times I see the message 'connection refused' with a stack
trace ending up in:
File "C:\Python25\lib\urllib2.py", line 1082, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error (10061, 'Connection refused')>
My app.yaml after deployment:
api_version: '1'
application: kalydo-console-v4
builtins:
- remote_api: 'on'
- default: 'on'
derived_file_type:
- python_precompiled
handlers:
- script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
secure: optional
url: /_ah/remote_api(/.*)?
I'll join the irc btw.. It's a bit easier to communicate through :)
regards,
Maarten
That doesn't seem to be working too :( When I tunnel to 8080 and try
make a connection through the remote_api_shell.py like
$ remote_api_shell.py -s localhost:8080 -p /_ah/remote_api kalydo-
console-v4
I'm still getting 'connection refused' after getting prompted for an
email and password. After entering bogus information or just hitting
enter two times I see the message 'connection refused' with a stack
trace ending up in:
File "C:\Python25\lib\urllib2.py", line 1082, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error (10061, 'Connection refused')>
My app.yaml after deployment:
api_version: '1'
application: kalydo-console-v4
builtins:
- remote_api: 'on'
- default: 'on'
derived_file_type:
- python_precompiled
handlers:
- script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
secure: optional
url: /_ah/remote_api(/.*)?
I'll join the irc btw.. It's a bit easier to communicate through :)
regards,
Maarten
> Given that you've tunneled the remote HTTP port to localhost:8080 use -
> s localhost:8080 instead. The main reason for this workaround is in
> the google.appengine.tools.remote_rpc.AbstractRpcServer implementationhttp://code.google.com/p/googleappengine/source/browse/trunk/python/g...
> s localhost:8080 instead. The main reason for this workaround is in
Maarten Siersema
Jun 14, 2011, 10:31:26 AM6/14/11
to typhoonae
ok, I just managed to access the remote_api from the server side. The
shell command line interface was slightly different from the one I was
running on my own machine. I also needed to modify the nginx
configuration file to include localhost as the server_name. After that
I'm able to access the shell through:
/home/typhoonae/typhoonae/parts/google_appengine/remote_api_shell.py -
s localhost:8080 kalydo_console_v4 /_ah/remote_api
Tunneling to my local machine isn't working yet. But this is a
start :)
shell command line interface was slightly different from the one I was
running on my own machine. I also needed to modify the nginx
configuration file to include localhost as the server_name. After that
I'm able to access the shell through:
/home/typhoonae/typhoonae/parts/google_appengine/remote_api_shell.py -
s localhost:8080 kalydo_console_v4 /_ah/remote_api
Tunneling to my local machine isn't working yet. But this is a
start :)
Maarten Siersema
Jun 14, 2011, 11:08:37 AM6/14/11
to typhoonae
ok local is running for me as well :) just a really stupid typo..
Thx for the support anyway!! :)
Thx for the support anyway!! :)
Reply all
Reply to author
Forward
0 new messages