Google Cloud Storage libraries interaction web2py appengine

[Jacinto Parga]

Hello, 

I am trying to manage (upload and download) big files from google cloud storage width a web2py app deployed in GAE.

First step, I have adapted the example: https://cloud.google.com/storage/docs/json_api/v1/json-api-python-samples

I works fine in localhost. Here it is the controller: 

import argparse

import httplib2

import os

import sys

import json

from apiclient import discovery

from oauth2client import file

from oauth2client import client

from oauth2client import tools

# Define sample variables.

_BUCKET_NAME = 'mybucket'

_API_VERSION = 'v1'

# Parser for command-line arguments.

parser = argparse.ArgumentParser(

    description=__doc__,

    formatter_class=argparse.RawDescriptionHelpFormatter,

    parents=[tools.argparser])

# CLIENT_SECRETS is name of a file containing the OAuth 2.0 information for this

# application, including client_id and client_secret. You can see the Client ID

# and Client secret on the APIs page in the Cloud Console:

# <https://console.developers.google.com/>

CLIENT_SECRETS = os.path.join(os.path.dirname(__file__), 'client_secrets.json')

# Set up a Flow object to be used for authentication.

# Add one or more of the following scopes. PLEASE ONLY ADD THE SCOPES YOU

# NEED. For more information on using scopes please see

# <https://developers.google.com/storage/docs/authentication#oauth>.

FLOW = client.flow_from_clientsecrets(CLIENT_SECRETS,

  scope=[

      'https://www.googleapis.com/auth/devstorage.full_control',

      'https://www.googleapis.com/auth/devstorage.read_only',

      'https://www.googleapis.com/auth/devstorage.read_write',

    ],

    message=tools.message_if_missing(CLIENT_SECRETS))

def index():

  cliente=CLIENT_SECRETS

  flow= FLOW

#    va= main(sys.argv)

   # Parse the command-line flags.

#  flags = parser.parse_args(argv[1:])

    # If the credentials don't exist or are invalid run through the native client

  # flow. The Storage object will ensure that if successful the good

  # credentials will get written back to the file.

  storage = file.Storage('sample.dat')

  credentials = storage.get()

  if credentials is None or credentials.invalid:

    credentials = tools.run_flow(FLOW, storage, flags)

  # Create an httplib2.Http object to handle our HTTP requests and authorize it

  # with our good Credentials.

  http = httplib2.Http()

  http = credentials.authorize(http)

  # Construct the service object for the interacting with the Cloud Storage API.

  service = discovery.build('storage', _API_VERSION, http=http)

  try:

    req = service.buckets().get(bucket=_BUCKET_NAME)

    resp = req.execute()

    print1= json.dumps(resp, indent=2)

   

    fields_to_return = 'nextPageToken,items(name,size,contentType,metadata(my-key))'

    req = service.objects().list(bucket=_BUCKET_NAME, fields=fields_to_return)

    # If you have too many items to list in one request, list_next() will

    # automatically handle paging with the pageToken.

    while req is not None:

      resp = req.execute()

      print2= json.dumps(resp, indent=2)

      req = service.objects().list_next(req, resp)

  except client.AccessTokenRefreshError:

    aviso= "The credentials have been revoked or expired, please re-run the application to re-authorize"

    

  form=SQLFORM(db.gfile)

  return dict(print1=print1,print2=print2, form=form)

 I get the result expected.

But when I deploy it to the google app engine, there rises an error ticket: 

14:49:26.005

Unable to store in FILE: /base/data/home/apps/s~merebafs/2.381697639759293929/applications/MRBFILE/controllers/default.py Traceback (most recent call last): File "/base/data/home/apps/s~merebafs/2.381697639759293929/gluon/restricted.py", line 224, in restricted exec ccode in environment File "/base/data/home/apps/s~merebafs/2.381697639759293929/applications/MRBFILE/controllers/default.py", line 12, in <module> import httplib2 File "/base/data/home/apps/s~merebafs/2.381697639759293929/gluon/custom_import.py", line 86, in custom_importer raise ImportError, 'Cannot import module %s' % str(e) ImportError: Cannot import module 'httplib2'

And I can't understand because it is a library they use in their example. Any idea...

The next thing is to achieve functions to upload and download files from the bucket I accessed to. 

Thanks

[Massimo Di Pierro]

GAE does not provide httplib2

they want you to use urlfetch (their api). web2py provides gluon.tools.fetch which wraps urlfetch and works like urllib.urlopen when not on GAE.

[Jacinto Parga]

Thanks, 

I'm just looking for a simple way to upload and download files from google cloud store, but it seems so tricky to me.

[Jacinto Parga]

Finally solved.

The solution: https://groups.google.com/forum/#!topic/google-appengine-stackoverflow/JZX2Yh67ylI

I had to include: apiclient, gflags, httplib2, oauth2client, uritemplate in site-packages folder

Thanks

El jueves, 22 de enero de 2015, 23:59:11 (UTC+1), Massimo Di Pierro escribió: