blobstore APIs and File APIs

[Jean-Remi Malzieu]

Hello,

I am currently looking at the Appscale system in order to build Virtual machine (as a first step) embedding my application.

I installed the appscale system from source, and everything worked like a charm. My application has been deployed successfully, and I can navigate through it without issues.

However, there are at least three functionality I have trouble to use using appscale:

 1 . Remote API. I have written a homemade backup and recovery solution (I didn't spend time to look at appscale one so far) that uses the remote API to fetch entities and store them in a json format. I tried to follow https://github.com/AppScale/appscale/wiki/Viewing-Data-in-AppScale-%28remote_api%29 but I am stuck and I get rejected whenever I try to connect the RemoteAPI using the following code:

        RemoteApiOptions options = new RemoteApiOptions()
        .server(anURL, aPort)
        .credentials(anUserName, aPassword);
   
        RemoteApiInstaller installer = new RemoteApiInstaller();
        installer.install(options);

Is there anything special I could do to make the remote API working normally?

 2. This second point is more problematic as a first step. I am trying to make the GCS and blobstore service working.

On upload, I generate an URL using the following code :

                 UploadOptions theOptions = UploadOptions.Builder.withGoogleStorageBucketName("BUCKET");

        String theString = BlobstoreServiceFactory.getBlobstoreService().createUploadUrl("/Upload?Type=test");

Using this, I am supposed to received a call to my servlet named Upload, with the informations of the file that has been uploaded. The call is correctly made, but whenever I call the following API : 

            Map<String, List<BlobKey>> theBlobs =  BlobstoreServiceFactory.getBlobstoreService().getUploads(aReq);

   Map<String, List<FileInfo>> theStringInfo = BlobstoreServiceFactory.getBlobstoreService().getFileInfos(aReq);

I get an exception :

java.lang.IllegalStateException: Must be called from a blob upload callback request.

at com.google.appengine.api.blobstore.BlobstoreServiceImpl.getFileInfos(Unknown Source)

3. Moreover, when I try to create a file over the GCS service, it seems the application is trying to connect to the google cloud storage (I was thinking the support of cloud storage would be locally on the VM). I then get an error :

java.lang.RuntimeException: Server replied with 401, probably bad authentication: Request: POST 

at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:87)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:129)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:123)
at com.google.appengine.tools.cloudstorage.GcsServiceImpl.createOrReplace(GcsServiceImpl.java:64)

For this last step, I think I will have to write some code to handle the file write/read in case I am not located on app engine. Indeed, I do not want the VM to use Cloud storage if it can handle the file by itself. However, it would be great if I can keep the exact same code of the application, but only modify the appscale system.

Anyway, thank you for the great system.

Cheers,

[Navraj Chohan]

Hey Jean-Remi,

For the remote API are you using the jar supplied by AppScale?

AppScale does not support GCS right now, nor supply an on-premise solution. You'll have to use the GCS stand alone library instead.

https://cloud.google.com/storage/docs/json_api/v1/libraries

Some of the API/service libraries are meant to work seamlessly when running in GAE (not having to worry about auth). This may require modification of the client if it happens to think its running in GAE when in AppScale.

Thanks,

Raj 

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.
To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

[Jean-Remi Malzieu]

Thanks for your feedback.

I indeed was using the jar supplied by AppScale, but didn't dig too far to understand what went wrong. I will give it another try.

For the GCS library, I have no requirements to use cloud storage. My only use of GCS when I deploy to app engine is to simulate (very simply) a file system, so that I can upload, write and serve files.

Thus for my application, I would only need to re-write all file related class (4~5) in order to directly map them on io class. I was thinking that there might be a way that app scale directly map this behind the scene (maybe I can implement this way myself, but I would need some directions). Wouldn't it be a great feature to be able to upload, download, write, read files as if we were on GCS but on a totally private cloud? 

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Navraj Chohan]

I'd love to have a GCS equivalent in AppScale. The basic API isn't that complicated from what I gather. Any contributions to the platform are gladly welcome, and I'm happy to help in whatever way if you would like to have a go at implementing it.

-Raj

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

--

You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Jean-Remi Malzieu]

Well I dig further for the implementation of the gcs client. I am currently using the jar app-engine-gcs.jar . After having looked at the source code, they made the implementation of the dev server pretty easy, and modifying two classes may allow to write a custom implementation on the AppScale solution.
The only small issue is that the dev server is compatible only starting from app engine 1.8.1 while I think AppScale is only supporting the 1.8.0 ( this should not be a big deal)

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_community+unsub...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

[Jean-Remi Malzieu]

I am facing a lot of issues while trying to use the remote API. 

I am using the jar provided by appscale. At first, I had issues about the SSL certificate not being valid for my server. I spent quite a lot of time to understand how to make everything work. After I add the certifcate, I have now the following issue : 

java.io.IOException: can't get appId from remote api; status code = 302, body: 

at com.google.appengine.tools.remoteapi.RemoteApiInstaller.getAppIdFromServer(Unknown Source)

at com.google.appengine.tools.remoteapi.RemoteApiInstaller.loginImpl(Unknown Source)

at com.google.appengine.tools.remoteapi.RemoteApiInstaller.login(Unknown Source)

at com.google.appengine.tools.remoteapi.RemoteApiInstaller.install(Unknown Source

I have spent the whole day trying to figure out what I am missing... 

In my web.xml, I have the following as described in the documentation :

        <servlet>

    <display-name>Remote API Servlet</display-name>

     <servlet-name>RemoteApiServlet</servlet-name>

     <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>

     <load-on-startup>1</load-on-startup>

</servlet>

<servlet-mapping>

    <servlet-name>RemoteApiServlet</servlet-name>

    <url-pattern>/remote_api</url-pattern>

</servlet-mapping>

 <filter>

    <filter-name>_ah_DevSocketFilter</filter-name>

    <filter-class>

      com.google.appengine.api.socket.dev.DevSocketFilter

    </filter-class>

    <init-param>

      <param-name>use-native-sockets</param-name>

      <param-value>true</param-value>

    </init-param>

  </filter>

  <filter-mapping>

    <filter-name>_ah_DevSocketFilter</filter-name>

    <url-pattern>/*</url-pattern>

  </filter-mapping>

Any help on this would be welcome...

[Navraj Chohan]

Can you try using the python remote shell and see if you can connect to your application? Also, if you can send me a stripped down version of your application that is using the remote API, I can try it on my end and see if I can debug it.

Thanks,

Raj

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.

--

Navraj Chohan (Raj), PhD
Co-Founder of AppScale Systems
A Google Cloud Partner,

Gartner 2014 Cool Vendor, 
and a GigaOm Structure/LaunchPad "Best of the Best" Startups

615 State Street
Santa Barbara, CA 93101
r...@appscale.com
http://navrajchohan.com

--
You received this message because you are subscribed to the Google Groups "AppScale Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to appscale_commun...@googlegroups.com.

To post to this group, send email to appscale_...@googlegroups.com.
Visit this group at http://groups.google.com/group/appscale_community.
For more options, visit https://groups.google.com/d/optout.