Issue with Cloud Storage GCS API on local Development Server

[k...@form-runner.com]

I've encountered a problem using the Google Cloud Storage GCS Client API, running on the local development server.  I'm trying to write the bytes from a PDF file, and then read them back.  

The code appears to write the (local fake)GCS file ok: (1) There appears to be an appropriate entry in the Development Console, and (2) there's a physical file in ~war/WEB-APP/appengine-generated (details below).  However, when I attempt to read the bytes from the GCS file, it throws a FileNotFound exception when it attempts to get the metadata (filesize).

First, here's the core code, versions of GCS client read/write, with my debugging stmts left in for reference:

private void writeToFile(GcsFilename fullGcsFilename, byte[] content) 

throws IOException 

{

System.out.println("writeToFile:full="+fullGcsFilename.toString());

    GcsOutputChannel outputChannel =

  gcsService.createOrReplace(fullGcsFilename, GcsFileOptions.getDefaultInstance());

    outputChannel.write(ByteBuffer.wrap(content));

    outputChannel.close();

}

private byte[] readFromFile(GcsFilename fullGcsFilename) 

throws IOException 

{

System.out.println("readFromFile:full="+fullGcsFilename.toString());

    int fileSize = (int) gcsService.getMetadata(fullGcsFilename).getLength();   [*][Exception thrown here]

    ByteBuffer result = ByteBuffer.allocate(fileSize);

    GcsInputChannel readChannel = gcsService.openReadChannel(fullGcsFilename, 0);

    try {

      readChannel.read(result);

    } finally {

      readChannel.close();

    }

    return result.array();

}

Here's the debugging output (in/out filenames appear to be the same):

----

writeToFile:full=GcsFilename(formrunnerbucket-r7yh23nb2, FA/MasterFormStore-6649846324789248)

----

readFromFile:full=GcsFilename(formrunnerbucket-r7yh23nb2, FA/MasterFormStore-6649846324789248)

----

Here's the observed results:

IN ~war/WEB-APP/appengine-generated:

-rw-r--r--  1 ken  staff  1679407 Nov 19 09:19 encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4

This is the expected PDF file, which can be opened (with Preview on a Mac).

----

In the Development Console (http://localhost:8888/_ah/admin/datastore):

In __GsFileInfo__ (in the empty Namespace)

Key: ag5mb3JtcnVubmVyLWhyZHJ7CxIOX19Hc0ZpbGVJbmZvX18iZ2VuY29kZWRfZ3Nfa2V5OkwyZHpMMlp2Y20xeWRXNXVaWEppZFdOclpYUXRjamQ1YURJemJtSXlMMFpCTDAxaGMzUmxja1p2Y20xVGRHOXlaUzAyTmpRNU9EUTJNekkwTnpnNU1qUTQM

ID/Name: encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4

content_type: application/octet-stream

filename:  /gs/formrunnerbucket-r7yh23nb2/FA/MasterFormStore-6649846324789248

size: 1679407

[[The ID/Name appears to be identical to the filename appearing in ~war/WEB-APP/appengine-generated]]

---------------

In the Development Console (http://localhost:8888/_ah/admin/datastore), in Namespace 5629499534213120 (where everything ran):

In _ah_FakeCloudStorate_formrunnerbucket-r7yh23nb2:

Key: ag5mb3JtcnVubmVyLWhyZHJZCxIwX2FoX0Zha2VDbG91ZFN0b3JhZ2VfX2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyIiNGQS9NYXN0ZXJGb3JtU3RvcmUtNjY0OTg0NjMyNDc4OTI0OAyiARA1NjI5NDk5NTM0MjEzMTIw

ID/Name:  FA/MasterFormStore-6649846324789248

options: <Blob: 483 bytes>

---------------

Here's the exception thrown in readFromFile above, at:

int fileSize = (int) gcsService.getMetadata(fullGcsFilename).getLength();   [*][Exception thrown here]

java.lang.RuntimeException: java.io.FileNotFoundException

at com.formrunner.pdf.PDFMaker.makePDFfromFormInstance(PDFMaker.java:132)

.........

Caused by: java.io.FileNotFoundException

at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:614)

at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:593)

at com.google.appengine.api.files.FileServiceImpl.stat(FileServiceImpl.java:383)

at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.getObjectMetadata(LocalRawGcsService.java:215)

at com.google.appengine.tools.cloudstorage.GcsServiceImpl$2.run(GcsServiceImpl.java:73)

at com.google.appengine.tools.cloudstorage.GcsServiceImpl$2.run(GcsServiceImpl.java:70)

at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:93)

at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:138)

at com.google.appengine.tools.cloudstorage.GcsServiceImpl.getMetadata(GcsServiceImpl.java:70)

at com.formrunner.data.GoogleCloudStorage.readFromFile(GoogleCloudStorage.java:62)

at com.formrunner.data.GoogleCloudStorage.getBytesAt(GoogleCloudStorage.java:55)

at com.formrunner.db.MasterFormStore.getContent(MasterFormStore.java:92)

at com.formrunner.forms.FormUtils.getFormBlob(FormUtils.java:314)

at com.formrunner.pdf.PDFUtils.getPDFDocFromInstance(PDFUtils.java:50)

at com.formrunner.pdf.PDFMaker.makePDFfromFormInstance(PDFMaker.java:75)

It may be worth noting that the LocalExample from the GCS Client API runs fine, but does not leave any trace in the 

folder ~war/WEB-APP/appengine-generated nor in the Development Consolte (so far as I can find).

I hope someone can point out to me where I'm stumbling.

Thanks in advance,

Ken Bowen

[Tom Kaitchuck]

Using the 1.8.8 version of the SDK and depending on appengine-gcs-client 0.3.3, the following servlet based on your above example works:

public class GcsTest extends HttpServlet {

  private void writeToFile(GcsService gcsService, GcsFilename fullGcsFilename, byte[] content)

      throws IOException {

    System.out.println("writeToFile:full=" + fullGcsFilename.toString());

    GcsOutputChannel outputChannel =

        gcsService.createOrReplace(fullGcsFilename, GcsFileOptions.getDefaultInstance());

    outputChannel.write(ByteBuffer.wrap(content));

    outputChannel.close();

  }

  private byte[] readFromFile(GcsService gcsService, GcsFilename fullGcsFilename)

      throws IOException {

    System.out.println("readFromFile:full=" + fullGcsFilename.toString());

    int fileSize = (int) gcsService.getMetadata(fullGcsFilename).getLength();

    ByteBuffer result = ByteBuffer.allocate(fileSize);

    GcsInputChannel readChannel = gcsService.openReadChannel(fullGcsFilename, 0);

    try {

      readChannel.read(result);

    } finally {

      readChannel.close();

    }

    return result.array();

  }

  

  @Override

  public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {

    GcsService gcsService = GcsServiceFactory.createGcsService();

    GcsFilename fullGcsFilename = new GcsFilename("Foo", "Bar");

    byte[] content = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

    writeToFile(gcsService, fullGcsFilename, content);

    byte[] result = readFromFile(gcsService, fullGcsFilename);

    PrintWriter writer = resp.getWriter();

    writer.append(Arrays.toString(content));

    writer.append(Arrays.toString(result));

  }

}

I'm not really sure what could be different about your setup to cause that. Try creating a new project with the minimal possible dependencies and run the servlet above and see what happens.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.

[Ken Bowen]

Thanks Tom.
I went through pretty much the same servlet exercise a couple of hours before seeing your post this afternoon.
The problem was that I missed the point the the docs that the code /must/ run in a servlet.
(I'm not clear where it's stated, but I guessed from the comment about the test harness in LocalExample.)
I assumed that it would be ok to run in a deployed app on AppEngine.
Thanks again,
--Ken

> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/TJ1d7wqZngo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Tom Kaitchuck]

I'm a bit confused by your statement.

If you want to run in the DevAppserver or in AppEngine then you don't need to use the LocalServiceTestHelper at all. The LocalExample only does that as a demo of how to run it as a local executable or if you want to use it within Junit. 

If you don't include such a line, like in GcsExampleServlet then it should work as part of your deployed application.

[Ken Bowen]

Hi Tom,

Re:

> you don't need to use the LocalServiceTestHelper

Understood. I ran LocalExample once when I began, just to follow the documentation.

The original description I posted is from a running app on the DevServer, no TestHelper involved. In fact, the test jars are not even in the project.

After posting, I was reviewing LocalExample to make comparisons, and the phrase

"...run locally as opposed to in a deployed servlet"

caught my eye. So I wrote a version of what I posted as a servlet (grabs a 1.2MB PDF out of a resource, stores it with the GCS library, retrieves it, and writes out the number of bytes it got). That works fine.

If you say this doesn't /have/ to be directly in a servlet, I'll dig in further to my original.

Cheers,
--Ken

[Tom Kaitchuck]

Correct. It does not have to be directly in the servlet. For example App Engine MapReduce: https://code.google.com/p/appengine-mapreduce/
uses the GCS client to write out data, but it is many levels removed from a servlet.

[Ken Bowen]

Indeed. FYI & for anyone coming on this thread, below is running code (in the DevServer),
invoked from a test page via DWR, and returning the String result to be displayed in an alert;
the result is:
startBytesSize = 1160921
bytesReadSize = 1160921
Code:
public static String doGCSTest()
throws IOException
{
String result = "";
String testPDFFile = "IRAD1-2.pdf"; // a 1.2MB PDF file accessible as a resource
String testPDFPath = "samples/" + testPDFFile;
byte[] startBytes = CaptureWARSamples.getBytesFromWARFile(testPDFPath);
int startBytesSize = startBytes.length;
result += "startBytesSize = " + startBytesSize;

String BUCKETNAME = "formrunnerbucket-r7yh23nb2";
final GcsService gcsService =
GcsServiceFactory.createGcsService(new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build()) ;

String gcsFilename = "TEST-IRAD1-2pdf";
GcsFilename fullGcsFilename = new GcsFilename(BUCKETNAME, gcsFilename);

GcsOutputChannel outputChannel =
gcsService.createOrReplace(fullGcsFilename, GcsFileOptions.getDefaultInstance());

outputChannel.write(ByteBuffer.wrap(startBytes));
outputChannel.close();

int fileSize = (int) gcsService.getMetadata(fullGcsFilename).getLength();

ByteBuffer byteBufferResult = ByteBuffer.allocate(fileSize);

GcsInputChannel readChannel = gcsService.openReadChannel(fullGcsFilename, 0);
try {

readChannel.read(byteBufferResult);
} finally {
readChannel.close();
}
byte[] bytesRead = byteBufferResult.array();
int bytesReadSize = bytesRead.length;
result += "\nbytesReadSize = " + bytesReadSize;
return result;
}
--Ken

[k...@form-runner.com]

Nailed the culprit!

We make extensive use of namespaces, and in my original post, there turns out to be an unfortunate interaction between namespaces and the GCS client API.

Down in my original post, where I laid out all the data I could find in the Development Console, you will see that there are entries in the empty namespace and a different namespace:

In __GsFileInfo__ (in the empty Namespace)

….blah … blah

In Namespace 5629499534213120 (where everything ran):

In _ah_FakeCloudStorate_formrunnerbucket-r7yh23nb2:

…. blah…. blah

The code for writing into GCS ran in namespace 5629499534213120, but it wrote part of it's data into the empty namespace.

The (successful) experiment was to ensure that all the GCS API code (read and write) runs in the empty namespace, conceptually thus:

private byte[] readFromFile(GcsFilename fullGcsFilename)  throws IOException 

{

String in_ns = NamespaceManager.get();

NamespaceManager.set(null);

int fileSize = (int) gcsService.getMetadata(fullGcsFilename).getLength();

ByteBuffer result = ByteBuffer.allocate(fileSize);

GcsInputChannel readChannel = gcsService.openReadChannel(fullGcsFilename, 0);

try {

     readChannel.read(result);

} finally {

     readChannel.close();

}

byte[] toreturn=result.array();

NamespaceManager.set(in_ns);

return toreturn;

}

Now the FileNotFound does not occur, and I do get (some) bytes back, about 1/10 of the 1.2MB.  Presumably when I switch to ObjectStreaming, I'll get all.

Cheers,

--Ken

[Troy High]

I was having the exact same problem with namespaces and the local dev server not honoring them for some entries. I applied the same workaround as Ken and am able to retrieve files programmatically.

So is this a bug in the Dev server implementation or does this mean cloud storage does not officially support namespaces?  I haven't pushed this code to the google servers yet so I am not sure if I would encounter the same issue without the workaround.

Thanks,

Troy

[Tom Kaitchuck]

Namespaces are a datastore specific concept. It caused trouble in this instance because the local mock of GCS uses the datastore for its metadata. Real GCS does not depend on datastore, so this problem will not occur with deployed code. 

--

[Gary Mort]

On Tuesday, November 19, 2013 3:57:09 PM UTC-5, k...@form-runner.com wrote:

I've encountered a problem using the Google Cloud Storage GCS Client API, running on the local development server.  I'm trying to write the bytes from a PDF file, and then read them back.  

The code appears to write the (local fake)GCS file ok: (1) There appears to be an appropriate entry in the Development Console, and (2) there's a physical file in ~war/WEB-APP/appengine-generated (details below).  However, when I attempt to read the bytes from the GCS file, it throws a FileNotFound exception when it attempts to get the metadata (filesize).\\

I have the same problem with 1.8.8 using PHP.

I can go to http://localhost:8000 and see that there are a bunch of datastore items being created from the PHP calls to save data to a file[and I can even see that the filenames in datastore match the ones that should have been created].  However, any subsequent attempts to stat those files results in a file not found error[ie check the filesize, check to see if the file exists, etc].

[Ken Bowen]

Gary,

Are you using namespaces and running your code in a non-default namespace? That was my problem (see the very end of my original post.) My revised code for 11/21 runs in the default namespace -- that's why it worked.

Ken

> --
> You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/TJ1d7wqZngo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Gary Mort]

On Saturday, December 21, 2013 12:52:13 PM UTC-5, Ken Bowen wrote:

Gary,

Are you using namespaces and running your code in a non-default namespace?  That was my problem (see the very end of my original post.)  My revised  code for 11/21 runs in the default namespace -- that's why it worked.

PHP doesn't support namespaces yet - so I suspect the problem has something to do with the way GS storage is emulated on the SDK and the pseodo datastore interface for PHP.  As a workaround, I added a special stream to read/write from so on my dev server it can use files while on the GAE server it uses google storage.