Unable to write > 1200 files to Google App Engine Java with GCS in development server

[Zensursula via StackOverflow]

Writing files to google cloud storage in production environment of GAE runs fine. But when I try to import my productive data (6-7GB) into the development server I always got errors, when I try to write more than ~1200 files. The first 1200 files work fine, but then I always got exceptions. In that case I need to delete some files, and afterwards I can successfully write other files to GCS.

My Testenvironment is:

• appengine-gcs-client 0.4.2.
• appengine-java-sdk-1.9.13
• Win 7 Prof (x64), NTFS, Eclipse Kepler

And here is the code I use:

byte[] content = ..
GcsFilename fn = new GcsFilename("cms-gi-pdf", "zip-dwdwos--Nwrg1aTk7DSRn9gAWnT");
GcsFileOptions options = new GcsFileOptions.Builder().mimeType("binary/octet-stream").acl("project-private").build();
GcsOutputChannel outputChannel = gcsService.createOrReplace(fn, options);
outputChannel.write(ByteBuffer.wrap(content));
outputChannel.close();

The exception happens on the last line (close()). Here is the stacktrace:

15:43:40 WARNING com.google.appengine.api.files.dev.FileMetadata:setFinalized:Caught IOException while attempting to write blob 15:43:40 INFO com.google.appengine.tools.cloudstorage.RetryHelper:doRetry:RetryHelper(11,88 ms, 1 attempts, com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1@6abbeea): Attempt #1 failed [java.io.IOException], sleeping for 281 ms

...

com.google.appengine.tools.cloudstorage.RetriesExhaustedException: RetryHelper(8,866 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1@6abbeea): Too many failures, giving up at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:127) at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166) at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:156) at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl.close(GcsOutputChannelImpl.java:198) at de.daysout.generic.filesystem.gcs.GoogleCloudStorage.write(GoogleCloudStorage.java:45)

...

Caused by: com.google.appengine.api.files.FinalizationException at com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:617) at com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:594) at com.google.appengine.api.files.FileServiceImpl.open(FileServiceImpl.java:527) at com.google.appengine.api.files.FileServiceImpl.openForAppend(FileServiceImpl.java:471) at com.google.appengine.api.files.FileServiceImpl.openForAppend(FileServiceImpl.java:190) at com.google.appengine.api.files.FileServiceImpl.openWriteChannel(FileServiceImpl.java:171) at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.append(LocalRawGcsService.java:181) at com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.finishObjectCreation(LocalRawGcsService.java:245) at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.call(GcsOutputChannelImpl.java:201) at com.google.appengine.tools.cloudstorage.GcsOutputChannelImpl$1.call(GcsOutputChannelImpl.java:198) at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:108)

... 72 more Caused by:

com.google.apphosting.api.ApiProxy$ApplicationException: ApplicationError: 101: /gs/writable:oOUcuZuodalavuikpIHUFA?name=cms-gi-pdf%2Fzip-dwdwos--Nwrg1aTk7DSRn9gAWnT is already finalized at com.google.appengine.api.files.dev.LocalFileService.throwError(LocalFileService.java:201) at com.google.appengine.api.files.dev.LocalFileService.open(LocalFileService.java:299) ...

This is reproducable. Until yesterday I used appengine-gcs-client 0.2. But I had the same problem with that version. This is annoying as I am not able to test my software with production data.

And here are my questions:

• Is this an error or are there any limitations in the devlopment server?
• Has someone used appengine-gcs-client in dev server with more than 1200 files?

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/26342264/unable-to-write-1200-files-to-google-app-engine-java-with-gcs-in-development-s