Decoding a file created with the filesystem API

[Kayode Odeyemi]

Hi everyone,

I have images uploaded to the blobstore using filesystem API. The files have been uploaded successfully and I can see the blobs in the respective directory.

But then, how do I decode these files such that when sent back to the client or retrieved using the blobstore API, I'm able to get an image? Right now /blobstore/writable:xxx is not useful for me because when I wrap the path in <img src="/blobstore/writable:xxx" />, it doesn't give me an image.

Any help is appreciated.

Thanks

--
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde blog: http://sinati.com/tree/java-cheat-sheet

[Jeff Schnitzer]

Look at the documentation for the images service.  There is a method that gives you a serving url directly to the CDN.

Warning:  This method has a surprising amount of latency, you probably want to cache the value in memcache for a reasonable duration.

Jeff

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Kayode Odeyemi]

On Thu, Mar 1, 2012 at 3:03 PM, Jeff Schnitzer <je...@infohazard.org> wrote:

Look at the documentation for the images service.  There is a method that gives you a serving url directly to the CDN.

Warning:  This method has a surprising amount of latency, you probably want to cache the value in memcache for a reasonable duration.

Can the Image Service work with the filesystem where the blob is stored? Everytime I try to retrieve it using the method you mentioned getServingUrl(blobkey), I get the error:

[java] java.lang.IllegalArgumentException: Failed to read image
[java]     at com.google.appengine.api.images.ImagesServiceImpl.getServingUrl(ImagesServiceImpl.java:222)

Also, I'm think this action is not possible on the dev server.

On Thu, Mar 1, 2012 at 7:42 AM, Kayode Odeyemi <dre...@gmail.com> wrote:

Hi everyone,

I have images uploaded to the blobstore using filesystem API. The files have been uploaded successfully and I can see the blobs in the respective directory.

But then, how do I decode these files such that when sent back to the client or retrieved using the blobstore API, I'm able to get an image? Right now /blobstore/writable:xxx is not useful for me because when I wrap the path in <img src="/blobstore/writable:xxx" />, it doesn't give me an image.

Any help is appreciated.

Thanks

--
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde blog: http://sinati.com/tree/java-cheat-sheet

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jeff Schnitzer]

I serve blobstore images via getServingUrl() all the time, including image blobs that I have created with the files API.  On the dev server.

I don't know your particular problem, but the first step in getting help would be to post a full stacktrace.

Jeff

[Ronoaldo José de Lana Pereira]

Hello Odeyemi,

If I got your problem right, yes, you can use the Files API to create an image blob, then use the ImageService to create a dedicated URL to serve that image as Jeff said. You can safely cache the generated URL, it will be available as long as you don't delete de blob.

Double check that you called the closeFinally as documented here when creating the blob, and use the method getBlobKey from the Files api to get the blob key after you finalize your file. You can then use this to call the ImageService.getServingUrl(). This proccess works for both your unit tests and the local dev server.

Hope this helps!

Best Regards,

-Ronoaldo

Em quinta-feira, 1 de março de 2012 14h19min41s UTC-3, drecute escreveu:

On Thu, Mar 1, 2012 at 3:03 PM, Jeff Schnitzer <je...@infohazard.org> wrote:

Look at the documentation for the images service.  There is a method that gives you a serving url directly to the CDN.

Warning:  This method has a surprising amount of latency, you probably want to cache the value in memcache for a reasonable duration.

Can the Image Service work with the filesystem where the blob is stored? Everytime I try to retrieve it using the method you mentioned getServingUrl(blobkey), I get the error:

[java] java.lang.IllegalArgumentException: Failed to read image
[java]     at com.google.appengine.api.images.ImagesServiceImpl.getServingUrl(ImagesServiceImpl.java:222)

Also, I'm think this action is not possible on the dev server.

On Thu, Mar 1, 2012 at 7:42 AM, Kayode Odeyemi <dre...@gmail.com> wrote:

Hi everyone,

I have images uploaded to the blobstore using filesystem API. The files have been uploaded successfully and I can see the blobs in the respective directory.

But then, how do I decode these files such that when sent back to the client or retrieved using the blobstore API, I'm able to get an image? Right now /blobstore/writable:xxx is not useful for me because when I wrap the path in <img src="/blobstore/writable:xxx" />, it doesn't give me an image.

Any help is appreciated.

Thanks

--
Odeyemi 'Kayode O.
http://ng.linkedin.com/in/kayodeodeyemi. t: @charyorde blog: http://sinati.com/tree/java-cheat-sheet

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jeff Schnitzer]

2012/3/1 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>

Hello Odeyemi,

If I got your problem right, yes, you can use the Files API to create an image blob, then use the ImageService to create a dedicated URL to serve that image as Jeff said. You can safely cache the generated URL, it will be available as long as you don't delete de blob.

Is that url guaranteed to be stable in perpetuity?  The javadocs don't seem to state this explicitly one way or another.

Jeff

[Kayode Odeyemi]

2012/3/1 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>

Hello Odeyemi,

If I got your problem right, yes, you can use the Files API to create an image blob, then use the ImageService to create a dedicated URL to serve that image as Jeff said. You can safely cache the generated URL, it will be available as long as you don't delete de blob.

Double check that you called the closeFinally as documented here when creating the blob, and use the method getBlobKey from the Files api to get the blob key after you finalize your file. You can then use this to call the ImageService.getServingUrl(). This proccess works for both your unit tests and the local dev server.

I think I implemented this just exactly the way you guys described it:

Here's the code I'm working it:
 

@RequestMapping(method=RequestMethod.POST)
    public void handleFileRequest(HttpServletRequest request, HttpServletResponse response) throws Exception {
        Assert.notNull(request, "HttpServletRequest required");
       
        // get the POST data
        Map files = request.getParameterMap();
        Iterator iter = files.values().iterator();
        List<Object> gg = new ArrayList<Object>();
        while(iter.hasNext()) {
            gg.add(iter.next());
        }

        log.log(Level.INFO, "Parameter values are {0}", request.getParameter("qqfile"));

        InputStream stream = request.getInputStream();
        FileService fileService = FileServiceFactory.getFileService();
        AppEngineFile writableFile = null;
       
        for(Object s : gg) {
            writableFile = fileService.createNewBlobFile(s.toString());
        }
        //writableFile = fileService.createNewBlobFile(request.getParameter("qqfile"));

        // Open a channel to write to it
        boolean lock = false;
        FileWriteChannel writeChannel = fileService.openWriteChannel(writableFile, lock);
        ByteBuffer buffer = ByteBuffer.allocateDirect(stream.available());
        lock = true;
        writeChannel = fileService.openWriteChannel(writableFile, lock);
        writeChannel.write(buffer);
        writeChannel.closeFinally();
       
        // get the created files path and pass to PrintWriter
        // Now read from the file using the Blobstore API
        BlobKey blobKey = fileService.getBlobKey(writableFile);
        BlobstoreService blobStoreService = BlobstoreServiceFactory.getBlobstoreService();

        // MAX_BLOB_FETCH_SIZE = 1015808
        String segment = new String(blobStoreService.fetchData(blobKey, 30, 1024));
       
        ImagesService imagesService = ImagesServiceFactory.getImagesService();
        String imageUrl = imagesService.getServingUrl(blobKey);
        log.log(Level.INFO, "Imgae url is {0}", imageUrl);
       
        response.setContentType("application/json; charset=UTF-8");
        PrintWriter out = response.getWriter();
        out.write("{\"result1\" : " + '\"' + writableFile.getFullPath() + '\"' + "}");

When the file was sent, it wasn't sent with the full path to the file. I only got the filename + extension. Something like: file-name.jpg. So I just simply used that to create the new blob file.

Perhaps I might be doing something wrong. I stand to be corrected.

Thanks

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uX0cs5hKeoEJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jeff Schnitzer]

What are you trying to do?

Jeff

[Kayode Odeyemi]

To store upload images as blob using the filesystem api. Then be able
to access the raw image files via a file/url resource path when
wrapped around <img tag or accessed directly through the browser.

On 3/1/12, Jeff Schnitzer <je...@infohazard.org> wrote:
> What are you trying to do?
>
> Jeff
>
> On Thu, Mar 1, 2012 at 2:29 PM, Kayode Odeyemi <dre...@gmail.com> wrote:
>
>> 2012/3/1 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>
>>
>>> Hello Odeyemi,
>>>
>>> If I got your problem right, yes, you can use the Files API to create an
>>> image blob, then use the ImageService to create a dedicated URL to serve
>>> that image as Jeff said. You can safely cache the generated URL, it will
>>> be
>>> available as long as you don't delete de blob.
>>>
>>> Double check that you called the closeFinally as documented here when
>>> creating the blob, and use the method

>>> getBlobKey<http://code.google.com/intl/en/appengine/docs/java/javadoc/com/google/appengine/api/files/FileService.html#getBlobKey%28com.google.appengine.api.files.AppEngineFile%29>from

>>>> [java] java.lang.**IllegalArgumentException: Failed to read image
>>>> [java] at com.google.appengine.api.**images.ImagesServiceImpl.**
>>>> getServingUrl(**ImagesServiceImpl.java:222)

>>>>
>>>> Also, I'm think this action is not possible on the dev server.
>>>>
>>>>>
>>>>> On Thu, Mar 1, 2012 at 7:42 AM, Kayode Odeyemi
>>>>> <dre...@gmail.com>wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I have images uploaded to the blobstore using filesystem API. The
>>>>>> files have been uploaded successfully and I can see the blobs in the
>>>>>> respective directory.
>>>>>>
>>>>>> But then, how do I decode these files such that when sent back to the
>>>>>> client or retrieved using the blobstore API, I'm able to get an image?
>>>>>> Right now /blobstore/writable:xxx is not useful for me because when I
>>>>>> wrap
>>>>>> the path in <img src="/blobstore/writable:xxx" />, it doesn't give me
>>>>>> an
>>>>>> image.
>>>>>>
>>>>>> Any help is appreciated.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Odeyemi 'Kayode O.

>>>>>> http://ng.linkedin.com/in/**kayodeodeyemi<http://ng.linkedin.com/in/kayodeodeyemi>.
>>>>>> t: @charyorde blog:
>>>>>> http://sinati.com/tree/java-**cheat-sheet<http://sinati.com/tree/java-cheat-sheet>

>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Google App Engine" group.

>>>>>> To post to this group, send email to google-appengine@googlegroups.**
>>>>>> com <google-a...@googlegroups.com>.

>>>>>> To unsubscribe from this group, send email to

>>>>>> google-appengine+unsubscribe@**googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>
>>>>>> .
>>>>>> For more options, visit this group at http://groups.google.com/**
>>>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>>>>> .

>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Google App Engine" group.

>>>>> To post to this group, send email to google-appengine@googlegroups.**
>>>>> com <google-a...@googlegroups.com>.

>>>>> To unsubscribe from this group, send email to

>>>>> google-appengine+unsubscribe@**googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>
>>>>> .
>>>>> For more options, visit this group at http://groups.google.com/**
>>>>> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>
>>>>> .
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Odeyemi 'Kayode O.
>>>> http://ng.linkedin.com/in/**kayodeodeyemi<http://ng.linkedin.com/in/kayodeodeyemi>.
>>>> t: @charyorde blog:
>>>> http://sinati.com/tree/java-**cheat-sheet<http://sinati.com/tree/java-cheat-sheet>

[Jeff Schnitzer]

On Thu, Mar 1, 2012 at 3:34 PM, Kayode Odeyemi <dre...@gmail.com> wrote:

To store upload images as blob using the filesystem api. Then be able
to access the raw image files via a file/url resource path when
wrapped around <img tag or accessed directly through the browser.

Forget file paths.  The only thing that matters is 1) getting your blob into the blobstore, and 2) calling getServingUrl().

Jeff

[Kayode Odeyemi]

I got it resolved with this:

FileService fileService = FileServiceFactory.getFileService();
    AppEngineFile writableFile = fileService.createNewBlobFile(request.getParameter("qqfile"));
    FileWriteChannel writeChannel = null;
        try {
                byte[] buffer = new byte[4096]; // 4MB

                lock = true;
                writeChannel = fileService.openWriteChannel(writableFile, lock);

                // increase the buffer size as you are reading from the
                // input stream. Read the input stream into buffer
                for (int n; (n = stream.read(buffer)) != -1; ){
                    writeChannel.write(ByteBuffer.wrap(buffer));
                }
            } finally {
                    writeChannel.closeFinally();
            }

    BlobKey blobKey = fileService.getBlobKey(writableFile);
    ImagesService imagesService = ImagesServiceFactory.getImagesService();

    // produces something like http://localhost:8888/_ah/img/SU52WMsoCRP3kqAvQqVW3g
    String imageUrl = imagesService.getServingUrl(blobKey);

Cool stuff!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Jose Montes de Oca]

Jeff,

Yes, the URL its unique for the life spam of the image. We should better document this on our side though.

Best,

Jose

[Ronoaldo José de Lana Pereira]

Jeff is right, I was actually assuming this behavior by reading this info from javadoc: "If you wish to stop serving the URL, delete the underlying blob key. This takes up to 24 hours to take effect. ". But this don't explicitly states that they will be available forever. Anyway, we are using them for about a year ago.

Just for curiosity, if you call the getServingUrl() twice for the same blob key, I was expecting that they would generate the same URL. Jeff, did you got different URL's?

Best Regards,

-Ronoaldo

[Jeff Schnitzer]

2012/3/5 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>

Just for curiosity, if you call the getServingUrl() twice for the same blob key, I was expecting that they would generate the same URL. Jeff, did you got different URL's?

I never looked at them closely enough, but I wouldn't have trusted my own short-term observations anyways.  In the absence of specific documentation, I just assumed that getServingUrl() should be called every time - for instance, it might generate geo-specific URLs.  One look at appstats in production showed me how wrong this approach was!

Good to know that the URLs are officially stable, though.

Jeff

[Kayode Odeyemi]

2012/3/5 Ronoaldo José de Lana Pereira <rper...@beneficiofacil.com.br>

Jeff is right, I was actually assuming this behavior by reading this info from javadoc: "If you wish to stop serving the URL, delete the underlying blob key. This takes up to 24 hours to take effect. ". But this don't explicitly states that they will be available forever. Anyway, we are using them for about a year ago.

In my use case, I just needed the files to be stored temporarily using the filesystem api (just for UI purposes). To permanently save the uploaded file, I implemented the blobstore api. This way I don't have to worry about whether the files will be available in the saved directory forever.

[Jeff Schnitzer]

On Mon, Mar 5, 2012 at 12:20 PM, Kayode Odeyemi <dre...@gmail.com> wrote:

In my use case, I just needed the files to be stored temporarily using the filesystem api (just for UI purposes). To permanently save the uploaded file, I implemented the blobstore api. This way I don't have to worry about whether the files will be available in the saved directory forever.

This may be a significant part of your misunderstanding.  There is no local filesystem you can write to.  The Files API lets you write to the Blobstore or Google Storage, that's it.  All writes are "permanent" until you delete them.

Jeff

[Kayode Odeyemi]

So whether the files are written using the Files API or directly uploaded to the blobstore, the blobstore still reside on disk?

(Please accept my misunderstanding - It is better to get to understand this once and for all )

[Jeff Schnitzer]

On Mon, Mar 5, 2012 at 12:34 PM, Kayode Odeyemi <dre...@gmail.com> wrote:

So whether the files are written using the Files API or directly uploaded to the blobstore, the blobstore still reside on disk?

The Files API is an interface to the Blobstore and Google Cloud Storage.  The data lives in one or the other; the Files API is just an API.

Jeff