Unserializing large Blobs from the Blobstore

[Luca Matteis]

Hello,

I store large Blobs in the datastore. These are XML files that can be
as large as 20mb. Storing a single 20mb XML file is fine, however the
issue comes when I need to unserialize it.

There are all sorts of limitations to this:

- I cannot read more than 1mb from Blobstore
- even if I could I'm still confronted with the 5mb RAM limitation
(since I need to put all the XML into RAM before unserializing it)

How would you say I can handle this? I'm open to all sorts of
solutions, but hopefully not something that involves using another
hosting provider.

Thanks,
Luca

[Matt Myers]

Hi Luca, did you ever figure this out?

[Luca Matteis]

Hi Matt,

There are some techniques for using as little memory as possible when
serializing those large XML files, but you'll most likely go over the
limit. Backends solve this issue. I think they need a little
improvement since they require a little of sys-admin work to setup...
but they're meant to fix this specific issue.

Luca

On Mon, Aug 1, 2011 at 9:37 PM, Matt Myers <mattmy...@gmail.com> wrote:
> Hi Luca, did you ever figure this out?
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/L2H3-B0iq-kJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

[Robert Kluin]

On Mon, Jul 11, 2011 at 05:55, Luca Matteis <lmat...@gmail.com> wrote:
> Hello,
>
> I store large Blobs in the datastore. These are XML files that can be
> as large as 20mb. Storing a single 20mb XML file is fine, however the
> issue comes when I need to unserialize it.
>
> There are all sorts of limitations to this:
>
> - I cannot read more than 1mb from Blobstore

Read in 1mb chunks, then assemble and parse.

> - even if I could I'm still confronted with the 5mb RAM limitation
> (since I need to put all the XML into RAM before unserializing it)

What 5mb RAM limit?

>
> How would you say I can handle this? I'm open to all sorts of
> solutions, but hopefully not something that involves using another
> hosting provider.

Your backend solution sounds like a good way to go here.

Robert

>
> Thanks,
> Luca

>
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.