Using cloudstorage library instead of Files API for shuffle in Python?

[Jeffrey Tratner]

Hi all,

Are there any plans to replace the Files API with the cloudstorage library for the shuffle phase? We've encountered issues using the mapreduce library for certain tasks because we consistently go over our File Bytes Sent limit, which then means that all other mapreduces can no longer run. For example, I was never able to successfully run a LogInputReader mapreduce, because our logs were just too big.  I've hit similar issues when trying to run mapreduces when there are a very large number of datastore objects involved.

Best,

Jeff

[Tom Kaitchuck]

Yes. Expect to see this released soon.

--
You received this message because you are subscribed to the Google Groups "Google App Engine Pipeline API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeli...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeffrey Tratner]

Great! Looking forward to it!

To unsubscribe from this group and stop receiving emails from it, send an email to app-engine-pipeline-api+unsub...@googlegroups.com.