New version of App Engine MapReduce for Java

[Christian Ohler]

Hi all,

The appengine-mapreduce repository now contains a new version of App
Engine MapReduce for Java. It finally includes support for shuffling
and reducing. It also has reliability improvements and integrates
with appengine-pipeline. See
http://code.google.com/p/appengine-mapreduce/wiki/GettingStartedInJava
if you want to start using it.

This is an early release; the code is still somewhat rough and has not
been tested extensively, but it's superior to the previous version in
many ways already. Some features are missing (such as a Hadoop
compatibility layer and an overview page that shows past and current
jobs); we'll publish improvements as we make them.

Please let us know through this list and the issue tracker what bugs
you run into and what features you'd like to see.

For documentation, see the above link and the Javadoc. More
documentation will be coming over the next months.

See http://code.google.com/p/appengine-mapreduce/wiki/LargeDatasets
for limitations on dataset size and what to do if you need to process
larger datasets.

Happy hacking,
Christian.

[Krzysiek]

Thank you!

Yesterday I've build a Mapper for some schema changes in my app with the new API. It did the job!

Thanks for updating GettingStartedInJava site with the info that "The current code does not offer an overview page that displays all running jobs". Yesterday I thought something was wrong when I saw "java.lang.RuntimeException: Not implemented" on the status page. I had to implement my own control panel for the jobs listing. It would be nice to have one out of the box.

[Scott Murphy]

Hi,

What's the status on this.  Is there an overview page yet?
I am holding off until I know I can check the status of jobs.

Best,

Scott

[Emanuele Ziglioli]

Hi everyone,

I'm trying to build some Mapper classes with the latest code of MapReduce for AppEngine for Java.

I get this compiler error:

public class TestMapper extends AppEngineMapper<Key, Entity, NullWritable, NullWritable> {

The class file Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> contains a signature '(Lorg/apache/hadoop/mapreduce/Mapper<TKEYIN;TVALUEIN;TKEYOUT;TVALUEOUT;>.Context;)V' ill-formed at position 73

Has anyone seen this? 

thanks,

Emanuele