0.6.1 is released

[Eric Torreborre]

Hi all,

Scoobi 0.6.1 is now released, with a quite a few changes and fixes:

New Features

• issue 138: support for the Avro Fixed type
• issue 150: added Avro Specific/GenericRecord support (thanks to @ccsevers!)
• issue 156: a DObject[T] can be transformed to a DList[T]with a single element and to a DList[Iterable[S]] if T <:< Iterable[S]
• issue 157: added a groupByKeyWith method to explicitly specify the Grouping instance to use
• issue 160: fast local mode with the inmemory argument to run the Scoobi code with Scala collections instead of Hadoop in order to faster check the application logic

Improvements

• issue 142: added a better error message if a Writable type can’t be instantiated because it doesn’t have a no-args constructor
• issue 154: (A, B) has a Grouping instance if A and B does
• issue 163: it is possible to set Scoobi’s directory
• issue 167: when using a “lean” jar, the dependent jars can be found from the $HADOOP_CLASSPATH variable
• issue 167: upload a jar to the libjars directory in the cluster if its size has changed
• issue 167: the libjars directory can be changed with a system variable
• issue 167: added a transpose method for matrices
• issue 167: added Grouping, Ordering and WireFormatinstances for tagged types
• issue 167: added lots of Scoobi debug log statements during the execution of a ScoobiApp
• added WireFormat instances for Shapeless tagged types
• made the Iterable returned by materialise lazy
• tuples of DListPersisters can be persisted as well
• support for WireFormats with up to 22 arguments in tuples and case classes
• extended the DataSink api to declare that outputs must be compressed
• a DMatrix can be multiplied by a function

Changes

• issue 166: Scoobi’s directory is /tmp/scoobi-{user.name}/by default now
• issue 167: the user classpath takes precedence by default ->mapreduce.task.classpath.user.precedence=true
• DList.materialize is deprecated, useDList.materialise instead

Fixes

• issue 142: added a better error message if a Writable type can’t be instantiated because it doesn’t have a no-args constructor
• issue 153: generating _SUCCESS files in the output directories
• issue 158: made sure that Avro is working with both cdh3 and cdh4
• issue 164: fixed the User Guide display on low resolution screens
• issue 169: MR output files should be moved to to output dirs, not copied
• issue 184: Pass along proper context to OutputFormat#close
• issue 185: output files can be copied to the S3 file system when the job is executed on EC2

Cheers,

Eric.