Modify core-site.xml io.compression.codecs on default image

[David Koch]

Hello

This is minor detail!

hadoop-lzo is installed on Dataproc, however, to really use it in M/R and Spark processing one still needs to modify core-site.xml by way of providing a value for --properties during cluster startup. We add:

 

gloud dataproc clusters create ... --properties '^;^core:io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec;core:io.compression.codec.lzo.class=com.hadoop.compression.lzo.LzoCodec'

If this was not done intentionally, would it be possible to add the line to the Dataproc images's core-site.xml?

See also here: Adding LZO support to Dataproc · source{d} blog

Thank you,

David