hadoop2 deprecated properties

[Alex Cozzi]

Running on hadoop 2 with 0.8.0 I noticed that scoobi is using a large amount of deprecated properties. I am not sure whether hadoop 2 just complains about them but still uses the properties or the setting are ignored, but I think it would be good if scoobi were to use the new version of the properties when running with hadoop 2. By the way: the warning are issue both on the cluster and in local mode, which should make it easier to test.

[INFO] deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar

[INFO] deprecation - mapred.map.child.log.level is deprecated. Instead, use mapreduce.map.log.level

[INFO] deprecation - mapred.reduce.child.log.level is deprecated. Instead, use mapreduce.reduce.log.level

[INFO] deprecation - mapred.output.value.groupfn.class is deprecated. Instead, use mapreduce.job.output.group.comparator.class

[INFO] deprecation - mapred.output.key.comparator.class is deprecated. Instead, use mapreduce.job.output.key.comparator.class

[INFO] deprecation - mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files

[INFO] deprecation - mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class

[INFO] deprecation - mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class

[INFO] deprecation - mapreduce.partitioner.class is deprecated. Instead, use mapreduce.job.partitioner.class

[INFO] deprecation - mapred.job.name is deprecated. Instead, use mapreduce.job.name

[INFO] deprecation - mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class

[INFO] deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir

[INFO] deprecation - fs.checkpoint.edits.dir is deprecated. Instead, use dfs.namenode.checkpoint.edits.dir

[INFO] deprecation - dfs.data.dir is deprecated. Instead, use dfs.datanode.data.dir

[INFO] deprecation - fs.checkpoint.dir is deprecated. Instead, use dfs.namenode.checkpoint.dir

[INFO] deprecation - mapred.temp.dir is deprecated. Instead, use mapreduce.cluster.temp.dir

[INFO] deprecation - dfs.name.dir is deprecated. Instead, use dfs.namenode.name.dir

[INFO] deprecation - mapred.system.dir is deprecated. Instead, use mapreduce.jobtracker.system.dir

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class

[INFO] deprecation - dfs.name.edits.dir is deprecated. Instead, use dfs.namenode.edits.dir

[INFO] deprecation - user.name is deprecated. Instead, use mapreduce.job.user.name

[INFO] deprecation - mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes

[INFO] deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS

[INFO] deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces

[INFO] deprecation - mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class

[INFO] deprecation - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir

[INFO] deprecation - mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps

[INFO] deprecation - mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps

[INFO] deprecation - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir

[Alex Cozzi]

here is a reference of what properties are deprecated and their replacements: https://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

[Eric Torreborre]

Hi Alex,

I haven't looked deeply into this but:

 - I remember that at some stage some of those warnings were spurious. Hadoop was loading those properties itself and then was saying they were deprecated

 - I still need to support CDH3 (or do I?) so I might only drop those properties when absolutely necessary (and stop supporting CDH3)

In the meantime do you think it would be good to filter out those warnings at the logging level?

Eric.

[Alex Cozzi]

I think you are right, hadoop will take care of setting the properties anyway, event though it complains. And no, I'd probably leave the warnings as well, but it might be good to put a note somewhere that those warnings are ignorable.