Monday, September 4, 2017

Remove keys from output files in MR

E.g.
public class WordCount {
 
    public static void main(String[] args) throws Exception {
        JobConf conf = new JobConf(WordCount.class);
        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(Text.class);
        conf.setOutputFormat(CustomMultipleTextOutputFormat.class);
        FileInputFormat.setInputPaths(conf, new Path(args[0]));
        FileOutputFormat.setOutputPath(conf, new Path(args[1]));
        JobClient.runJob(conf);
    }
}
 
class CustomMultipleTextOutputFormat extends MultipleTextOutputFormat<Object, Object> {
    protected Object generateActualKey(Object key, Object value) {
        return NullWritable.get();
    }
}

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.