代码之家 › 专栏 › 技术社区 › Haleemur Ali

Hadoop WordCount为所有单词提供0计数

mapreduce hadoop

Haleemur Ali · 技术社区 · 7 年前

我在hadoop中的WordCount程序中遇到了问题。字数不正确,所有的字都显示为0,但是输出中存在所有不同的字。

这是我的样本数据,加载到hdfs中

# filename: file01.txt
Hello World Bye World

和

# filename: file02.txt
Hello Hadoop Bye Hadoop

这是来源:

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapred.*;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.io.*;


public class WordCount {
    public static class Map
            extends MapReduceBase
            implements Mapper<LongWritable, Text, Text, IntWritable> {


        private final static IntWritable one = new IntWritable();
        private Text word = new Text();

        public void map(LongWritable longWritable, Text value,
                        OutputCollector<Text, IntWritable> output,
                        Reporter reporter) throws IOException {

            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                output.collect(word, one);
            }
        }
    }


    public static class Reduce
            extends MapReduceBase
            implements Reducer<Text, IntWritable, Text, IntWritable> {

        public void reduce(Text key, Iterator<IntWritable> values,
                           OutputCollector<Text, IntWritable> output,
                           Reporter reporter) throws IOException {

            int sum = 0;
            while(values.hasNext()) {
                sum += values.next().get();
            }
            output.collect(key, new IntWritable(sum));
        }
    }

    public static void main(String[] args) throws IOException {

        JobConf jobConf = new JobConf(WordCount.class);
        jobConf.setJobName("wordcount");

        jobConf.setOutputKeyClass(Text.class);
        jobConf.setOutputValueClass(IntWritable.class);

        jobConf.setCombinerClass(WordCount.Reduce.class);
        jobConf.setReducerClass(WordCount.Reduce.class);
        jobConf.setMapperClass(WordCount.Map.class);

        jobConf.setInputFormat(TextInputFormat.class);
        jobConf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(jobConf, new Path(args[0]));
        FileOutputFormat.setOutputPath(jobConf, new Path(args[1]));

        JobClient.runJob(jobConf);
    }
}

当我运行jar时,输出文件在输出文件夹中生成,但它显示以下内容:

$ bin/hdfs dfs -cat ./output/part-00000
17/11/09 02:50:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Bye 0
Hadoop  0
Hello   0
World   0

如您所见,所有计数均为零,但我无法找到在实现中出现错误的地方。

1 回复 | 直到 7 年前

andani 7 年前

是的,我试过调试你的代码错误在你的Map类中

 public static class Map
        extends MapReduceBase
        implements Mapper<LongWritable, Text, Text, IntWritable> {


    private final static IntWritable one = new IntWritable();
    private Text word = new Text();

    public void map(LongWritable longWritable, Text value,
                    OutputCollector<Text, IntWritable> output,
                    Reporter reporter) throws IOException {

        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }
    }
}

由于映射器类返回null(0)作为值,因此reducer无法减少该值

因此,初始化值1,使其为每个单词返回值1。

这是代码

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable();
    private Text word = new Text();

    public void map(LongWritable longWritable, Text value, OutputCollector<Text, IntWritable> output,
            Reporter reporter) throws IOException {

        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            one.set(1);

            output.collect(word, one);
        }
    }

它会起作用。。。。