运行Hadoop示例程序WordCount

标签：

WordCount程序的目标是统计几个文件中每个单词出现的次数，是官方提供的示例程序，这里使用的hadoop的版本为hadoop-1.2.1。

1）、首先编写代码，将WordCount.java文件放到wordcount_classes文件夹中，代码如下：

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class WordCount {

        public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
                private final static IntWritable one = new IntWritable(1);
                private Text word = new Text();

                public void map(LongWritable key, Text value,OutputCollector<Text, IntWritable> output, Reporter reporter)
                                throws IOException {
                        String line = value.toString();
                        StringTokenizer tokenizer = new StringTokenizer(line);
                        while (tokenizer.hasMoreTokens()) {
                                word.set(tokenizer.nextToken());
                                output.collect(word, one);
                        }
                }
        }

        public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
                public void reduce(Text key, Iterator<IntWritable> values,
                                OutputCollector<Text, IntWritable> output, Reporter reporter)
                                throws IOException {
                        int sum = 0;
                        while (values.hasNext()) {
                                sum += values.next().get();
                        }
                        output.collect(key, new IntWritable(sum));
                }
        }

        public static void main(String[] args) throws Exception {
                JobConf conf = new JobConf(WordCount.class);
                conf.setJobName("wordcount");

                conf.setOutputKeyClass(Text.class);
                conf.setOutputValueClass(IntWritable.class);

                conf.setMapperClass(Map.class);
                conf.setCombinerClass(Reduce.class);
                conf.setReducerClass(Reduce.class);

                conf.setInputFormat(TextInputFormat.class);
                conf.setOutputFormat(TextOutputFormat.class);

                FileInputFormat.setInputPaths(conf, new Path(args[0]));
                FileOutputFormat.setOutputPath(conf, new Path(args[1]));

                JobClient.runJob(conf);
        }
}

$ javac –classpath ~/hadoop-1.2.1/hadoop-core-1.2.1.jar-dwordcount_classes  ~/wordcount_classes/WordCount.java
$ jar –cvf wordcount.jar -C wordcount_classes/.

3）、在HDFS中新建输入文件的文件夹input，建好后使用-ls命令查看

$ bin/hadoop dfs –mkdir input
$ bin/hadoop dfs –ls

$ bin/hadoop dfs -put ~/input_files/* input

$ bin/hadoop jar~/wordcount_classes/wordcount.jar WordCount input output

也可以使用下面的命令，将output文件下载下来查看

$ bin/hadoop dfs –get output output_local

示例程序结束，下面是HDFS常用的操作:

hadoop dfs -ls in 列出HDFS下某个文档中的文件
hadoop dfs -put test1.txt test 上传文件到指定目录并且重新命名，只有所有的DataNode都接收完数据才算成功
hadoop dfs -get in getin 从HDFS获取文件并且重新命名为getin，同put一样可操作文件也可操作目录
hadoop dfs -rmr out 删除指定文件从HDFS上
hadoop dfs -cat in/* 查看HDFS上in目录的内容
hadoop dfsadmin -report 查看HDFS的基本统计信息，结果如下
hadoop dfsadmin -safemode leave 退出安全模式
hadoop dfsadmin -safemode enter 进入安全模式

运行Hadoop示例程序WordCount

标签：

原文地址：http://blog.csdn.net/x_i_y_u_e/article/details/44944099