码迷,mamicode.com
首页 > 其他好文 > 详细

MRUnit的安装和使用

时间:2015-01-19 12:54:56      阅读:729      评论:0      收藏:0      [点我收藏+]

标签:mrunit   mapreduce   

MRUnit是对MapReduce程序进行单元测试的工具,可以对Mapper和Reducer程序分别进行测试。但是它没有集成在Hadoop安装环境中,如果想在开发MapReduce程序时使用这个工具,就需要自己安装。


MRUnit的安装

安装环境:

Eclipse版本为3.6.0

Hadoop版本为1.0.4


安装步骤:

(1)下载MRUnit,网址为http://mrunit.apache.org/,我下载的是 apache-mrunit-1.0.0-hadoop1-bin.tar.gz

(2)解压缩下载的文件,将lib目录下的hamcrest-core-1.1.jar,junit-4.10.jar,mockito-all-1.8.5.jar和mrunit-1.0.0-hadoop1.jar加入到Eclipse的项目中。

方法如下:选中待测试项目-->右键Buid Path-->Configure Buid Path-->点击Libraries-->点击Add External JARs  


MRUnit的使用

以Hadoop权威指南上的MaxTemperature程序为例,整个项目中包括如下4个源文件,前两个分别是Mapper程序和Reducer程序,后两个分别是针对Mapper和Reducer的测试程序:

MaxTemperatureMapper.java,MaxTemperatureReducer.java,MaxTemperatureMapperTest.java,MaxTemperatureReducerTest.java


MaxTemperatureMapper.java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MaxTemperatureMapper 
	extends Mapper<LongWritable, Text, Text, IntWritable> {
	private static final int MISSING = 9999;
	
	public void map(LongWritable key, Text value, Context context)
		throws IOException, InterruptedException {
		String line = value.toString();
		String year = line.substring(15, 19);
		
		int airTemperature;
		if(line.charAt(87) == '+')
			airTemperature = Integer.parseInt(line.substring(88, 92));
		else
			airTemperature = Integer.parseInt(line.substring(87, 92));
		
		String quality = line.substring(92, 93);
		if(airTemperature != MISSING && quality.matches("[01459]"))
			context.write(new Text(year), new IntWritable(airTemperature));
	}
}


MaxTemperatureReducer.java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxTemperatureReducer
	extends Reducer<Text, IntWritable, Text, IntWritable> {
	
	public void reduce(Text key, Iterable<IntWritable> values, Context context) 
		throws IOException, InterruptedException {
		
		int maxValue = Integer.MIN_VALUE;
		for(IntWritable val : values)
			maxValue = Math.max(maxValue, val.get());
		
		context.write(key, new IntWritable(maxValue));
	}
}

MaxTemperatureMapperTest.java

import java.io.IOException;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.junit.Test;

public class MaxTemperatureMapperTest {
	@Test
	public void processesValidRecord() throws IOException, InterruptedException {
		Text value = new Text("0057332130999991950010103004+51317+028783FM-12+017199999V0203201N00721004501CN0100001N9-01281-01391102681");
		
		new MapDriver<LongWritable, Text, Text, IntWritable>()
			.withMapper(new MaxTemperatureMapper())
			.withInput(new LongWritable(0), value)
			.withOutput(new Text("1950"), new IntWritable(-128))
			.runTest();
	}
}

MaxTemperatureReducerTest.java

import java.io.IOException;
import java.util.Arrays;

import org.junit.Test;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;

public class MaxTemperatureReducerTest {
	@Test
	public void returnsMaximumIntegerInValues() throws IOException, InterruptedException {
		new ReduceDriver<Text, IntWritable, Text, IntWritable>()
			.withReducer(new MaxTemperatureReducer())
			.withInput(new Text("1950"), Arrays.asList(new IntWritable(10), new IntWritable(5)))
			.withOutput(new Text("1950"), new IntWritable(10))
			.runTest();
	}
}

需要注意程序中的@Test是不可或缺的


下面就可以运行测试程序了,如果要测试Mapper程序,那么就选中MaxTemperatureMapperTest.java,右键Run As-->JUnit Test,如果进度条为绿色,则表示测试正确,否则代表有错误。测试Reducer程序也是如此。

MRUnit的安装和使用

标签:mrunit   mapreduce   

原文地址:http://blog.csdn.net/warren912/article/details/42870639

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!