（转）基于MapReduce的ItemBase推荐算法的共现矩阵实现（一）

时间：2014-12-15 17:15:03 阅读：542 评论：0 收藏：0 [点我收藏+]

转自：http://zengzhaozheng.blog.51cto.com/8219051/1557054

一、概述

这2个月研究根据用户标签情况对用户的相似度进行评估，其中涉及一些推荐算法知识，在这段时间研究了一遍《推荐算法实践》和《Mahout in action》，在这里主要是根据这两本书的一些思想和自己的一些理解对分布式基于ItemBase的推荐算法进行实现。其中分两部分，第一部分是根据共现矩阵的方式来简单的推算出用户的推荐项，第二部分则是通过传统的相似度矩阵的方法来实践ItemBase推荐算法。这篇blog主要记录第一部分的内容，并且利用MapReduce进行实现，下一篇blog则是记录第二部分的内容和实现。

二、算法原理

协同推荐算法，作为众多推荐算法中的一种已经被广泛的应用。其主要分为2种，第一种就是基于用户的协同过滤，第二种就是基于物品的协同过滤。

所谓的itemBase推荐算法简单直白的描述就是：用户A喜欢物品X1，用户B喜欢物品X2，如果X1和X2相似则，将A之前喜欢过的物品推荐给B，或者B之前喜欢过的物品推荐给A。这种算法是完全依赖于用户的历史喜欢物品的；所谓的UserBase推荐算法直白地说就是：用户A喜欢物品X1，用户B喜欢物品X2，如果用户A和用户B相似则将物品X1推荐给用户B，将物品X2推荐给用户A。简单的示意图：

bubuko.com,布布扣

至于选择哪种要看自己的实际情况，如果用户量比物品种类多得多那么就采用ItemBase的协同过滤推荐算法，如果是用户量比物品种类少的多则采用UserBase的协同顾虑推荐算，这样选择的一个原因是为了让物品的相似度矩阵或者用户相似度矩阵或者共现矩阵的规模最小化。

三、数据建模

基本的算法上面已经大概说了一下，对于算法来说，对数据建模使之运用在算法之上是重点也是难点。这小节主要根据自己相关项目的经验和《推荐引擎实践》的一些观点来讨论一些。分开2部分说，一是根据共现矩阵推荐、而是根据相似度算法进行推荐。

(1)共现矩阵方式：

第一步：转换成用户向量

1[102:0.1,103:0.2,104:0.3]：表示用户1喜欢的物品列表，以及他们对应的喜好评分。

2[101:0.1,102:0.7,105:0.9]：表示用户2喜欢的物品列表，以及他们对应的喜好评分。

3[102:0.1,103:0.7,104:0.2]：表示用户3喜欢的物品列表，以及他们对应的喜好评分。

第二步：计算共现矩阵

简单地说就是将同时喜欢物品x1和x2的用户数组成矩阵。

第三步：

生成用户对物品的评分矩阵

第四步：物品共现矩阵和用户对物品的评分矩阵相乘得到推荐结果

举个例子计算用户1的推荐列表过程：

用户1对物品101的总评分计算：

1*0+1*0.1+0*0.2+0*0.3+1*0=0.1

用户1对物品102的总评分计算：

1*0+3*0.1+1*0.2+2*0.3+2*0=1.1

用户1对物品103的总评分计算：

0*0+1*0.1+1*0.2+1*0.3+0*0=0.6

用户1对物品104的总评分计算：

0*0+2*0.1+1*0.2+2*0.3+1*0=1.0

用户1对物品105的总评分计算：

1*0+2*0.1+0*0.2+1*0.3+2*0=0.5

从而得到用户1的推荐列表为1[101:0.1,102:1.1,103:0.6,104:1.0,105:0.5]再经过排序得到最终推荐列表1[102:1.1,104:1.0,103:0.6,105:0.5,101:0.1]。

(2)通过计算机物品相似度方式计算用户的推荐向量。

通过计算机物品相似度方式计算用户的推荐向量和上面通过共现矩阵的方式差不多，就是将物品相似度矩阵代替掉共现矩阵和用户对物品的评分矩阵相乘，然后在计算推荐向量。

计算相似度矩阵：

在计算之前我们先了解一下物品相似度相关的计算方法。

对于计算物品相似度的算法有很多，要根据自己的数据模型进行选择。基于皮尔逊相关系数计算、欧几里德定理（实际上是算两点距离）、基于余弦相似度计算斯皮尔曼相关系数计算、基于谷本系数计算、基于对数似然比计算。其中谷本系数和对数似然比这两种方式主要是针对那些没有指名对物品喜欢度的数据模型进行相似度计算，也就是mahout中所指的Boolean数据模型。下面主要介绍2种，欧几里德和余弦相似度算法。

现在关键是怎么将现有数据转化成对应的空间向量模型使之适用这些定理，这是个关键点。下面我以欧几里德定理作为例子看看那如何建立模型：

第一步：将用户向量转化为物品向量

用户向量：

1[102:0.1,103:0.2,104:0.3]

2[101:0.1,102:0.7,105:0.9]

3[102:0.1,103:0.7,104:0.2]

转为为物品向量：

101[2:0.1]

102[1:0.1,2:0.7,3:0.1]

103[1:0.2,3:0.7]

104[1:0.3,3:0.2]

105[2:0.9]

第二步：

那么物品相似度计算为：

第三步：

最终得到物品相似度矩阵为：(这里省略掉没有意义的自关联相似度)

第四步：物品相似度矩阵和用户对物品的评分矩阵相乘得到推荐结果：

举个例子计算用户1的类似推荐列表过程：

用户1对物品101的总评分计算：

1*0+1*0.6186429+0*0.6964322+0*0.7277142+1*0.55555556=1.174198

用户1对物品102的总评分计算：

1*0.6186429+3*0+1*0.5188439+2*0.5764197+2*0.8032458=3.896818

用户1对物品103的总评分计算：

0*0.6964322+1*0.5188439+1*0+1*0.662294+0*0.463481=1.181138

用户1对物品104的总评分计算：

0*0.7277142+2*0.5764197+1*0.662294+2*0+1*0.5077338=2.322867

用户1对物品105的总评分计算：

1*0.55555556+2*0.8032458+0*0.463481+1*0.5077338=2.669780

四、共现矩阵方式的MapReduce实现

这里主要是利用MapReduce结合Mahout连的一些数据类型对共现矩阵方式的推荐方法进行实现,至于相似度矩阵方式进行推荐的在下一篇blog写。这里采用Boolean数据模型，即用户是没有对喜欢的物品进行初始打分的，我们在程序中默认都为1。

先看看整个MapReduce的数据流向图：

具体代码实现：HadoopUtil

package com.util;
import java.io.IOException;
import java.util.Arrays;
import java.util.Iterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.PathFilter;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.OutputFormat;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.common.iterator.sequencefile.PathType;
import org.apache.mahout.common.iterator.sequencefile.SequenceFileDirValueIterator;
import org.apache.mahout.common.iterator.sequencefile.SequenceFileValueIterator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
publicfinalclassHadoopUtil{
privatestaticfinalLogger log =LoggerFactory.getLogger(HadoopUtil.class);
privateHadoopUtil(){}
publicstaticJob prepareJob(String jobName,
String[] inputPath,
String outputPath,
Class<?extendsInputFormat> inputFormat,
Class<?extendsMapper> mapper,
Class<?extendsWritable> mapperKey,
Class<?extendsWritable> mapperValue,
Class<?extendsOutputFormat> outputFormat,Configuration conf)throwsIOException{
Job job =newJob(newConfiguration(conf));
job.setJobName(jobName);
Configuration jobConf = job.getConfiguration();
if(mapper.equals(Mapper.class)){
thrownewIllegalStateException("Can‘t figure out the user class jar file from mapper/reducer");
}
job.setJarByClass(mapper);
job.setInputFormatClass(inputFormat);
job.setInputFormatClass(inputFormat);
StringBuilder inputPathsStringBuilder =newStringBuilder();
for(String p : inputPath){
inputPathsStringBuilder.append(",").append(p);
}
inputPathsStringBuilder.deleteCharAt(0);
jobConf.set("mapred.input.dir", inputPathsStringBuilder.toString());
job.setMapperClass(mapper);
job.setMapOutputKeyClass(mapperKey);
job.setMapOutputValueClass(mapperValue);
job.setOutputKeyClass(mapperKey);
job.setOutputValueClass(mapperValue);
jobConf.setBoolean("mapred.compress.map.output",true);
job.setNumReduceTasks(0);
job.setOutputFormatClass(outputFormat);
jobConf.set("mapred.output.dir", outputPath);
return job;
}
publicstaticJob prepareJob(String jobName,
String[] inputPath,
String outputPath,
Class<?extendsInputFormat> inputFormat,
Class<?extendsMapper> mapper,
Class<?extendsWritable> mapperKey,
Class<?extendsWritable> mapperValue,
Class<?extendsReducer> reducer,
Class<?extendsWritable> reducerKey,
Class<?extendsWritable> reducerValue,
Class<?extendsOutputFormat> outputFormat,
Configuration conf)throwsIOException{
Job job =newJob(newConfiguration(conf));
job.setJobName(jobName);
Configuration jobConf = job.getConfiguration();
if(reducer.equals(Reducer.class)){
if(mapper.equals(Mapper.class)){
thrownewIllegalStateException("Can‘t figure out the user class jar file from mapper/reducer");
}
job.setJarByClass(mapper);
}else{
job.setJarByClass(reducer);
}
job.setInputFormatClass(inputFormat);
StringBuilder inputPathsStringBuilder =newStringBuilder();
for(String p : inputPath){
inputPathsStringBuilder.append(",").append(p);
}
inputPathsStringBuilder.deleteCharAt(0);
jobConf.set("mapred.input.dir", inputPathsStringBuilder.toString());
job.setMapperClass(mapper);
if(mapperKey !=null){
job.setMapOutputKeyClass(mapperKey);
}
if(mapperValue !=null){
job.setMapOutputValueClass(mapperValue);
}
jobConf.setBoolean("mapred.compress.map.output",true);
job.setReducerClass(reducer);
job.setOutputKeyClass(reducerKey);
job.setOutputValueClass(reducerValue);
job.setOutputFormatClass(outputFormat);
jobConf.set("mapred.output.dir", outputPath);
return job;
}
publicstaticJob prepareJob(String jobName,String[] inputPath,
String outputPath,Class<?extendsInputFormat> inputFormat,
Class<?extendsMapper> mapper,
Class<?extendsWritable> mapperKey,
Class<?extendsWritable> mapperValue,
Class<?extendsReducer> combiner,
Class<?extendsReducer> reducer,
Class<?extendsWritable> reducerKey,
Class<?extendsWritable> reducerValue,
Class<?extendsOutputFormat> outputFormat,Configuration conf)
throwsIOException{
Job job =newJob(newConfiguration(conf));
job.setJobName(jobName);
Configuration jobConf = job.getConfiguration();
if(reducer.equals(Reducer.class)){
if(mapper.equals(Mapper.class)){
thrownewIllegalStateException(
"Can‘t figure out the user class jar file from mapper/reducer");
}
job.setJarByClass(mapper);
}else{
job.setJarByClass(reducer);
}
job.setInputFormatClass(inputFormat);
StringBuilder inputPathsStringBuilder =newStringBuilder();
for(String p : inputPath){
inputPathsStringBuilder.append(",").append(p);
}
inputPathsStringBuilder.deleteCharAt(0);
jobConf.set("mapred.input.dir", inputPathsStringBuilder.toString());
job.setMapperClass(mapper);
if(mapperKey !=null){
job.setMapOutputKeyClass(mapperKey);
}
if(mapperValue !=null){
job.setMapOutputValueClass(mapperValue);
}
jobConf.setBoolean("mapred.compress.map.output",true);
job.setCombinerClass(combiner);
job.setReducerClass(reducer);
job.setOutputKeyClass(reducerKey);
job.setOutputValueClass(reducerValue);
job.setOutputFormatClass(outputFormat);
jobConf.set("mapred.output.dir", outputPath);
return job;
}
publicstaticString getCustomJobName(String className,JobContext job,
Class<?extendsMapper> mapper,
Class<?extendsReducer> reducer){
StringBuilder name =newStringBuilder(100);
String customJobName = job.getJobName();
if(customJobName ==null|| customJobName.trim().isEmpty()){
name.append(className);
}else{
name.append(customJobName);
}
name.append(‘-‘).append(mapper.getSimpleName());
name.append(‘-‘).append(reducer.getSimpleName());
return name.toString();
}
publicstaticvoiddelete(Configuration conf,Iterable<Path> paths)throwsIOException{
if(conf ==null){
conf =newConfiguration();
}
for(Path path : paths){
FileSystem fs = path.getFileSystem(conf);
if(fs.exists(path)){
log.info("Deleting {}", path);
fs.delete(path,true);
}
}
}
publicstaticvoiddelete(Configuration conf,Path... paths)throwsIOException{
delete(conf,Arrays.asList(paths));
}
publicstaticlong countRecords(Path path,Configuration conf)throwsIOException{
long count =0;
Iterator<?> iterator =newSequenceFileValueIterator<Writable>(path,true, conf);
while(iterator.hasNext()){
iterator.next();
count++;
}
return count;
}
publicstaticlong countRecords(Path path,PathType pt,PathFilter filter,Configuration conf)throwsIOException{
long count =0;
Iterator<?> iterator =newSequenceFileDirValueIterator<Writable>(path, pt, filter,null,true, conf);
while(iterator.hasNext()){
iterator.next();
count++;
}
return count;
}
}

先看看写的工具类：

第一步：处理原始输入数据

处理原始数据的SourceDataToItemPrefsJob作业的mapper：SourceDataToItemPrefsMapper

package com.mapper;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.mahout.math.VarLongWritable;
/**
* mapper输入格式：userID:itemID1 itemID2 itemID3....
* mapper输出格式:<userID,itemID>
* @author 曾昭正
*/
publicclassSourceDataToItemPrefsMapperextendsMapper<LongWritable,Text,VarLongWritable,VarLongWritable>{
//private static final Logger logger = LoggerFactory.getLogger(SourceDataToItemPrefsMapper.class);
privatestaticfinalPattern NUMBERS =Pattern.compile("(\\d+)");
privateString line =null;
@Override
protectedvoid map(LongWritable key,Text value,Context context)
throwsIOException,InterruptedException{
line = value.toString();
if(line ==null)return;
// logger.info("line:"+line);
Matcher matcher = NUMBERS.matcher(line);
matcher.find();//寻找第一个分组，即userID
VarLongWritable userID =newVarLongWritable(Long.parseLong(matcher.group()));//这个类型是在mahout中独立进行封装的
VarLongWritable itemID =newVarLongWritable();
while(matcher.find()){
itemID.set(Long.parseLong(matcher.group()));
// logger.info(userID + " " + itemID);
context.write(userID, itemID);
}
}
}

处理原始数据的SourceDataToItemPrefsJob作业的reducer：SourceDataToItemPrefsMapper

package com.reducer;
import java.io.IOException;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.math.RandomAccessSparseVector;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* reducer输入：<userID,Iterable<itemID>>
* reducer输出:<userID,VecotrWriable<index=itemID,valuce=pres>....>
* @author 曾昭正
*/
publicclassSourceDataToUserVectorReducerextendsReducer<VarLongWritable,VarLongWritable,VarLongWritable,VectorWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(SourceDataToUserVectorReducer.class);
@Override
protectedvoid reduce(VarLongWritable userID,Iterable<VarLongWritable> itemPrefs,Context context)
throwsIOException,InterruptedException{
/**
* DenseVector，它的实现就是一个浮点数数组，对向量里所有域都进行存储，适合用于存储密集向量。
RandomAccessSparseVector 基于浮点数的 HashMap 实现的，key 是整形 (int) 类型，value 是浮点数 (double) 类型，它只存储向量中不为空的值，并提供随机访问。
SequentialAccessVector 实现为整形 (int) 类型和浮点数 (double) 类型的并行数组，它也只存储向量中不为空的值，但只提供顺序访问。
用户可以根据自己算法的需求选择合适的向量实现类，如果算法需要很多随机访问，应该选择 DenseVector 或者 RandomAccessSparseVector，如果大部分都是顺序访问，SequentialAccessVector 的效果应该更好。
介绍了向量的实现，下面我们看看如何将现有的数据建模成向量，术语就是“如何对数据进行向量化”，以便采用 Mahout 的各种高效的聚类算法。
*/
Vector userVector =newRandomAccessSparseVector(Integer.MAX_VALUE,100);
for(VarLongWritable itemPref : itemPrefs){
userVector.set((int)itemPref.get(),1.0f);//RandomAccessSparseVector.set(index,value),用户偏好类型为boolean类型，将偏好值默认都为1.0f
}
logger.info(userID+" "+newVectorWritable(userVector));
context.write(userID,newVectorWritable(userVector));
}
}

第二步：将SourceDataToItemPrefsJob作业的reduce输出结果组合成共现矩阵

UserVectorToCooccurrenceJob作业的mapper：UserVectorToCooccurrenceMapper

package com.mapper;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;
/**
* mapper输入：<userID,VecotrWriable<index=itemID,valuce=pres>....>
* mapper输出:<itemID,itemID>(共现物品id对)
* @author 曾昭正
*/
publicclassUserVectorToCooccurrenceMapperextendsMapper<VarLongWritable,VectorWritable,IntWritable,IntWritable>{
@Override
protectedvoid map(VarLongWritable userID,VectorWritable userVector,Context context)
throwsIOException,InterruptedException{
Iterator<Vector.Element> it = userVector.get().nonZeroes().iterator();//过滤掉非空元素
while(it.hasNext()){
int index1 = it.next().index();
Iterator<Vector.Element> it2 = userVector.get().nonZeroes().iterator();
while(it2.hasNext()){
int index2 = it2.next().index();
context.write(newIntWritable(index1),newIntWritable(index2));
}
}
}
}

UserVectorToCooccurrenceJob作业的reducer：UserVectorToCoocurrenceReducer

package com.reducer;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.cf.taste.hadoop.item.VectorOrPrefWritable;
import org.apache.mahout.math.RandomAccessSparseVector;
import org.apache.mahout.math.Vector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* reducer输入:<itemID,Iterable<itemIDs>>
* reducer输出:<mainItemID,Vector<coocItemID,coocTime(共现次数)>....>
* @author 曾昭正
*/
publicclassUserVectorToCoocurrenceReducerextendsReducer<IntWritable,IntWritable,IntWritable,VectorOrPrefWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(UserVectorToCoocurrenceReducer.class);
@Override
protectedvoid reduce(IntWritable mainItemID,Iterable<IntWritable> coocItemIDs,Context context)
throwsIOException,InterruptedException{
Vector coocItemIDVectorRow =newRandomAccessSparseVector(Integer.MAX_VALUE,100);
for(IntWritable coocItem : coocItemIDs){
int itemCoocTime = coocItem.get();
coocItemIDVectorRow.set(itemCoocTime,coocItemIDVectorRow.get(itemCoocTime)+1.0);//将共现次数累加
}
logger.info(mainItemID +" "+newVectorOrPrefWritable(coocItemIDVectorRow));
context.write(mainItemID,newVectorOrPrefWritable(coocItemIDVectorRow));//记录mainItemID的完整共现关系
}
}

第三步：将SourceDataToItemPrefsJob作业的reduce输出结果进行分割

userVecotrSplitJob作业的mapper：UserVecotrSplitMapper

package com.mapper;
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.mahout.cf.taste.hadoop.item.VectorOrPrefWritable;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.Vector.Element;
import org.apache.mahout.math.VectorWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 将用户向量分割，以便和物品的共现向量进行合并
* mapper输入:<userID,Vector<itemIDIndex,preferenceValuce>....>
* reducer输出:<itemID,Vecotor<userID,preferenceValuce>....>
* @author 曾昭正
*/
publicclassUserVecotrSplitMapperextendsMapper<VarLongWritable,VectorWritable,IntWritable,VectorOrPrefWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(UserVecotrSplitMapper.class);
@Override
protectedvoid map(VarLongWritable userIDWritable,VectorWritable value,Context context)
throwsIOException,InterruptedException{
IntWritable itemIDIndex =newIntWritable();
long userID = userIDWritable.get();
Vector userVector = value.get();
Iterator<Element> it = userVector.nonZeroes().iterator();//只取非空用户向量
while(it.hasNext()){
Element e = it.next();
int itemID = e.index();
itemIDIndex.set(itemID);
float preferenceValuce =(float) e.get();
logger.info(itemIDIndex +" "+newVectorOrPrefWritable(userID,preferenceValuce));
context.write(itemIDIndex,newVectorOrPrefWritable(userID,preferenceValuce));
}
}
}

第四步：将userVecotrSplitJob和UserVectorToCooccurrenceJob作业的输出结果合并

combineUserVectorAndCoocMatrixJob作业的mapper：CombineUserVectorAndCoocMatrixMapper

package com.mapper;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.mahout.cf.taste.hadoop.item.VectorOrPrefWritable;
/**
* 将共现矩阵和分割后的用户向量进行合并，以便计算部分的推荐向量
* 这个mapper其实没有什么逻辑处理功能，只是将数据按照输入格式输出
* 注意：这里的mapper输入为共现矩阵和分割后的用户向量计算过程中的共同输出的2个目录
* mapper输入：<itemID,Vecotor<userID,preferenceValuce>> or <itemID,Vecotor<coocItemID,coocTimes>>
* mapper输出:<itemID,Vecotor<userID,preferenceValuce>/Vecotor<coocItemID,coocTimes>>
* @author 曾昭正
*/
publicclassCombineUserVectorAndCoocMatrixMapperextendsMapper<IntWritable,VectorOrPrefWritable,IntWritable,VectorOrPrefWritable>{
@Override
protectedvoid map(IntWritable itemID,VectorOrPrefWritable value,Context context)
throwsIOException,InterruptedException{
context.write(itemID, value);
}
}

combineUserVectorAndCoocMatrixJob作业的CombineUserVectorAndCoocMatrixReducer

package com.reducer;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.cf.taste.hadoop.item.VectorAndPrefsWritable;
import org.apache.mahout.cf.taste.hadoop.item.VectorOrPrefWritable;
import org.apache.mahout.math.Vector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 将共现矩阵和分割后的用户向量进行合并，以便计算部分的推荐向量
* @author 曾昭正
*/
publicclassCombineUserVectorAndCoocMatrixReducerextendsReducer<IntWritable,VectorOrPrefWritable,IntWritable,VectorAndPrefsWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(CombineUserVectorAndCoocMatrixReducer.class);
@Override
protectedvoid reduce(IntWritable itemID,Iterable<VectorOrPrefWritable> values,Context context)
throwsIOException,InterruptedException{
VectorAndPrefsWritable vectorAndPrefsWritable =newVectorAndPrefsWritable();
List<Long> userIDs =newArrayList<Long>();
List<Float> preferenceValues =newArrayList<Float>();
Vector coocVector =null;
Vector coocVectorTemp =null;
Iterator<VectorOrPrefWritable> it = values.iterator();
while(it.hasNext()){
VectorOrPrefWritable e = it.next();
coocVectorTemp = e.getVector();
if(coocVectorTemp ==null){
userIDs.add(e.getUserID());
preferenceValues.add(e.getValue());
}else{
coocVector = coocVectorTemp;
}
}
if(coocVector !=null){
//这个需要注意，根据共现矩阵的计算reduce聚合之后，到了这个一个Reudce分组就有且只有一个vecotr(即共现矩阵的一列或者一行，这里行和列是一样的)了。
vectorAndPrefsWritable.set(coocVector, userIDs, preferenceValues);
logger.info(itemID+" "+vectorAndPrefsWritable);
context.write(itemID, vectorAndPrefsWritable);
}
}
}

第五步：将combineUserVectorAndCoocMatrixJob作业的输出结果生成推荐列表

caclPartialRecomUserVectorJob作业的mapper：CaclPartialRecomUserVectorMapper

package com.mapper;
import java.io.IOException;
import java.util.List;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.mahout.cf.taste.hadoop.item.VectorAndPrefsWritable;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 计算部分用户推荐向量
* @author 曾昭正
*/
publicclassCaclPartialRecomUserVectorMapperextendsMapper<IntWritable,VectorAndPrefsWritable,VarLongWritable,VectorWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(CaclPartialRecomUserVectorMapper.class);
@Override
protectedvoid map(IntWritable itemID,VectorAndPrefsWritable values,Context context)
throwsIOException,InterruptedException{
Vector coocVectorColumn = values.getVector();
List<Long> userIDs = values.getUserIDs();
List<Float> preferenceValues = values.getValues();
for(int i =0; i< userIDs.size(); i++){
long userID = userIDs.get(i);
float preferenceValue = preferenceValues.get(i);
logger.info("userID:"+ userID);
logger.info("preferenceValue:"+preferenceValue);
//将共现矩阵中userID对应的列相乘，算出部分用户对应的推荐列表分数
Vector preferenceParScores = coocVectorColumn.times(preferenceValue);
context.write(newVarLongWritable(userID),newVectorWritable(preferenceParScores));
}
}
}

caclPartialRecomUserVectorJob作业的combiner：ParRecomUserVectorCombiner

package com.reducer;
import java.io.IOException;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.VectorWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 将计算部分用户推荐向量的结果进行合并，将userID对应的贡现向量的分值进行相加(注意：这个只是将一个map的输出进行合并，所以这个也是只部分结果)
* @author 曾昭正
*/
publicclassParRecomUserVectorCombinerextendsReducer<VarLongWritable,VectorWritable,VarLongWritable,VectorWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(ParRecomUserVectorCombiner.class);
@Override
protectedvoid reduce(VarLongWritable userID,Iterable<VectorWritable> coocVectorColunms,Context context)
throwsIOException,InterruptedException{
Vector vectorColunms =null;
for(VectorWritable coocVectorColunm : coocVectorColunms){
vectorColunms = vectorColunms ==null? coocVectorColunm.get(): vectorColunms.plus(coocVectorColunm.get());
}
logger.info(userID +" "+newVectorWritable(vectorColunms));
context.write(userID,newVectorWritable(vectorColunms));
}
}

caclPartialRecomUserVectorJob作业的reducer：MergeAndGenerateRecommendReducer

package com.reducer;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.PriorityQueue;
import java.util.Queue;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.mahout.cf.taste.hadoop.RecommendedItemsWritable;
import org.apache.mahout.cf.taste.impl.recommender.ByValueRecommendedItemComparator;
import org.apache.mahout.cf.taste.impl.recommender.GenericRecommendedItem;
import org.apache.mahout.cf.taste.recommender.RecommendedItem;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.Vector;
import org.apache.mahout.math.Vector.Element;
import org.apache.mahout.math.VectorWritable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* 合并所有已经评分的共现矩阵
* @author 曾昭正
*/
publicclassMergeAndGenerateRecommendReducerextendsReducer<VarLongWritable,VectorWritable,VarLongWritable,RecommendedItemsWritable>{
privatestaticfinalLogger logger =LoggerFactory.getLogger(MergeAndGenerateRecommendReducer.class);
privateint recommendationsPerUser;
@Override
protectedvoid setup(Context context)
throwsIOException,InterruptedException{
recommendationsPerUser = context.getConfiguration().getInt("recomandItems.recommendationsPerUser",5);
}
@Override
protectedvoid reduce(VarLongWritable userID,Iterable<VectorWritable> cooVectorColumn,Context context)
throwsIOException,InterruptedException{
//分数求和合并
Vector recommdVector =null;
for(VectorWritable vector : cooVectorColumn){
recommdVector = recommdVector ==null? vector.get(): recommdVector.plus(vector.get());
}
//对推荐向量进行排序，为每个UserID找出topM个推荐选项(默认找出5个)，此队列按照item对应的分数进行排序
//注意下：PriorityQueue队列的头一定是最小的元素,另外这个队列容量增加1是为了为添加更大的新元素时使用的临时空间
Queue<RecommendedItem> topItems =newPriorityQueue<RecommendedItem>(recommendationsPerUser+1,ByValueRecommendedItemComparator.getInstance());
Iterator<Element> it = recommdVector.nonZeroes().iterator();
while(it.hasNext()){
Element e = it.next();
int itemID = e.index();
float preValue =(float) e.get();
//当队列容量小于推荐个数，往队列中填item和分数
if(topItems.size()< recommendationsPerUser){
topItems.add(newGenericRecommendedItem(itemID, preValue));
}
//当前item对应的分数比队列中的item的最小分数大，则将队列头原始（即最小元素）弹出，并且将当前item：分数加入队列
elseif(preValue > topItems.peek().getValue()){
topItems.add(newGenericRecommendedItem(itemID, preValue));
//弹出头元素（最小元素）
topItems.poll();
}
}
//重新调整队列的元素的顺序
List<RecommendedItem> recommdations =newArrayList<RecommendedItem>(topItems.size());
recommdations.addAll(topItems);//将队列中所有元素添加即将排序的集合
Collections.sort(recommdations,ByValueRecommendedItemComparator.getInstance());//排序
//输出推荐向量信息
logger.info(userID+" "+newRecommendedItemsWritable(recommdations));
context.write(userID,newRecommendedItemsWritable(recommdations));
}
}

第六步：组装各个作业关系

PackageRecomendJob

package com.mapreduceMain;
import java.io.IOException;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.mahout.cf.taste.hadoop.RecommendedItemsWritable;
import org.apache.mahout.cf.taste.hadoop.item.VectorAndPrefsWritable;
import org.apache.mahout.cf.taste.hadoop.item.VectorOrPrefWritable;
import org.apache.mahout.math.VarLongWritable;
import org.apache.mahout.math.VectorWritable;
import com.mapper.CaclPartialRecomUserVectorMapper;
import com.mapper.CombineUserVectorAndCoocMatrixMapper;
import com.mapper.UserVecotrSplitMapper;
import com.mapper.UserVectorToCooccurrenceMapper;
import com.mapper.SourceDataToItemPrefsMapper;
import com.reducer.CombineUserVectorAndCoocMatrixReducer;
import com.reducer.MergeAndGenerateRecommendReducer;
import com.reducer.ParRecomUserVectorCombiner;
import com.reducer.UserVectorToCoocurrenceReducer;
import com.reducer.SourceDataToUserVectorReducer;
import com.util.HadoopUtil;
/**
* 组装各个作业组件，完成推荐作业
* @author 曾昭正
*/
publicclassPackageRecomendJobextendsConfiguredimplementsTool{
String[] dataSourceInputPath ={"/user/hadoop/z.zeng/distruteItemCF/dataSourceInput"};
String[] uesrVectorOutput ={"/user/hadoop/z.zeng/distruteItemCF/uesrVectorOutput/"};
String[] userVectorSpliltOutPut ={"/user/hadoop/z.zeng/distruteItemCF/userVectorSpliltOutPut"};
String[] cooccurrenceMatrixOuptPath ={"/user/hadoop/z.zeng/distruteItemCF/CooccurrenceMatrixOuptPath"};
String[] combineUserVectorAndCoocMatrixOutPutPath ={"/user/hadoop/z.zeng/distruteItemCF/combineUserVectorAndCoocMatrixOutPutPath"};
String[] caclPartialRecomUserVectorOutPutPath ={"/user/hadoop/z.zeng/distruteItemCF/CaclPartialRecomUserVectorOutPutPath"};
protectedvoid setup(Configuration configuration)
throwsIOException,InterruptedException{
FileSystem hdfs =FileSystem.get(URI.create("hdfs://cluster-master"), configuration);
Path p1 =newPath(uesrVectorOutput[0]);
Path p2 =newPath(userVectorSpliltOutPut[0]);
Path p3 =newPath(cooccurrenceMatrixOuptPath[0]);
Path p4 =newPath(combineUserVectorAndCoocMatrixOutPutPath[0]);
Path p5 =newPath(caclPartialRecomUserVectorOutPutPath[0]);
if(hdfs.exists(p1)){
hdfs.delete(p1,true);
}
if(hdfs.exists(p2)){
hdfs.delete(p2,true);
}
if(hdfs.exists(p3)){
hdfs.delete(p3,true);
}
if(hdfs.exists(p4)){
hdfs.delete(p4,true);
}
if(hdfs.exists(p5)){
hdfs.delete(p5,true);
}
}
@Override
publicint run(String[] args)throwsException{
Configuration conf=getConf();//获得配置文件对象
setup(conf);
// DistributedCache.addArchiveToClassPath(new Path("/user/hadoop/z.zeng/distruteItemCF/lib"), conf);
//配置计算用户向量作业
Job wikipediaToItemPrefsJob =HadoopUtil.prepareJob(
"WikipediaToItemPrefsJob",
dataSourceInputPath,
uesrVectorOutput[0],
TextInputFormat.class,
SourceDataToItemPrefsMapper.class,
VarLongWritable.class,
VarLongWritable.class,
SourceDataToUserVectorReducer.class,
VarLongWritable.class,
VectorWritable.class,
SequenceFileOutputFormat.class,
conf);
//配置计算共现向量作业
Job userVectorToCooccurrenceJob =HadoopUtil.prepareJob(
"UserVectorToCooccurrenceJob",
uesrVectorOutput,
cooccurrenceMatrixOuptPath[0],
SequenceFileInputFormat.class,
UserVectorToCooccurrenceMapper.class,
IntWritable.class,
IntWritable.class,
UserVectorToCoocurrenceReducer.class,
IntWritable.class,
VectorOrPrefWritable.class,
SequenceFileOutputFormat.class,
conf);
//配置分割用户向量作业
Job userVecotrSplitJob =HadoopUtil.prepareJob(
"userVecotrSplitJob",
uesrVectorOutput,
userVectorSpliltOutPut[0],
SequenceFileInputFormat.class,
UserVecotrSplitMapper.class,
IntWritable.class,
VectorOrPrefWritable.class,
SequenceFileOutputFormat.class,
conf);
//合并共现向量和分割之后的用户向量作业
//这个主意要将分割用户向量和共现向量的输出结果一起作为输入
String[] combineUserVectorAndCoocMatrixIutPutPath ={cooccurrenceMatrixOuptPath[0],userVectorSpliltOutPut[0]};
Job combineUserVectorAndCoocMatrixJob =HadoopUtil.prepareJob(
"combineUserVectorAndCoocMatrixJob",
combineUserVectorAndCoocMatrixIutPutPath,
combineUserVectorAndCoocMatrixOutPutPath[0],
SequenceFileInputFormat.class,
CombineUserVectorAndCoocMatrixMapper.class,
IntWritable.class,
VectorOrPrefWritable.class,
CombineUserVectorAndCoocMatrixReducer.class,
IntWritable.class,
VectorAndPrefsWritable.class,
SequenceFileOutputFormat.class,
conf);
//计算用户推荐向量
Job caclPartialRecomUserVectorJob=HadoopUtil.prepareJob(
"caclPartialRecomUserVectorJob",
combineUserVectorAndCoocMatrixOutPutPath,
caclPartialRecomUserVectorOutPutPath[0],
SequenceFileInputFormat.class,
CaclPartialRecomUserVectorMapper.class,
VarLongWritable.class,
VectorWritable.class,
ParRecomUserVectorCombiner.class,//为map设置combiner减少网络IO
MergeAndGenerateRecommendReducer.class,
VarLongWritable.class,
RecommendedItemsWritable.class,
TextOutputFormat.class,
conf);
//串联各个job
if(wikipediaToItemPrefsJob.waitForCompletion(true)){
if(userVectorToCooccurrenceJob.waitForCompletion(true)){
if(userVecotrSplitJob.waitForCompletion(true)){
if(combineUserVectorAndCoocMatrixJob.waitForCompletion(true)){
int rs = caclPartialRecomUserVectorJob.waitForCompletion(true)?1:0;
return rs;
}else{
thrownewException("合并共现向量和分割之后的用户向量作业失败！！");
}
}else{
thrownewException("分割用户向量作业失败！！");
}
}else{
thrownewException("计算共现向量作业失败！！");
}
}else{
thrownewException("计算用户向量作业失败！！");
}
}
publicstaticvoid main(String[] args)throwsIOException,
ClassNotFoundException,InterruptedException{
try{
int returnCode =ToolRunner.run(newPackageRecomendJob(),args);
System.exit(returnCode);
}catch(Exception e){
}
}
}

五、总结

本blog主要说了下itemBase推荐算法的一些概念，以及如何多现有数据进行建模。其中对共现矩阵方式的推荐用MapReduce结合Mahout的内置数据类型进行了实现。写完这篇blog和对算法实现完毕后，发现Mapreduce编程虽然数据模型非常简单，只有2个过程：数据的分散与合并，但是在分散与合并的过程中可以使用自定义的各种数据组合类型使其能够完成很多复杂的功能。

参考文献：《Mahout in action》、《推荐引擎实践》

来自为知笔记(Wiz)

（转）基于MapReduce的ItemBase推荐算法的共现矩阵实现（一）

标签：style blog http io ar color os 使用 sp

原文地址：http://www.cnblogs.com/baixl/p/4165156.html

踩

(0)

评论一句话评论（0）

分享档案

更多>

2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)

周排行

（转） 基于MapReduce的ItemBase推荐算法的共现矩阵实现（一）

（转）基于MapReduce的ItemBase推荐算法的共现矩阵实现（一）