标签：spark graphx aggregatemessages 图计算
Spark Graphx图计算案例实战之aggregateMessages求社交网络中的最大年纪追求者和平均年纪！
Spark Graphx提供了mapReduceTriplets来对图进行聚合计算，但是1.2以后不再推荐使用，源代码如下：
@deprecated("use aggregateMessages", "1.2.0")
def mapReduceTriplets[A: ClassTag](
mapFunc: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)],
reduceFunc: (A, A) => A,
activeSetOpt: Option[(VertexRDD[_], EdgeDirection)] = None)
: VertexRDD[A]* Aggregates values from the neighboring edges and vertices of each vertex. The user supplied
* `mapFunc` function is invoked on each edge of the graph, generating 0 or more "messages" to be
* "sent" to either vertex in the edge. The `reduceFunc` is then used to combine the output of
* the map phase destined to each vertex.
*
* This function is deprecated in 1.2.0 because of SPARK-3936.
Use aggregateMessages instead.
*
推荐使用的是aggregateMessages：
def aggregateMessages[A: ClassTag](
sendMsg: EdgeContext[VD, ED, A] => Unit,
mergeMsg: (A, A) => A,
tripletFields: TripletFields = TripletFields.All)
: VertexRDD[A] = {
aggregateMessagesWithActiveSet(sendMsg, mergeMsg, tripletFields, None)
}并举了一个简单的例子：
* vertex
* {{{
* val rawGraph: Graph[_, _] = Graph.textFile("twittergraph")
* val inDeg: RDD[(VertexId, Int)] =
* rawGraph.aggregateMessages[Int](ctx => ctx.sendToDst(1), _ + _)
* }}}
可以看见能够进行消息传递和聚合操作。
案例实战：求社交网络中的年纪最大的追求者和追求者的平均年龄：
val oldestFollower: VertexRDD[(String,Int)]=userGraph.aggregateMessages[(String, Int)](
triplet => {
triplet.sendToDst(triplet.srcAttr.name, triplet.srcAttr.age)
},
(a, b) => if (a._2 > b._2) a else b
)
oldestFollower.collect.foreach(println(_))
averageAge: VertexRDD[] = userGraph.aggregateMessages[()](
triplet => {
triplet.sendToDst(triplet.srcAttr.age)
}(ab) => ((a._1 + b._1)(a._2 + b._2))
).mapValues((idp) => p._2 / p._1)
averageAge.collect().foreach((_))很好很强大啊！
结果如下：
聚合操作
**********************************************************
找出年纪最大的追求者：
(4,(Bob,27))
(1,(David,42))
(6,(Charlie,65))
(2,(Charlie,65))
(3,(Ed,55))
**********************************************************
找出追求者的平均年纪：
(4,27.0)
(1,34.5)
(6,60.0)
(2,60.0)
(3,55.0)
**********************************************************
源码是最好的学习素材！
王家林老师DT大数据梦工厂学习之路！
Spark Graphx图计算案例实战之aggregateMessages求社交网络中的最大年纪追求者和平均年纪！
标签：spark graphx aggregatemessages 图计算
原文地址：http://36006798.blog.51cto.com/988282/1873861
