码迷,mamicode.com
首页 > 其他好文 > 详细

RDD常用方法之subtract&intersection&cartesian

时间:2015-03-04 18:20:49      阅读:151      评论:0      收藏:0      [点我收藏+]

标签:

subtract
Return an RDD with the elements from `this` that are not in `other` .    
def subtract(other: RDD[T]): RDD[T]
def subtract(other: RDD[T], numPartitions: Int): RDD[T]
def subtract(other: RDD[T], p: Partitioner): RDD[T]
val a = sc.parallelize(1 to 5)
val b = sc.parallelize(1 to 3)
val c = a.subtract(b)
c.collect
 Array[Int] = Array(4, 5)

 

 
intersection
Return the intersection of this RDD and another one.  The output will not contain any duplicate elements, even if the input RDDs did.   交集
def intersection(other: RDD[T], numPartitions: Int): RDD[T]
def intersection(other: RDD[T], partitioner: Partitioner)(implicit ord: Ordering[T] = null): RDD[T]
def intersection(other: RDD[T]): RDD[T]
val x = sc.parallelize(1 to 10)
val y = sc.parallelize(2 to 8)
val z = x.intersection(y)
z.collect
 Array[Int] = Array(4, 6, 8, 2, 3, 7, 5)

 

cartesian
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in `this` and b is in `other` .   笛卡尔积
def cartesian[U: ClassTag](other: RDD[U]): RDD[(T, U)] 
val x = sc.parallelize(List(1,2,3))
val y = sc.parallelize(List(6,7,8))
x.cartesian(y).collect
 Array[(Int, Int)] = Array((1,6), (1,7), (1,8), (2,6), (3,6), (2,7), (2,8), (3,7), (3,8))

 

RDD常用方法之subtract&intersection&cartesian

标签:

原文地址:http://www.cnblogs.com/luogankun/p/4313645.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!