标签：als 统计 div img put col ber 技术分享 for
Part I：词频统计并返回topN
统计的文本数据：
what do you do how do you do how do you do how are you
from operator import add from pyspark import SparkContext def sort_t(): sc = SparkContext(appName="testWC") data = sc.parallelize(["what do you do", "how do you do", "how do you do", "how are you"]) result = data.flatMap(lambda x: x.split(" ")) .map(lambda x: (x, 1)). reduceByKey(add). sortBy(lambda x: x[1], False).take(3) for k, v in result: print k, v if __name__ == ‘__main__‘: sort_t()
Part II：调用排序算法并返回topN
样本数据 numbers_data.txt：
15561 112 -40 51467112 234 8561 112 -34 53467111 121 2345 789 34 14561 -21 12112 101 100 -4 23 51467111 2434 15567 132 -14 51467111 237
from pyspark import SparkContext def solve(): sc = SparkContext(appName="Sort_test_example") lines = sc.textFile("../input/numbers_data.txt") results = lines.flatMap(lambda x: x.split(" ")) .map(lambda x: (int(x), 1)) .sortByKey(ascending=False).take(3) output = results for (key, value) in output: print key print key if __name__ == ‘__main__‘:
solve()
注：若出现并列时，返回多个并列的数
标签：als 统计 div img put col ber 技术分享 for
原文地址：http://www.cnblogs.com/SeaSky0606/p/7762703.html
