spark开发程序编写注意点

标签：spark   scala   

看书时发现书中写的不一定是正确的。

eg：

scala> val line = sc.textFile("hdfs://Spark:9000/user/root/README.md")
15/03/19 20:03:04 INFO MemoryStore: ensureFreeSpace(202004) called with curMem=744765, maxMem=280248975
15/03/19 20:03:04 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 197.3 KB, free 266.4 MB)
15/03/19 20:03:04 INFO MemoryStore: ensureFreeSpace(16322) called with curMem=946769, maxMem=280248975
15/03/19 20:03:04 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 15.9 KB, free 266.3 MB)
15/03/19 20:03:04 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on a261.datanode.hadoop.qingdao.youku:46352 (size: 15.9 KB, free: 267.2 MB)
15/03/19 20:03:04 INFO BlockManagerMaster: Updated info of block broadcast_4_piece0
line: org.apache.spark.rdd.RDD[String] = hdfs://Spark:9000/user/root/README.md MappedRDD[11] at textFile at <console>:16

scala> val linenum = line.filter(x=>x.contains("spark"))
linenum: org.apache.spark.rdd.RDD[String] = FilteredRDD[12] at filter at <console>:18

       其实求出的是“spark”这个字符串出现在多少行当中；一行中可能有多个“spark”字符串，切不可认为是“spark”字符串在文章中一共出现了几次。

spark开发程序编写注意点

标签：spark   scala   

原文地址：http://blog.csdn.net/u013361361/article/details/44463563