码迷,mamicode.com
首页 > 其他好文 > 详细

Spark Streaming之旅

时间:2014-06-28 23:07:17      阅读:324      评论:0      收藏:0      [点我收藏+]

标签:style   blog   java   color   os   art   

1. 打开spark-shell

2. 建立StreamingContext
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.api.java.function._
import org.apache.spark.streaming._
import org.apache.spark.streaming.api._
// Create a StreamingContext with a local master
val ssc = new StreamingContext(sc, Seconds(1))

3.用StreamingContext建立一个DStream

val lines = ssc.socketTextStream("localhost", 9999)

4.运算

// Split each line into words
val words = lines.flatMap(_.split(" "))

import org.apache.spark.streaming.StreamingContext._
// Count each word in each batch
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)

// Print a few of the counts to the console
wordCounts.print()

5.启动DStream

ssc.start()             // Start the computation
ssc.awaitTermination()  // Wait for the computation to terminate

6.启动一个新控制台,并执行如下命令

$ nc -lk 9999

7.在新控制台上输入任意语句,并在spark-shell窗口观察输出

hello world how are you hello are you there

you are the son of a bitch

 



Spark Streaming之旅,布布扣,bubuko.com

Spark Streaming之旅

标签:style   blog   java   color   os   art   

原文地址:http://www.cnblogs.com/littlesuccess/p/3794225.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!