码迷,mamicode.com
首页 > Web开发 > 详细

spark streaming集成flume

时间:2017-11-26 11:09:58      阅读:253      评论:0      收藏:0      [点我收藏+]

标签:配置文件   界面   logger   令行   修改   因此   doc   only   put   

1、 安装flume

flume安装,解压后修改flume_env.sh配置文件,指定java_home即可。

cp hdfs jar包到flume lib目录下(否则无法抽取数据到hdfs上):

$ cp /opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.6.jar

/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/lib/

2、 spark streaming集成flume

2.1)编译spark,获得集成flume jar包:

参考文档:http://www.cnblogs.com/wcwen1990/p/7688027.html

说明:spark streaming集成flume或者kafka需要一些支持jar包,这些jar包在编译spark过程中会自动在external目录下生成相应的jar文件,因此,这里需要编译spark来获得这些jar包。

Spark streaming集成flume主要需要:spark-streaming-flume_2.10-1.3.0.jar包。

2.2)集成jar包

$mkdir –p /opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs

$cp spark-1.3.0/external/flume/target/spark-streaming-flume_2.10-1.3.0.jar

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ pwd

/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/lib

$ cp flume-avro-source-1.5.0-cdh5.3.6.jar flume-ng-sdk-1.5.0-cdh5.3.6.jar

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ cd /opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/

$ ll

flume-avro-source-1.5.0-cdh5.3.6.jar

flume-ng-sdk-1.5.0-cdh5.3.6.jar

spark-streaming-flume_2.10-1.3.0.jar

3、 编译flume配置文件(配置sources、channel、sink):

$ cat flume-spark-push.conf

ss.sources = sss

ss.channels = ssc

ss.sinks = ssk

ss.sources.sss.type = exec

ss.sources.sss.command = tail -f /opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/wctotal.log

ss.sources.sss.shell = /bin/bash -c

ss.channels.ssc.type = memory

ss.channels.ssc.capacity = 1000

ss.channels.ssc.transactionCapacity = 100

ss.sinks.ssk.type = avro

ss.sinks.ssk.hostname = chavin.king

ss.sinks.ssk.port = 9999

ss.sources.sss.channels = ssc

ss.sinks.ssk.channel = ssc

4、 编写spark streaming程序:

import org.apache.spark._

import org.apache.spark.streaming._

import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.flume._

import org.apache.spark.storage.StorageLevel

val ssc = new StreamingContext(sc, Seconds(5))

// read data

val stream = FlumeUtils.createStream(ssc, "chavin.king", 9999, StorageLevel.MEMORY_ONLY_SER_2)

stream.count().map(cnt => "Received " + cnt + " flume events." ).print()

ssc.start() // Start the computation

ssc.awaitTermination() // Wait for the computation to terminate

5、 spark-shell local模式测试spark streaming集成flume

$ bin/spark-shell --master local[2] --jars \

/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/spark-streaming-flume_2.10-1.3.0.jar,/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/flume-avro-source-1.5.0-cdh5.3.6.jar,/opt/cdh-5.3.6/spark-1.3.0-bin-2.5.0-cdh5.3.6/externalLibs/flume-ng-sdk-1.5.0-cdh5.3.6.jar

执行步骤4中程序:

scala> import org.apache.spark._

import org.apache.spark._

scala> import org.apache.spark.streaming._

import org.apache.spark.streaming._

scala> import org.apache.spark.streaming.StreamingContext._

import org.apache.spark.streaming.StreamingContext._

scala> import org.apache.spark.streaming.flume._

import org.apache.spark.streaming.flume._

scala> import org.apache.spark.storage.StorageLevel

import org.apache.spark.storage.StorageLevel

scala> val ssc = new StreamingContext(sc, Seconds(5))

ssc: org.apache.spark.streaming.StreamingContext = org.apache.spark.streaming.StreamingContext@412dea3c

scala> val stream = FlumeUtils.createStream(ssc, "chavin.king", 9999, StorageLevel.MEMORY_ONLY_SER_2)

stream: org.apache.spark.streaming.dstream.ReceiverInputDStream[org.apache.spark.streaming.flume.SparkFlumeEvent] = org.apache.spark.streaming.flume.FlumeInputDStream@2bf9884

scala> stream.count().map(cnt => "Received " + cnt + " flume events." ).print()

//输入以下命令启动spark streaming

scala> ssc.start()

scala> ssc.awaitTermination()

6、 启动flume

bin/flume-ng agent -c conf -n ss -f conf/flume-spark-push.conf -Dflume.root.logger=DEBUG,console

7、 测试spark streaming集成flume:

$ echo “hadoop oracle mysql” >>/opt/cdh-5.3.6/flume-1.5.0-cdh5.3.6-bin/wctotal.log

执行上边命令,可以在spark streaming命令行界面下看到如下内容:

-------------------------------------------

Time: 1499976790000 ms

-------------------------------------------

Received 1 flume events.

8、参考文档:http://spark.apache.org/docs/1.3.0/streaming-flume-integration.html

spark streaming集成flume

标签:配置文件   界面   logger   令行   修改   因此   doc   only   put   

原文地址:http://www.cnblogs.com/wcwen1990/p/7898168.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!