码迷,mamicode.com
首页 > 其他好文 > 详细

【慕课网实战】Spark Streaming实时流处理项目实战笔记十八之铭文升级版

时间:2018-02-03 00:49:42      阅读:371      评论:0      收藏:0      [点我收藏+]

标签:tor   sea   project   version   users   class   r.java   提交   div   

铭文一级:

功能二:功能一+从搜索引擎引流过来的


HBase表设计
create ‘imooc_course_search_clickcount‘,‘info‘
rowkey设计:也是根据我们的业务需求来的

20171111 +search+ 1


项目打包:mvn clean package -DskipTests

报错:
[ERROR] /Users/rocky/source/work/sparktrain/src/main/scala/com/imooc/spark/project/dao/CourseClickCountDAO.scala:4: error: object HBaseUtils is not a member
of package com.imooc.spark.project.utils

 

spark-submit --master local[5] \
--class com.imooc.spark.project.spark.ImoocStatStreamingApp \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000:2181 test streamingtopic 1


报错:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils$
at com.imooc.spark.project.spark.ImoocStatStreamingApp$.main(ImoocStatStreamingApp.scala:31)
at com.imooc.spark.project.spark.ImoocStatStreamingApp.main(ImoocStatStreamingApp.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtils$
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more


spark-submit --master local[5] \
--class com.imooc.spark.project.spark.ImoocStatStreamingApp \
--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000:2181 test streamingtopic 1

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/HBaseAdmin
at com.imooc.spark.project.utils.HBaseUtils.<init>(HBaseUtils.java:30)
at com.imooc.spark.project.utils.HBaseUtils.getInstance(HBaseUtils.java:40)
at com.imooc.spark.project.dao.CourseClickCountDAO$.save(CourseClickCountDAO.scala:26)
at com.imooc.spark.project.spark.ImoocStatStreamingApp$$anonfun$main$4$$anonfun$apply$1.a


spark-submit --master local[5] \
--jars $(echo /home/hadoop/app/hbase-1.2.0-cdh5.7.0/lib/*.jar | tr ‘ ‘ ‘,‘) \
--class com.imooc.spark.project.spark.ImoocStatStreamingApp \
--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000:2181 test streamingtopic 1

 

提交作业时,注意事项:
1)--packages的使用
2)--jars的使用

 

铭文二级:

功能二:功能一+从搜索引擎引流过来的

建表:create ‘imooc_course_search_clickcount‘,‘info‘

删除表的内容:truncate ‘imooc_course_search_clickcount‘

 

打包到服务器运行:

注意点1.打包 2.--packages 3.--jars 

1、java与scala混合打包,所以要去掉pom.xml里的<build>下的SourceDirectory与testSourceDirectory

否则:mvn clean package -DskipTests 会报错,也可以maven project栏右键打包

2、--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.1

格式:分别是pom.xml里面的groupId、Arcticid、spark-version(用分号分开)

3、--jars(一个一个导包太繁琐,所以可以采用这种方式)

--jars $(echo /home/hadoop/app/hbase-1.2.0-cdh5.7.0/lib/*.jar | tr ‘ ‘ ‘,‘)

如有多个hbase.submit进程,可以杀掉

 

第13章 可视化实战

课程目录:

为什么要可视化

Echarts搭建静态可视化与动态可视化

阿里云DataV数据可视化(炫酷)

【慕课网实战】Spark Streaming实时流处理项目实战笔记十八之铭文升级版

标签:tor   sea   project   version   users   class   r.java   提交   div   

原文地址:https://www.cnblogs.com/kkxwz/p/8407444.html

(0)
(1)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!