标签：rgs image www source div rop mave maven打包 font
我习惯用Maven项目 所以用IDEA新建一个Maven项目
下面是pom文件 我粘上来吧
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com</groupId>
<artifactId>ScalaMavenTest2</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.11.8</scala.version>
<scala.compat.version>2.11</scala.compat.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.6.5</version>
</dependency>
</dependencies>
<!-- 指定插件-->
<build>
<plugins>
<!--编译java的插件-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
</plugin>
<!-- 指定编译scala的插件 -->
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
</plugin>
<!-- maven打包的插件 -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
<!-- 指定main方法 -->
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass></mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
代码如下
import org.apache.spark.{SparkConf, SparkContext} object WordCount { def main(args: Array[String]): Unit = { //设置配置文件和名字 val conf = new SparkConf().setAppName("wordConut") //生成Spark的上下文 val sc = new SparkContext(conf) //传入文件 val rdd = sc.textFile(args(0)) //先用split()按照空格进行分词 在通过flatMap对分割的单词进行展评,展评完毕后 使用map(x=>(x,1))对每个单词 //计数1最后使用ReduceByKey(_+_) 根据Key也就是单词进行计数 这是一个Shuffer过程 val wordcount = rdd.flatMap(_.split(" ")).map(x=>(x,1)).reduceByKey(_+_) //先使用map(x=>(x._2,x._1))对单词结果的k V 进行转换然后通过sortByKey(false)根据K V也就是词频进行排序 false要求 //按照倒序进行排列,最后再次通过map(x=>(x._2,x._1))让 K V 再次进行互换 形成最后结果 val wordsort = wordcount.map(x=>(x._2,x._1)).sortByKey(false).map(x=>(x._2,x._1)) //写出路径 wordsort.saveAsTextFile(args(1)) sc.stop() } }
之后设置参数 指定master 和文件所在目录 和输出目录
成功之后的结果
标签：rgs image www source div rop mave maven打包 font
原文地址：https://www.cnblogs.com/xiaocan66/p/9451250.html
