标签:intern hdfs 分发 两种 rri art asi imm win
1.spark核心模块是整个项目的基础。提供了分布式的任务分发,调度以及基本的IO功能,Spark使用基础的数据结构,叫做RDD(弹性分布式数据集),是一个逻辑的数据分区的集合,可以跨机器。RDD可以通过两种方式进行创建,一种是从外部的数据集引用数据,第二种方式是通过在现有的RDD上做数据转换。RDD抽象是通过语言集成的API来进行暴露,它简化了编程的复杂度,因为这种操纵RDD的方式类似于操纵本地数据集合
** * A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, * partitioned collection of elements that can be operated on in parallel. This class contains the * basic operations available on all RDDs, such as `map`, `filter`, and `persist`. In addition, * [[org.apache.spark.rdd.PairRDDFunctions]] contains operations available only on RDDs of key-value * pairs, such as `groupByKey` and `join`; * [[org.apache.spark.rdd.DoubleRDDFunctions]] contains operations available only on RDDs of * Doubles; and * [[org.apache.spark.rdd.SequenceFileRDDFunctions]] contains operations available on RDDs that * can be saved as SequenceFiles. * All operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] * through implicit. * * Internally, each RDD is characterized by five main properties: * * - A list of partitions * - A function for computing each split * - A list of dependencies on other RDDs * - Optionally, a Partitioner for key-value RDDs (e.g. to say that the RDD is hash-partitioned) * - Optionally, a list of preferred locations to compute each split on (e.g. block locations for * an HDFS file) * * All of the scheduling and execution in Spark is done based on these methods, allowing each RDD * to implement its own way of computing itself. Indeed, users can implement custom RDDs (e.g. for * reading data from a new storage system) by overriding these functions. Please refer to the * <a href="http://people.csail.mit.edu/matei/papers/2012/nsdi_spark.pdf">Spark paper</a> * for more details on RDD internals. */
1.RDD变换返回一个指向新RDD的指针并且允许你在RDD之间创建依赖,在依赖链条中的每个RDD都有一个计算数据的函数以及一个指向父RDD的指针。Spark是懒惰的,所以除非你调用一些除法任务创建以及执行的转换或者Action,否则什么都不干。
因此RDD变换不是一个数据集,而是在一个程序中的一个步骤,用来告诉如何获取数据以及怎么进行数据的相关的处理。
2.下面给出的是一个RDD变换列表
(1)map(func):返回一个分布式数据集,通过对每一个函数应用func函数形成。
(2)flatMap(func):与map函数相似,但是每个输入项可以被映射为0个或者多个输出项(所以func函数应该返回一个Seq而不是一个单独的数据项)
标签:intern hdfs 分发 两种 rri art asi imm win
原文地址:https://www.cnblogs.com/bigdata-stone/p/9936622.html