Spark关键类

标签：

• sparkEnv

  (D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\SparkEnv.scala)

Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext).

• Block manager

  (D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\storage\BlockManager.scala)

 Manager running on every node (driver and executors) which provides interfaces for putting and retrieving blocks both locally and remotely into various stores (memory, disk, and off-heap).

 Note that #initialize() must be called before the BlockManager is usable.

• SparkConf

  (D:\Spark\SourceCode\spark\core\src\main\scala\org\apache\spark\SparkConf.scala)

1. Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
2. Most of the time, you would create a SparkConf object with `new SparkConf()`, which will load values from any `spark.*` Java system properties set in your application as well. In this case,parameters you set directly on the `SparkConf` object take priority over system properties.
3. For unit tests, you can also call `new SparkConf(false)` to skip loading external settings and get the same configuration no matter what the system properties are.
4. All setter methods in this class support chaining. For example, you can write `new SparkConf().setMaster("local").setAppName("My app")`.
5. Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.
6. setMaster(): The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run o``n a Spark standalone cluster.
7. setAppName(): Set a name for your application. Shown in the Spark web UI

Spark关键类

标签：

原文地址：http://www.cnblogs.com/shexinwei/p/4671727.html