Spark Configuration (2.2.0) 笔记

标签：ati   scala   bugs   parallel   only   represent   present   script   machine   

Spark的配置项分为三部分

• Spark properties control most application parameters and can be set by using a SparkConf object, or through Java system properties.
• Environment variables can be used to set per-machine settings, such as the IP address, through the conf/spark-env.sh script on each node.
• Logging can be configured through log4j.properties.

Spark properities

Note that we run with local[2], meaning two threads - which represents “minimal” parallelism, which can help detect bugs that only exist when we run in a distributed context.

Spark Configuration (2.2.0) 笔记

标签：ati   scala   bugs   parallel   only   represent   present   script   machine   

原文地址：http://www.cnblogs.com/zcy-backend/p/7552779.html