码迷,mamicode.com
首页 > 其他好文 > 详细

Spark分析之Master

时间:2014-07-01 20:21:15      阅读:251      评论:0      收藏:0      [点我收藏+]

标签:style   blog   http   color   strong   art   

override def preStart() {
    logInfo("Starting Spark master at " + masterUrl)
    webUi.bind()    //绑定WEBUI
    masterWebUiUrl = "http://" + masterPublicAddress + ":" + webUi.boundPort
    context.system.scheduler.schedule(0 millis, WORKER_TIMEOUT millis, self, CheckForWorkerTimeOut) //定时检查超时未发送心跳的Worker

    masterMetricsSystem.registerSource(masterSource)
    masterMetricsSystem.start()
    applicationMetricsSystem.start()

//用作master recover persistenceEngine
= RECOVERY_MODE match { case "ZOOKEEPER" => logInfo("Persisting recovery state to ZooKeeper") new ZooKeeperPersistenceEngine(SerializationExtension(context.system), conf) case "FILESYSTEM" => logInfo("Persisting recovery state to directory: " + RECOVERY_DIR) new FileSystemPersistenceEngine(RECOVERY_DIR, SerializationExtension(context.system)) case _ => new BlackHolePersistenceEngine() }
//用作master的leader选举 leaderElectionAgent
= RECOVERY_MODE match { case "ZOOKEEPER" => context.actorOf(Props(classOf[ZooKeeperLeaderElectionAgent], self, masterUrl, conf)) case _ => context.actorOf(Props(classOf[MonarchyLeaderAgent], self)) } }

override def receive = {
  case RegisterWorker

case RequestSubmitDriver

case RegisterApplication

case Heartbeat

case ExecutorStateChanged

......
}

 

Master的主要功能总结:

1、Master的Leader选举;

2、对Driver、Worker、Application的管理

  1)Driver:RequestSubmitDriver/RequestKillDriver/RequestDriverStatus/DriverStateChanged

  2)Worker:RegisterWorker/Heartbeat/WorkerSchedulerStateResponse/CheckForWorkerTimeOut/ExecutorStateChanged

  3)Application:RegisterApplication

 

 

 

 

 

 

 

 

Spark分析之Master,布布扣,bubuko.com

Spark分析之Master

标签:style   blog   http   color   strong   art   

原文地址:http://www.cnblogs.com/luogankun/p/3816909.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!