码迷,mamicode.com
首页 > 其他好文 > 详细

spark-submit提交任务到yarn错误

时间:2020-06-29 15:02:48      阅读:96      评论:0      收藏:0      [点我收藏+]

标签:ado   park   exception   page   sub   local   错误   mit   director   

1.Error initializing SparkContext.

20/06/29 05:52:43 INFO yarn.Client: Deleted staging directory hdfs://master:9000/user/hadoop/.sparkStaging/application_1593402611188_0003
20/06/29 05:52:43 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

遇到这问题先不要纠结这个点,先看其他错误。我的另一个错误是

For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1593402611188_0003Then, click on links to logs of each attempt.
Diagnostics: Wrong FS: file://home/hadoop/data/hadoop/tmp/nm-local-dir, expected: file:///
Failing this attempt. Failing the application.

也就是找不到yarn的nm-local-dir所以应该在yarn-site.xmlite.xml里配置yarn.nodemanager.local-dirs

<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/data/hadoop/tmp/nm-local-dir</value>
</property>

 

spark-submit提交任务到yarn错误

标签:ado   park   exception   page   sub   local   错误   mit   director   

原文地址:https://www.cnblogs.com/mergy/p/13207977.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!