码迷,mamicode.com
首页 > 系统相关 > 详细

启动spark-shell --master yarn的bug

时间:2018-06-06 21:40:39      阅读:423      评论:0      收藏:0      [点我收藏+]

标签:head   --   manager   enabled   ilo   enable   client   end   net   

报错如下

18/06/06 15:55:31 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
18/06/06 15:55:31 ERROR client.TransportClient: Failed to send RPC 8494797597505892687 to /192.168.1.21:44578: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
18/06/06 15:55:31 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC 8494797597505892687 to /192.168.1.21:44578: java.nio.channels.ClosedChannelException
        at org.apache.spark.network.client.TransportClient.lambda$sendRpc$2(TransportClient.java:237)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:852)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:738)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:733)
        at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:725)

需要在yarn-site.xml中添加如下配置

    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>

    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>

分发yarn-site.xml并重启yarn,再试一次spark-shell --master yarn就可以了.

FIN

启动spark-shell --master yarn的bug

标签:head   --   manager   enabled   ilo   enable   client   end   net   

原文地址:https://www.cnblogs.com/krcys/p/9146322.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!