码迷,mamicode.com
首页 > 其他好文 > 详细

【原创】大叔问题定位分享(33)oozie提交任务报错ArithmeticException: / by zero

时间:2019-07-05 23:59:42      阅读:260      评论:0      收藏:0      [点我收藏+]

标签:compute   require   edr   launcher   mat   art   red   star   enum   

oozie提交workflow后执行task报错:

2019-07-04 17:19:00,559 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM. 
java.lang.ArithmeticException: / by zero
    at org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:981)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:873)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:252)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282)
    at java.lang.Thread.run(Thread.java:748)

查看代码

org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils

    public static int computeAvailableContainers(Resource available, Resource required, EnumSet<SchedulerResourceTypes> resourceTypes) {
        return resourceTypes.contains(SchedulerResourceTypes.CPU)?Math.min(available.getMemory() / required.getMemory(), available.getVirtualCores() / required.getVirtualCores()):available.getMemory() / required.getMemory();
    }

应该是以下两者之一为0

required.getMemory()
required.getVirtualCores()

注意之前的日志还有一行

2019-07-04 17:18:58,557 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:0, vCores:1>

问题在于required.getMemory()=0

 

从job history server查看失败task对应的application的conf发现问题

              <tr>
                <td>
                  mapreduce.map.memory.mb
                </td>
                <td>
                  0
                </td>
                <td>
                  job.xml &#11013; programatically
                </td>
              </tr>

查看oozie代码发现

org.apache.oozie.action.hadoop.JavaActionExecutor

    private static void injectLauncherProperties(Configuration srcConf, Configuration launcherConf) {
        for (Map.Entry<String, String> entry : srcConf) {
            if (entry.getKey().startsWith("oozie.launcher.")) {
                String name = entry.getKey().substring("oozie.launcher.".length());
                String value = entry.getValue();
                // setting original KEY
                launcherConf.set(entry.getKey(), value);
                // setting un-prefixed key (to allow Hadoop job config
                // for the launcher job
                launcherConf.set(name, value);
            }
        }
    }

所有的hadoop配置,都需要增加 oozie.launcher. 前缀(大量的组件都是这么搞得),即需要传递参数

oozie.launcher.mapreduce.map.memory.mb=1024

问题解决

 

【原创】大叔问题定位分享(33)oozie提交任务报错ArithmeticException: / by zero

标签:compute   require   edr   launcher   mat   art   red   star   enum   

原文地址:https://www.cnblogs.com/barneywill/p/11140996.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!