标签:compute require edr launcher mat art red star enum
oozie提交workflow后执行task报错:
2019-07-04 17:19:00,559 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM. java.lang.ArithmeticException: / by zero at org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:981) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:873) at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:252) at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282) at java.lang.Thread.run(Thread.java:748)
查看代码
org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils
public static int computeAvailableContainers(Resource available, Resource required, EnumSet<SchedulerResourceTypes> resourceTypes) { return resourceTypes.contains(SchedulerResourceTypes.CPU)?Math.min(available.getMemory() / required.getMemory(), available.getVirtualCores() / required.getVirtualCores()):available.getMemory() / required.getMemory(); }
应该是以下两者之一为0
required.getMemory()
required.getVirtualCores()
注意之前的日志还有一行
2019-07-04 17:18:58,557 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:0, vCores:1>
问题在于required.getMemory()=0
从job history server查看失败task对应的application的conf发现问题
<tr> <td> mapreduce.map.memory.mb </td> <td> 0 </td> <td> job.xml ⬅ programatically </td> </tr>
查看oozie代码发现
org.apache.oozie.action.hadoop.JavaActionExecutor
private static void injectLauncherProperties(Configuration srcConf, Configuration launcherConf) { for (Map.Entry<String, String> entry : srcConf) { if (entry.getKey().startsWith("oozie.launcher.")) { String name = entry.getKey().substring("oozie.launcher.".length()); String value = entry.getValue(); // setting original KEY launcherConf.set(entry.getKey(), value); // setting un-prefixed key (to allow Hadoop job config // for the launcher job launcherConf.set(name, value); } } }
所有的hadoop配置,都需要增加 oozie.launcher. 前缀(大量的组件都是这么搞得),即需要传递参数
oozie.launcher.mapreduce.map.memory.mb=1024
问题解决
【原创】大叔问题定位分享(33)oozie提交任务报错ArithmeticException: / by zero
标签:compute require edr launcher mat art red star enum
原文地址:https://www.cnblogs.com/barneywill/p/11140996.html