标签：cer mapr 没有 targe inpu pac tput cal let
1.问题描述：在调试mapreduce辅助排序（二次排序）的过程中，运行程序总是报错
Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2021)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8860 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
8872 [Thread-6] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1887386767_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2.原因分析：
（1）初步判断是是调用了空指针；
（2）当点击WritableComparator.java:157 会跳转到报错的地方是this.buffer.reset(b1, s1, l1);
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
this.buffer.reset(b1, s1, l1);
this.key1.readFields(this.buffer);
this.buffer.reset(b2, s2, l2);
this.key2.readFields(this.buffer);
this.buffer.reset((byte[])null, 0, 0);
} catch (IOException var8) {
throw new RuntimeException(var8);
}
（3）也就是说WritableComparator buffer没有初始化
public class WritableComparator implements RawComparator, Configurable {
private static final ConcurrentHashMap<Class, WritableComparator> comparators = new ConcurrentHashMap();
private Configuration conf;
private final Class<? extends WritableComparable> keyClass;
private final WritableComparable key1;
private final WritableComparable key2;
private final DataInputBuffer buffer;
（4）查找程序中使用到WritableComparator 相关的类的定义是分组类ReducerGroupingComparator 和第一字段的比较类FirstSecondComparator 没有默认的构造函数
public static class ReducerGroupingComparator extends WritableComparator
{//一定要加默认构造函数，且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c56
public ReducerGroupingComparator()
{
super(IntPair.class,true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
IntPair pairA=(IntPair)a;
IntPair pairB=(IntPair)b;
//年份相同，返回值为0的分为一组
return (int) (pairA.getFirst()-pairB.getFirst());
}
}
//与FirstSecondComparator中定义的默认对比函数功能相同，这个用于显示设置对比类
public static class FirstSecondComparator extends WritableComparator
{
//
//一定要加默认构造函数，且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewOutputCollector@1398c
public FirstSecondComparator()
{
super(IntPair.class,true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
IntPair pairA=(IntPair)a;
IntPair pairB=(IntPair)b;
int cmp= (int) (pairA.getFirst()-pairA.getFirst());
if (cmp!=0)
{
return cmp;
}
double A=pairA.getSecond();
double B=pairB.getSecond();
return -((A==B)? 0:(A>B? 1:-1));
}
}
（5）为什么一定要加这个才不报java.lang.NullPointerException异常？点开super(IntPair.class,true);
会跳转到WritableComparator的构造函数
protected WritableComparator(Class<? extends WritableComparable> keyClass, boolean createInstances) {
this(keyClass, (Configuration)null, createInstances);
}
在点击this，进一步跳转到另外一个重载构造函数，看到这里大家应该名明白了，上面报异常的地方，就是这些变量没有申请内存空间
protected WritableComparator(Class<? extends WritableComparable> keyClass, Configuration conf, boolean createInstances) {
this.keyClass = keyClass;
this.conf = conf != null ? conf : new Configuration();
if (createInstances) {
this.key1 = this.newKey();
this.key2 = this.newKey();
this.buffer = new DataInputBuffer();
} else {
this.key1 = this.key2 = null;
this.buffer = null;
}
}
自己开发了一个股票智能分析软件，功能很强大，需要的点击下面的链接获取：
https://www.cnblogs.com/bclshuai/p/11380657.html
标签：cer mapr 没有 targe inpu pac tput cal let
原文地址：https://www.cnblogs.com/bclshuai/p/12329390.html
