标签：hadoop hadoop异常 reduce拉取数据失败 error in shuffle in
主要错误信息：Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#43
解决办法：限制reduce的shuffle内存使用
hive：set mapreduce.reduce.shuffle.memory.limit.percent=0.1;
MR：job.getConfiguration().setStrings("mapreduce.reduce.shuffle.memory.limit.percent", "0.1");
原理分析：reduce会在map执行到一定比例启动多个fetch线程去拉取map的输出结果，放到reduce的内存、磁盘中，然后进行merge。当数据量大时，拉取到内存的数据就会引起OOM，所以此时要减少fetch占内存的百分比，将fetch的数据直接放在磁盘上。
mapreduce.reduce.shuffle.memory.limit.percent：每个fetch取到的map输出的大小能够占的内存比的大小。默认是0.25。因此实际每个fetcher的输出能放在内存的大小是reducer的java heap size*0.9*0.25。
完整的错误日志：
INFO mapreduce.Job: Task Id : attempt_xxxxxx, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#43
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1550)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:123)
at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:98)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:192)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:434)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:341)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
hadoop异常之 reduce拉取数据失败 （error in shuffle in fetcher）
标签：hadoop hadoop异常 reduce拉取数据失败 error in shuffle in
原文地址：http://blog.csdn.net/smile0198/article/details/44133379
