MapReduce分布式缓存程序，无法在Windows下的Eclipse中执行问题解决

标签：

在写mapreduce程序中经常要用到hadoop自动的分布式缓存DistributedCache（新版本已经换新的API），但是在windows下Eclipse中执行，会出现类似如下错误：

2016-03-03 10:53:21,424 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-03-03 10:53:22,152 INFO [main] Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1019)) - session.id is deprecated. Instead, use dfs.metrics.session-id
2016-03-03 10:53:22,152 INFO [main] jvm.JvmMetrics (JvmMetrics.java:init(76)) - Initializing JVM Metrics with processName=JobTracker, sessionId=
2016-03-03 10:53:24,366 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-03-03 10:53:26,447 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-03-03 10:53:26,487 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 4
2016-03-03 10:53:30,876 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:4
2016-03-03 10:53:31,065 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_local1862629830_0001
2016-03-03 10:53:31,133 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/staging/root1862629830/.staging/job_local1862629830_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-03-03 10:53:31,142 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/staging/root1862629830/.staging/job_local1862629830_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-03-03 10:53:31,953 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(207)) - Creating symlink: \tmp\hadoop-root\mapred\local\1456973611218\part-r-00003 <- F:\JavaWork2014\mapreduce/part-r-00003
2016-03-03 10:53:32,004 WARN [main] fs.FileUtil (FileUtil.java:symLink(824)) - Fail to create symbolic links on Windows. The default security settings in Windows disallow non-elevated administrators and all non-administrators from creating symbolic links. This behavior can be changed in the Local Security Policy management console
2016-03-03 10:53:32,004 WARN [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(209)) - Failed to create symlink: \tmp\hadoop-root\mapred\local\1456973611218\part-r-00003 <- F:\JavaWork2014\mapreduce/part-r-00003
2016-03-03 10:53:32,005 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(171)) - Localized hdfs://node1:8020/usr/output/weibo1/part-r-00003 as file:/tmp/hadoop-root/mapred/local/1456973611218/part-r-00003
2016-03-03 10:53:32,012 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(207)) - Creating symlink: \tmp\hadoop-root\mapred\local\1456973611219\part-r-00000 <- F:\JavaWork2014\mapreduce/part-r-00000
2016-03-03 10:53:32,050 WARN [main] fs.FileUtil (FileUtil.java:symLink(824)) - Fail to create symbolic links on Windows. The default security settings in Windows disallow non-elevated administrators and all non-administrators from creating symbolic links. This behavior can be changed in the Local Security Policy management console
2016-03-03 10:53:32,052 WARN [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(209)) - Failed to create symlink: \tmp\hadoop-root\mapred\local\1456973611219\part-r-00000 <- F:\JavaWork2014\mapreduce/part-r-00000
2016-03-03 10:53:32,052 INFO [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:setup(171)) - Localized hdfs://node1:8020/usr/output/weibo2/part-r-00000 as file:/tmp/hadoop-root/mapred/local/1456973611219/part-r-00000
2016-03-03 10:53:32,172 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1862629830_0001/job_local1862629830_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2016-03-03 10:53:32,177 WARN [main] conf.Configuration (Configuration.java:loadProperty(2368)) - file:/tmp/hadoop-root/mapred/local/localRunner/root/job_local1862629830_0001/job_local1862629830_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2016-03-03 10:53:32,182 INFO [main] mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://localhost:8080/
2016-03-03 10:53:32,183 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_local1862629830_0001
2016-03-03 10:53:32,184 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(471)) - OutputCommitter set in config null
2016-03-03 10:53:32,190 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:createOutputCommitter(489)) - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2016-03-03 10:53:32,236 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(448)) - Waiting for map tasks
2016-03-03 10:53:32,237 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:run(224)) - Starting task: attempt_local1862629830_0001_m_000000_0
2016-03-03 10:53:32,262 INFO [LocalJobRunner Map Task Executor #0] util.ProcfsBasedProcessTree (ProcfsBasedProcessTree.java:isAvailable(181)) - ProcfsBasedProcessTree currently is supported only on Linux.
2016-03-03 10:53:32,306 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:initialize(587)) - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@da44a9a
2016-03-03 10:53:32,310 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:runNewMapper(733)) - Processing split: hdfs://node1:8020/usr/output/weibo1/part-r-00001:0+195718
2016-03-03 10:53:32,319 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:createSortingCollector(388)) - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:setEquator(1182)) - (EQUATOR) 0 kvi 26214396(104857584)
2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(975)) - mapreduce.task.io.sort.mb: 100
2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(976)) - soft limit at 83886080
2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(977)) - bufstart = 0; bufvoid = 104857600
2016-03-03 10:53:32,344 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:init(978)) - kvstart = 26214396; length = 6553600

................................
2016-03-03 10:53:32,614 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1437)) - Starting flush of map output
2016-03-03 10:53:32,626 INFO [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete.
2016-03-03 10:53:32,734 WARN [Thread-12] mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local1862629830_0001
java.lang.Exception: java.io.FileNotFoundException: part-r-00003 (系统找不到指定的文件。)
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.FileNotFoundException: part-r-00003 (系统找不到指定的文件。)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at java.io.FileReader.<init>(FileReader.java:58)
at com.laoxiao.mr.tf.LastMapper.setup(LastMapper.java:49)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-03-03 10:53:33,186 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Job job_local1862629830_0001 running in uber mode : false
2016-03-03 10:53:33,188 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1362)) - map 0% reduce 0%
2016-03-03 10:53:33,191 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1375)) - Job job_local1862629830_0001 failed with state FAILED due to: NA
2016-03-03 10:53:33,197 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 0

经过对错误的仔细分析后，发现问题出现在本地机器上无法把缓存中的数据拷贝的到本地磁盘，具体错误参考上面标红的文字

解决方法

1.直接把程序打成jar包，然后在Linux服务器下利用hadoop jar命令运行

2.或者关闭windows7 UAC，然后就可以在Eclipse下本地测试模式跑mapreduce

另外如果用上面的两种方法还出现类似如下错误：

java.lang.Exception: java.io.FileNotFoundException: file:\tmp\hadoop-root\mapred\local\1456979002661\tb_dim_city.dat (文件名、目录名或卷标语法不正确。)
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.FileNotFoundException: file:\tmp\hadoop-root\mapred\local\1456979002661\tb_dim_city.dat (文件名、目录名或卷标语法不正确。)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at java.io.FileInputStream.<init>(FileInputStream.java:101)
at java.io.FileReader.<init>(FileReader.java:58)
at MapJoin.MapSideJoin_LeftOuterJoin$LeftOutJoinMapper.setup(MapSideJoin_LeftOuterJoin.java:100)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

那么可能的原因是hdfs上传的文件名后缀引起的，比如.dat之类的，可以把文件后缀删除后试试

MapReduce分布式缓存程序，无法在Windows下的Eclipse中执行问题解决

标签：

原文地址：http://www.cnblogs.com/QTHelper/p/5237949.html