因磁盘爆满而导致NameNode HA无法启动

时间：2017-08-15 10:23:29 阅读：312 评论：0 收藏：0 [点我收藏+]

标签：data mod write server 开启 names eterm read oop

场景回顾：

测试集群节点分配：35，36是namenode且开启HA，37,38,39即作为datanode，又作为journalnode。

某时间 38节点磁盘爆满，集群中hdfs及依赖的服务全部宕掉。

之后虽然将38节点清除一部分多余数据，但集群服务仍然不正常：namenode启动的时候两个都是standby，几分钟后自动关闭；datanode正常；journal正常；zkfc正常

调查过程：

查看namenode日志，显示一直在尝试连接journal，并显示

2017-08-14 17:03:58,669 WARN  client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 88071 ms (timeout=90000 ms) for a response for selectInputStreams. Succeeded so far: [192.168.0.37:8485]
2017-08-14 17:03:59,670 WARN  client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 89072 ms (timeout=90000 ms) for a response for selectInputStreams. Succeeded so far: [192.168.0.37:8485]
2017-08-14 17:04:00,600 WARN  namenode.FSEditLog (JournalSet.java:selectInputStreams(280)) - Unable to determine input streams from QJM to [192.168.0.37:8485, 192.168.0.38:8485, 192.168.0.39:8485]. Skipping.
java.io.IOException: Timed out waiting 90000ms for a quorum of nodes to respond.
	at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
	at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:278)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1528)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1552)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1015)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:690)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:976)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1686)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1754)

根据这些错误信息，尝试了很多网上的方案都没有起作用。

在启动namenode的时候观察hadoop UI中的Startup Progress

技术分享