标签：srp gen get services 完成 问题排查 今天 rmi llb
今天线上遇到一个问题：有一台服务器的cpu持续冲高，排查发现是我们的一个java应用进程造成的，该进程在向hbase中写入数据时，日志不断地打印下面的异常：
org.apache.hadoop.hbase.NotServingRegionException: Region iot_flow_cdr_201811,4379692584601-2101152593-20181115072326-355,1536703383699.82804f639798d0502dd64e6e47d75d84. is not online on shqz-ps-iot3-cdr-dn01,60020,1524812940505
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2921)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1053)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2096)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)排查思路如下：
(1) 检查该表是否存在一致性问题
hbase hbck -details iot_flow_cdr_201811
确实发现了不一致的异常
8 inconsistencies detected
(2) 尝试修复该问题
hbase hbck -repair iot_flow_cdr_201811
执行该命令出现下述错误
18/11/15 11:28:15 WARN util.HBaseFsck: Got AccessDeniedException when preCheckPermission
org.apache.hadoop.hbase.security.AccessDeniedException: Permission denied: action=WRITE path=hdfs://nameservice1/hbase/.hbase-snapshot user=root
at org.apache.hadoop.hbase.util.FSUtils.checkAccess(FSUtils.java:1797)
at org.apache.hadoop.hbase.util.HBaseFsck.preCheckPermission(HBaseFsck.java:1932)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4734)
at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:4562)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:4550)
Current user root does not have write perms to hdfs://nameservice1/hbase/.hbase-snapshot. Please rerun hbck as hdfs user hbase根据提示可以看到，错误原因是没有权限Permission denied
然后我们以hbase用户身份执行该命令
sudo - hbase hbase hbck -repair iot_flow_cdr_201811
这次执行成功了，等命令执行完成后，修复了inconsistencies（数据不一致）的错误。
最后重启应用，观察日志，程序正常执行，NotServingRegionException异常不再出现了，服务器cpu也恢复了正常。
线上问题排查-HBase写数据出现NotServingRegionException(Region ... is not online)异常
标签：srp gen get services 完成 问题排查 今天 rmi llb
原文地址：https://www.cnblogs.com/wxyidea/p/9963205.html
