Hadoop集群日常运维

标签：hadoop   运维   

#!/bin/bash
export dirname=/mnt/tmphadoop/dfs/namesecondary/current/`date +%y%m%d%H`
if [ ! -d ${dirname} ]
then
mkdir  ${dirname}
cp /mnt/tmphadoop/dfs/namesecondary/current/*  ${dirname}
fi
scp -r ${dirname} slave1:/mnt/namenode_backup/
rm -r ${dirname}

[jediael@master ~]$ hadoop fsck /
……省略输出（若有错误，则在此外出现，否则只会出现点，一个点表示一个文件）……
.........Status: HEALTHY
 Total size:    14466494870 B
 Total dirs:    502
 Total files:   1592 (Files currently being written: 2)
 Total blocks (validated):      1725 (avg. block size 8386373 B)
 Minimally replicated blocks:   1725 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       648 (37.565216 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              760 (22.028986 %)
 Number of data-nodes:          2
 Number of racks:               1
FSCK ended at Sun Mar 01 20:17:57 CST 2015 in 608 milliseconds

The filesystem under path '/' is HEALTHY

[jediael@master conf]$ hadoop fsck /hbase/Feb2621_webpage/c23aa183c7cb86af27f15d4c2aee2795/s/30bee5fb620b4cd184412c69f70d24a7 -files -blocks -racks

FSCK started by jediael from /10.171.29.191 for path /hbase/Feb2621_webpage/c23aa183c7cb86af27f15d4c2aee2795/s/30bee5fb620b4cd184412c69f70d24a7 at Sun Mar 01 20:39:35 CST 2015
/hbase/Feb2621_webpage/c23aa183c7cb86af27f15d4c2aee2795/s/30bee5fb620b4cd184412c69f70d24a7 21507169 bytes, 1 block(s):  Under replicated blk_7117944555454804881_3655. Target Replicas is 3 but found 2 replica(s).
0. blk_7117944555454804881_3655 len=21507169 repl=2 [/default-rack/10.171.94.155:50010, /default-rack/10.251.0.197:50010]

Status: HEALTHY
 Total size:    21507169 B
 Total dirs:    0
 Total files:   1
 Total blocks (validated):      1 (avg. block size 21507169 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1 (100.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              1 (50.0 %)
 Number of data-nodes:          2
 Number of racks:               1
FSCK ended at Sun Mar 01 20:39:35 CST 2015 in 0 milliseconds


The filesystem under path '/hbase/Feb2621_webpage/c23aa183c7cb86af27f15d4c2aee2795/s/30bee5fb620b4cd184412c69f70d24a7' is HEALTHY

[jediael@master ~]$ hadoop fsck -files
Usage: DFSck <path> [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]
        <path>  start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -blocks print out block report
        -locations      print out locations for every block
        -racks  print out network topology for data-node locations
                By default fsck ignores files opened for write, use -openforwrite to report such files. They are usually  tagged CORRUPT or HEALTHY depending on their block allocation status
Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|jobtracker:port>    specify a job tracker
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

均衡器是一个hadoop守护进程，它将块从忙碌的DN移动相对空闲的DN，同时坚持块复本放置策略，将复本分散到不同的机器、机架。

建议定期执行均衡器，如每天或者每周。

[jediael@master log]$ start-balancer.sh
starting balancer, logging to /var/log/hadoop/hadoop-jediael-balancer-master.out

[jediael@master hadoop]$ pwd
/var/log/hadoop
[jediael@master hadoop]$ ls
hadoop-jediael-balancer-master.log  hadoop-jediael-balancer-master.out
[jediael@master hadoop]$ cat hadoop-jediael-balancer-master.log
2015-03-01 21:08:08,027 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/10.251.0.197:50010
2015-03-01 21:08:08,028 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/10.171.94.155:50010
2015-03-01 21:08:08,028 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 0 over utilized nodes:
2015-03-01 21:08:08,028 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: 0 under utilized nodes:

Hadoop集群日常运维

标签：hadoop   运维   

原文地址：http://blog.csdn.net/jediael_lu/article/details/44006629