缩写

全写

说明

JVM

Java Virtual Matchine

Java虚拟机

jps

JVM Process Status

Java虚拟机进程状态工具

HDFS

Hadoop Distributed File System

Hadoop分布式文件系统

HBase

Hadoop Database

Hadoop数据库

WAL

Write-Ahead Log

预写日志，类似于MySQL的binlog

RS

Region Server

zk

Zookeeper

mr

Mapreduce

rs

RegionServer

DistCp

Distributed Copy

分布式复制

RFA

RollingFileAppender

Log4j的一种日志器类型，文件大小到达指定大小时产生一个新的文件

DRFA

DailyRollingFileAppender

Log4j的一种日志器类型，每天产生一个日志文件

安装用户名

运行用户名

安装和运行用户组名

HBase

hbase

hbase

supergroup

Hadoop

hadoop

hadoop

Zookeeper

zk

zk

安装目录

HBase

/data/hbase

Hadoop

/data/hadoop

Zookeeper

/data/zookeeper

2888

ZooKeeper，如果是Leader，用来监听Follower的连接

3888

ZooKeeper，用于Leader选举

2181

ZooKeeper，用来监听客户端的连接

16010

hbase.master.info.port，HMaster的http端口

16000

hbase.master.port，HMaster的RPC端口

16030

hbase.regionserver.info.port，HRegionServer的http端口

16020

hbase.regionserver.port，HRegionServer的RPC端口

8080

hbase.rest.port，HBase REST server的端口

9095

hbase.thrift.info.port，HBase Thrift Server的http端口号

hadoop@test_64:~/hbase/conf> cat regionservers

192.168.31.30

192.168.31.31

192.168.31.32

# The java implementation to use.  Java 1.6 required.

export JAVA_HOME=/data/jdk

export HADOOP_HOME=/data/hadoop

# Extra Java CLASSPATH elements.  Optional.

export HBASE_CLASSPATH=$HADOOP_HOME/etc/hadoop

# Tell HBase whether it should manage it‘s own instance of Zookeeper or not.

export HBASE_MANAGES_ZK=false

JVM项

值

说明

HBASE_THRIFT_OPTS

export HBASE_THRIFT_OPTS="$HBASE_THRIFT_OPTS -Xmx2048m -Xms2048m"

HBase Thrift JVM设置，-Xmx指定最大堆占用的内存，-Xms指定起始时分析的堆内存，如果物理内存为32G，可以考虑设置为2048m。

SERVER_GC_OPTS

开始GC日志

可用来分析线程的停顿时长等，有三处SERVER_GC_OPTS，只需要取消任意一注释即可。

# HBASE_ROOT_LOGGER=INFO,DRFA

HBASE_ROOT_LOGGER=OFF,DRFA

hbase.log.dir=/data/hbase/log

log4j.logger.org.apache.hadoop.hbase.master.cleaner.CleanerChore=DEBUG

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!--

/**

 *

 * Licensed to the Apache Software Foundation (ASF) under one

 * or more contributor license agreements.  See the NOTICE file

 * distributed with this work for additional information

 * regarding copyright ownership.  The ASF licenses this file

 * to you under the Apache License, Version 2.0 (the

 * "License"); you may not use this file except in compliance

 * with the License.  You may obtain a copy of the License at

 *

 *     http://www.apache.org/licenses/LICENSE-2.0

 *

 * Unless required by applicable law or agreed to in writing, software

 * distributed under the License is distributed on an "AS IS" BASIS,

 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

 * See the License for the specific language governing permissions and

 * limitations under the License.

 */

-->

<configuration>

</configuration>

<configuration>

  <property>

    <name>hbase.rootdir</name>

    <value>hdfs://hadoop10102:8020/hbase</value>

    <description>The directory shared by RegionServers.</description>

  </property>

  <property>

    <name>hbase.cluster.distributed</name>

    <value>true</value>

    <description>The mode the cluster will be in. Possible values are

      false: standalone and pseudo-distributed setups with managed Zookeeper

      true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)

    </description>

  </property>

  <property>

    <name>hbase.zookeeper.quorum</name>

    <value>hadoop10108,hadoop10109,hadoop10110,hadoop10111,hadoop10112</value>

    <description>Comma separated list of servers in the ZooKeeper Quorum.

      For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".

      By default this is set to localhost for local and pseudo-distributed modes

      of operation. For a fully-distributed setup, this should be set to a full

      list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh

      this is the list of servers which we will start/stop ZooKeeper on.

    </description>

  </property>

  <property>

    <name>hbase.master.maxclockskew</name>

    <value>600000</value>

    <description>Time(ms) difference of regionserver from master</description>

  </property>

</configuration>

 <property>

        <name>hbase.rootdir</name>

        <value>hdfs://test/hbase</value>

 </property>

major_compact ‘tablename‘

echo "tablename"|hbase shell

Examples:

Compact all regions in a table: // 合并整个表

hbase> major_compact ‘t1‘

hbase> major_compact ‘ns1:t1‘

Compact an entire region: // 合并一个Region

hbase> major_compact ‘r1‘

Compact a single column family within a region: // 合并一个Region的一个列族

hbase> major_compact ‘r1‘, ‘c1‘

Compact a single column family within a table: // 合并一个表的一个列族

hbase> major_compact ‘t1‘, ‘c1‘

参数名

默认值

说明

hbase.regionserver.thrift.port

9090

ThriftServer的服务端口号

hbase.regionserver.thrift.http

false

是否为HTTP模式。注意要么是HTTP模式，要么是非HTTP模式，不能同时是HTTP模式，又是其它模式。

hbase.thrift.ssl.enabled

false

ThriftServer是否启用SSL

hbase.master.hfilecleaner.ttl

300000

单位为毫秒，默认5分钟，为org.apache.hadoop.hbase.master.cleaner.CleanerChore所使用，仅源代码文件StorefileRefresherChore.java和TimeToLiveHFileCleaner.java直接使用了hbase.master.hfilecleaner.ttl。如果使用ExportSnapshot集群间迁移数据，则该值要比最大表的迁移时间长，否则迁移过程中已迁移的表文件可能被删除，如果看到错误“Can‘t find hfile”，一般是这个原因，新版本将修复该BUG，此BUG由HBASE-21511引入。

hbase.cleaner.scan.dir.concurrent.size

Cleaner线程数系数

hbase.hregion.majorcompaction

604800000

单位为毫秒，默认为7天，Major Compact的周期，值为0时表示关闭自动Major Compact

hbase.hregion.majorcompaction.jetter

0.2

防止RegionServer在同一时间进行Major Compact

hbase.hstore.compactionThreshold

3

Minor Compact的最少文件数

hbase.hstore.compaction.max

10

表示一次Minor Compact中最多选取10个StoreFile

hbase.hregion.max.filesize

10737418240

默认为10G，StoreFile达到此大小时分裂

hbase.hregion.memstore.flush.size

134217728

单位为字节，默认为整128M，当一个MemStore达到该大小时，即Flush到HDFS生成HFile文件。

hbase.hregion.memstore.block.multiplier

4

hbase-common

/org/apache/hadoop/hbase/HConstants.java

系数值，当一个Region的大小达到（hbase.hregion.memstore.block.multiplier*hbase.hregion.memstore.flush.size）时，会阻塞该Region的写操作，并强制Flush。在Flush成功后才会恢复写操作，这个时候可看到错误“RegionTooBusyException: Over memstore limit”。

hbase.regionserver.global.memstore.upperLimit（老）

hbase.regionserver.global.memstore.size（新）

0.4

hbase-server

/org/apache/hadoop/hbase/io/util/MemorySizeUtil.java

浮点值，RegionServer全局memstore的大小。

当RegionServer的memstore超过这个比率，该RegionServer的所有update都会被阻塞，所以应当避免RegionServer出现memstore总大小超过upperLimit。

hbase.regionserver.global.memstore.lowerLimit（老）

hbase.regionserver.global.memstore.size.lower.limit（新）

0.95

hbase-server

/org/apache/hadoop/hbase/io/util/MemorySizeUtil.java

浮点值，当RegionServer的memstore超过这个比率时，即使没有Region达到Flush，MemstoreFlusher也会挑选一个Region去Flush。

hbase.hstore.compaction.kv.max

10

Compact时每次从HFile中读取的KV数

hbase.hstore.blockingStoreFiles

10

当StoreFile文件数超过该值时，则在Flush前先进行Split或Compact，并阻塞Flush操作。直到完成或达到hbase.hstore.blockingWaitTime指定的时间后才可以Flush。当RegionServer日志大量出现“has too many store files; delaying flush up to”时表示需要调整该值，一般可以调大些，比如调成100。

hbase.hstore.blockingWaitTime

90000

单位为毫秒，默认为90秒

hbase.hregion.memstore.mslab.enabled

true

MemStore-Local Allocation Buffer，是否开启mslab，作用是减少因内存碎片导致的Full GC

hbase.regionserver.regionSplitLimit

1000

单台RegionServer管理的最多Region个数

hbase.regionserver.logroll.period

3600000

单位为毫秒，默认为1小时，WAL文件滚动时间间隔

hbase.server.thread.wakefrequency

10000

定时检查是否需要Compact，检查周期为：(hbase.server.thread.wakefrequency*hbase.server.compactchecker.interval.multiplier)

hbase.server.compactchecker.interval.multiplier

1000

定时检查是否需要Compact，检查周期为：(hbase.server.thread.wakefrequency*hbase.server.compactchecker.interval.multiplier)

hbase.thrift.info.port

9095

信息WEB端口，如果值小于0，则不会开启WEB

hbase.thrift.info.bindAddress

0.0.0.0

信息WEB地址，默认为“0.0.0.0”

hbase.regionserver.hlog.blocksize

WALUtil.java

当WAL的大小达到

（hbase.regionserver.hlog.blocksize

*hbase.regionserver.maxlogs）时，触发Flush

hbase.regionserver.maxlogs

AbstractFSWAL.java

hbase.regionserver.logroll.multiplier

AbstractFSWAL.java

系数值，注意区分

hbase.hregion.memstore.block.multiplier

hbase-daemon.sh start master

hbase-daemon.sh start regionserver

hbase-daemon.sh start thrift2 --framed -nonblocking

hbase-daemon.sh start thrift2 --framed -hsha --workers 30或

hbase-daemon.sh start thrift2 --framed -hsha -w 30或

hbase thrift2 --framed -hsha -w 30

hbase-daemon.sh start thrift2 --framed -threadedselector -s 10 -m 10 -w 50或

hbase thrift2 --framed -threadedselector -s 10 -m 10 -w 50

ERROR [thrift-worker-18] client.AsyncRequestFutureImpl: Cannot get replica 0 location for

WARN  [Thread-16] server.TThreadedSelectorServer: ExecutorService rejected execution!

java.util.concurrent.RejectedExecutionException: Task org.apache.thrift.server.Invocation@1642f20a rejected from org.apache.hadoop.hbase.thrift.THBaseThreadPoolExecutor@7cc22596[Running, pool size = 20, active threads = 20, queued tasks = 1000, completed tasks = 32792177]

INFO  [thrift-worker-2] client.RpcRetryingCallerImpl: RegionTooBusyException: Over memstore limit=512.0M, regionName=f146450279edde75342d003affa36be6, server=hadoop003,16020,1572244447374

at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4421)

at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3096)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2877)

INFO  [thrift-worker-24] client.RpcRetryingCallerImpl: Exception: Over memstore limit=512.0M, regionName=f80b7247cce17f3faac8731212ce43ba, server=hadoop003,16020,1572244447374

at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4421)

at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3096)

at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2877)

。。。

, details=row ‘13871489517670268‘ on table ‘test‘ at region=test,13628864449,1572513486889.f80b7247cce17f3faac8731212ce43ba., hostname=hadoop003,16020,1572244447374, seqNum=10634659, see https://s.apache.org/timeout

bin/hbase-daemon.sh start rest -p 8080

Endpoint

HTTP Verb

说明

示例

/version/cluster

GET

查看HBase版本

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/version/cluster"

/status/cluster

GET

查看集群状态

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/status/cluster"

/

GET

列出所有的非系统表

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/"

Endpoint

HTTP Verb

说明

示例

/namespaces

GET

列出所有namespaces

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/namespaces/"

/namespaces/namespace

GET

查看指定namespace的说明

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/namespaces/special_ns"

/namespaces/namespace

POST

创建一个新的namespace

curl -vi -X POST \

-H "Accept: text/xml" \

"example.com:8000/namespaces/special_ns"

/namespaces/namespace/tables

GET

列出指定namespace下的所有表

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/namespaces/special_ns/tables"

/namespaces/namespace

PUT

修改一个已存在的namespace

curl -vi -X PUT \

-H "Accept: text/xml" \

"http://example.com:8000/namespaces/special_ns

/namespaces/namespace

DELETE

删除一个namespace，前提是该namespace已为空

curl -vi -X DELETE \

-H "Accept: text/xml" \

"example.com:8000/namespaces/special_ns"

Endpoint

HTTP Verb

说明

示例

/table/schema

GET

查看指定表的schema

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/schema"

/table/schema

POST

使用schema创建一个新的表或修改已存在表的schema

curl -vi -X POST \

-H "Accept: text/xml" \

-H "Content-Type: text/xml" \

-d ‘<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" /></TableSchema>‘ \

"http://example.com:8000/users/schema"

/table/schema

PUT

使用schema更新已存在的表

curl -vi -X PUT \

-H "Accept: text/xml" \

-H "Content-Type: text/xml" \

-d ‘<?xml version="1.0" encoding="UTF-8"?><TableSchema name="users"><ColumnSchema name="cf" KEEP_DELETED_CELLS="true" /></TableSchema>‘ \

"http://example.com:8000/users/schema"

/table/schema

DELETE

删除表

curl -vi -X DELETE \

-H "Accept: text/xml" \

"http://example.com:8000/users/schema"

/table/regions

GET

列出表的所有regions

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/regions

Endpoint

HTTP Verb

说明

示例

/table/row/column:qualifier/timestamp

GET

取指定表指定列族下指定列的指定时间戳的值，返回的值为经过base64编码的，因此使用时需要做base64解码

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/row1"

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/row1/cf:a/1458586888395"

/table/row/column:qualifier

GET

取指定表的指定列族下指定列的值

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/row1/cf:a"

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/row1/cf:a/"

/table/row/column:qualifier/?v=number_of_versions

GET

取指定表的指定列族下指定列的指定版本值

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/row1/cf:a?v=2"

Endpoint

HTTP Verb

说明

示例

/table/scanner/

PUT

创建一个scanner

curl -vi -X PUT \

-H "Accept: text/xml" \

-H "Content-Type: text/xml" \

-d ‘<Scanner batch="1"/>‘ \

"http://example.com:8000/users/scanner/"

/table/scanner/

PUT

带Filter创建一个scanner，过滤器可以写在一个文本文件中，格式如：

<Scanner batch="100">

  <filter>

    {

      "type": "PrefixFilter",

      "value": "u123"

    }

  </filter>

</Scanner>

curl -vi -X PUT \

-H "Accept: text/xml" \

-H "Content-Type:text/xml" \

-d @filter.txt \

"http://example.com:8000/users/scanner/"

/table/scanner/scanner-id

GET

取下一批数据，如果已无数据，则返回的HTTP代码为204

curl -vi -X GET \

-H "Accept: text/xml" \

"http://example.com:8000/users/scanner/145869072824375522207"

table/scanner/scanner-id

DELETE

删除指定的scanner，释放资源

curl -vi -X DELETE \

-H "Accept: text/xml" \

"http://example.com:8000/users/scanner/145869072824375522207"

Endpoint

HTTP Verb

说明

示例

/table/row_key

PUT

往指定表写一行数据，注意行键、列族、列名和列值都必须采用base64编码

curl -vi -X PUT \

-H "Accept: text/xml" \

-H "Content-Type: text/xml" \

-d ‘<?xml version="1.0" encoding="UTF-8" standalone="yes"?><CellSet><Row key="cm93NQo="><Cell column="Y2Y6ZQo=">dmFsdWU1Cg==</Cell></Row></CellSet>‘ \

"http://example.com:8000/users/fakerow"

curl -vi -X PUT \

-H "Accept: text/json" \

-H "Content-Type: text/json" \

-d ‘{"Row":[{"key":"cm93NQo=", "Cell": [{"column":"Y2Y6ZQo=", "$":"dmFsdWU1Cg=="}]}]}‘‘ \

"example.com:8000/users/fakerow"

# 查看有哪些表

hbase(main):003:0> list

hbase(main):003:0> create ‘test‘, ‘cf‘  # 创建表test，一个列族cf

0 row(s) in 1.2200 seconds

hbase(main):003:0> list ‘test‘

# 只看指定namespace下的表

list_namespace_tables ‘default‘

hbase(main):003:0> desc ‘test‘  # 查看表状态

1 row(s) in 0.0550 seconds

hbase(main):004:0> put ‘test‘, ‘row1‘, ‘cf:a‘, ‘value1‘  # 往表test的cf列族的a字段插入值value1

0 row(s) in 0.0560 seconds

hbase(main):005:0> put ‘test‘, ‘row2‘, ‘cf:b‘, ‘value2‘

0 row(s) in 0.0370 seconds

hbase(main):006:0> put ‘test‘, ‘row3‘, ‘cf:c‘, ‘value3‘

0 row(s) in 0.0450 seconds

hbase(main):007:0> scan ‘test‘  # 扫描表test

ROW        COLUMN+CELL

row1       column=cf:a, timestamp=1288380727188, value=value1

row2       column=cf:b, timestamp=1288380738440, value=value2

row3       column=cf:c, timestamp=1288380747365, value=value3

3 row(s) in 0.0590 seconds

# 带条件scan

hbase(main):007:0> scan ‘test‘,{LIMIT=>1}

hbase(main):007:0> scan ‘test‘,{LIMIT=>1,STARTROW=>‘0000‘}

hbase(main):008:0> get ‘test‘, ‘row1‘  # 从表test取一行数据

COLUMN      CELL

cf:a        timestamp=1288380727188, value=value1

1 row(s) in 0.0400 seconds

# 取某列的数据

get ‘test‘, ‘row1‘, ‘cf1:col1‘

# 或者

get ‘test‘, ‘row1‘, {COLUMN=>‘cf1:col1‘}

语法：

get <table>,<rowkey>,[<family:column>,....]

scan <table>, {COLUMNS => [ <family:column>,.... ], LIMIT => num}

示例（在执行drop删除一个表之前，需先使用disable禁止该表）：

scan ‘ns1:t1‘, {COLUMNS => [‘c1‘, ‘c2‘], LIMIT => 10, STARTROW => 20180906}

hbase(main):012:0> disable ‘test‘

0 row(s) in 1.0930 seconds

hbase(main):013:0> drop ‘test‘

0 row(s) in 0.0770 seconds

# 清空一个表

truncate ‘test‘

# 查表行数用法1

count ‘test’

# 查表行数用法2

count ‘test‘,{INTERVAL=>10000}

# 删除行中的某个列值

delete ‘t1‘,‘row1‘,‘cf1:col1‘

# 删除整行

deleteall ‘t1‘,‘row1‘

# 退出hbase shell

hbase(main):014:0> exit

# 预分区1：命令行指定EndKey

create ‘mytable‘,‘cf1‘,SPLITS=>[‘EndKey1‘,‘EndKey2‘,‘EndKey3‘]

# 预分区2：文件指定EndKey（mytable.splits文件每行一个EndKey）

create ‘mytable‘,‘cf1‘,SPLITS_FILE=>‘mytable.splits‘

# 对例族做Snappy压缩

create ‘mytable‘,{NAME=>‘cf1‘,COMPRESSION=>‘SNAPPY‘},SPLITS=>[‘EndKey1‘,‘EndKey2‘,‘EndKey3‘]

# 分割策略分区

create ‘mytable‘,‘cf1‘,{NUMREGIONS=>7,SPLITALGO=>‘HexStringSplit‘}

bin/hbase org.apache.hadoop.hbase.mapreduce.RowCounter ‘test‘

hbase(main):007:0> help "count"  # 查看count命令的用法

hbase(main):007:0> create_namespace ‘mynamespace‘  # 创建名字空间

hbase(main):007:0> delete_namespace ‘mynamespace‘  # 删除名字空间

hbase(main):007:0> list_namespace  # 列出名字空间

test=get_table "test"  # 后续可以对象方式操作表"test"

test.count  # 统计表行数

test.scan  # 扫描整张表

test.scan LIMIT=>1  # 有限扫描

t.scan LIMIT=>1,STARTROW=>"0"  # 带条件有限扫描

import org.apache.hadoop.hbase.filter.SingleColumnValueFilter

import org.apache.hadoop.hbase.filter.CompareFilter

import org.apache.hadoop.hbase.util.Bytes

# 包含所有列

scan ‘test‘,{STARTROW =>‘2016081100AA1600011516‘, STOPROW =>‘2016081124ZZ1600011516‘,LIMIT=>2, FILTER=>SingleColumnValueFilter.new(Bytes.toBytes(‘cf1‘),Bytes.toBytes(‘id‘),CompareFilter::CompareOp.valueOf(‘EQUAL‘),Bytes.toBytes(‘1299840901201608111600011516‘))}

# 不包含过滤的列的其它所有列

import org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter

scan ‘test‘,{STARTROW =>‘2016081100AA1600011516‘, STOPROW =>‘2016081124ZZ1600011516‘,LIMIT=>2, FILTER=>SingleColumnValueExcludeFilter.new(Bytes.toBytes(‘cf1‘),Bytes.toBytes(‘id‘),CompareFilter::CompareOp.valueOf(‘EQUAL‘),Bytes.toBytes(‘1299840901201608111600011516‘))}

# 预分区建表（splits是针对整个表的，而非某列族,因此独立的{}）

create ‘test‘,{NAME => ‘cf1‘, VERSIONS => 1},{SPLITS_FILE => ‘splits.txt‘}

<property>

<name>hbase.coprocessor.master.classes</name>

<value>org.apache.hadoop.hbase.security.access.AccessController</value>

</property>

<property>

<name>hbase.coprocessor.region.classes</name>

<value>

org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController

</value>

</property>

grant <user> <permissions> <table> [ <column family> [ <column qualifier> ] ]

revoke <user> <table> [ <column family> [ <column qualifier> ] ]

alter ‘tablename‘, {OWNER => ‘username‘}

http://192.168.1.31:8080/region.jsp?name=72d8e0fec34a847d6e8c6ad62e6ec154

Region: test,196401049269744,1506394586723.72d8e0fec34a847d6e8c6ad62e6ec154.

Column Family: cf1

Memstore size (MB): 48

Store Files

4 StoreFile(s) in set.

Store File Size (MB) Modification time

hdfs://hadoop/hbase/data/default/test/72d8e0fec34a847d6e8c6ad62e6ec154/cf1/1d0a9744d1084de7b87629bad9459ac9 1041 Tue Sep 26 10:59:17 CST 2017

hdfs://hadoop/hbase/data/default/test/72d8e0fec34a847d6e8c6ad62e6ec154/cf1/56ab5f353fe94c15b183d4cf0880772a 206 Tue Sep 26 10:59:44 CST 2017

hdfs://hadoop/hbase/data/default/test/72d8e0fec34a847d6e8c6ad62e6ec154/cf1/a1b48379e66846a8b15bf42e830e0711 68 Tue Sep 26 10:59:52 CST 2017

hdfs://hadoop/hbase/data/default/test/72d8e0fec34a847d6e8c6ad62e6ec154/cf1/69595130c92c439ca939f366bcd2a843 68 Tue Sep 26 11:00:02 CST 2017

graceful_stop.sh --restart --reload --debug regionserver_hostname

Valid region move targets:

hadoop-095,16020,1505315231353

hadoop-092,16020,1505315376659

hadoop-094,16020,1505389352958

hadoop-096,16020,1505459565691

[main] region_mover: Moving 60 region(s) from hadoop-091,16020,1505315285824 on 4 servers using 1 threads.

thread-pool.rb:28] region_mover: Moving region ... (1 of 60) to ... for ...

thread-pool.rb:28] region_mover: Moving region ... from ... to ...

thread-pool.rb:28] region_mover: Moved region ... cost: 0.646

Reloaded hadoop-091 region(s)

Restoring balancer state to true

hbase(main):042:0> help "balance_switch"

Enable/Disable balancer. Returns previous balancer state.

Examples:

  hbase> balance_switch true

  hbase> balance_switch false

hbase(main):041:0> help "split"

Split entire table or pass a region to split individual region.  With the

second parameter, you can specify an explicit split key for the region.

Examples:

    split ‘TABLENAME‘

    split ‘REGIONNAME‘

    split ‘ENCODED_REGIONNAME‘

    split ‘TABLENAME‘, ‘splitKey‘

    split ‘REGIONNAME‘, ‘splitKey‘

    split ‘ENCODED_REGIONNAME‘, ‘splitKey‘

hbase(main):039:0> help "merge_region"

Merge two regions. Passing ‘true‘ as the optional third parameter will force

a merge (‘force‘ merges regardless else merge will fail unless passed

adjacent regions. ‘force‘ is for expert use only).

You can pass the encoded region name or the full region name.  The encoded

region name is the hash suffix on region names: e.g. if the region name were

TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396. then

the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396

Examples:

  hbase> merge_region ‘FULL_REGIONNAME‘, ‘FULL_REGIONNAME‘

  hbase> merge_region ‘FULL_REGIONNAME‘, ‘FULL_REGIONNAME‘, true

  hbase> merge_region ‘ENCODED_REGIONNAME‘, ‘ENCODED_REGIONNAME‘

  hbase> merge_region ‘ENCODED_REGIONNAME‘, ‘ENCODED_REGIONNAME‘, true

hbase(main):003:0> merge_region ‘000d96eef8380430d650c6936b9cef7d‘,‘b27a07c88dbbc070f716ee87fab15106‘

0 row(s) in 0.0730 seconds

hbase hfile -p -f /hbase/data/default/test/045ed11a1fdcd8bfa621f1160788a124/cf1/2502aeb799b84dda8340cdf4ad59e1f8

test是表名，045ed11a1fdcd8bfa621f1160788a124是Region name，2502aeb799b84dda8340cdf4ad59e1f8是hfile文件

K: 471200000000798496948594841700001313/cf1:c1/1493957134206/Put/vlen=28/seqid=0 V: 1214271401201503056013176259

K: 471200000000798496948594841700001313/cf1:c2/1493957134206/Put/vlen=4/seqid=0 V: 110

K: 471200000000798496948594841700001313/cf1:c3/1493957134206/Put/vlen=3/seqid=0 V: ABC

K: 471200000000798496948594841700001313/cf1:c4/1493957134206/Put/vlen=12/seqid=0 V: 16.12.57.1

K: 471200000000798496948594841700001313/cf1:c5/1493957134206/Put/vlen=28/seqid=0 V: 1214271401201503051700001313

K: 471200000000798496948594841700001313/cf1:c6/1493957134206/Put/vlen=0/seqid=0 V:

K: 471200000000798496948594841700001313/cf1:c7/1493957134206/Put/vlen=19/seqid=0 V: 2015-09-05 12:05:15

K: 471200000000798496948594841700001313/cf1:c8/1493957134206/Put/vlen=3/seqid=0 V: bko

move ‘被迁region的ENCODED值‘,‘目标RegionServer全名‘

move ‘bd118dd803bb4c8f8e28ea87ebec8335‘,‘hadoop-391,16020,1499860514933‘

306      hdfs:///hbase/data/default/mytable/.tabledesc

0        hdfs:///hbase/data/default/mytable/.tmp

2866280  hdfs:///hbase/data/default/mytable/023b0259adb715e3b4f6abecc1073ef3

2846541  hdfs:///hbase/data/default/mytable/2d2b9c6497e94a1f5227e9e4d26c4dc3

2911682  hdfs:///hbase/data/default/mytable/46e92458ce457ae69650013ca12e2415

2846062  hdfs:///hbase/data/default/mytable/6c1e8b29a5fc941434343dce11648e01

2846135  hdfs:///hbase/data/default/mytable/6c86670d1d3a2b23d9eee836efb959ca

2875079  hdfs:///hbase/data/default/mytable/7ee492fbc245a34a075e961f75e20f85

2846527  hdfs:///hbase/data/default/mytable/8b41e528bd529b123c9ce8d66816a21b

2846194  hdfs:///hbase/data/default/mytable/913f32ee373d382be3062562b87367fe

2846291  hdfs:///hbase/data/default/mytable/b9a1ef3d48f3fa45637473ae82d258c1

2912207  hdfs:///hbase/data/default/mytable/b9f3d37ce1c7a73035a517decde7ffec

2846121  hdfs:///hbase/data/default/mytable/bd118dd803bb4c8f8e28ea87ebec8335

2912197  hdfs:///hbase/data/default/mytable/c2c25cdc9376377c96b9d45768c7d793

2846253  hdfs:///hbase/data/default/mytable/c9e1362c88aeae241a5737989d2c8bf8

2911869  hdfs:///hbase/data/default/mytable/cf4f2bda053820b56db112929e077b74

2913065  hdfs:///hbase/data/default/mytable/e6ed2d434d3e0b8b59f859fdcb615a80

2912094  hdfs:///hbase/data/default/mytable/fec9c2376b0c3466c6f872aef3c3177d

$ hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog

Usage: FSHLog <ARGS>

Arguments:

 --dump  Dump textual representation of passed one or more files

         For example: FSHLog --dump hdfs://example.com:9000/hbase/.logs/MACHINE/LOGFILE

 --split Split the passed directory of WAL logs

         For example: FSHLog --split hdfs://example.com:9000/hbase/.logs/DIR

$ hbase org.apache.hadoop.hbase.regionserver.wal.FSHLog --dump hdfs:///hbase/WALs/hadoop-203,16020,1493809707539/hadoop-203%2C16020%2C1493809707539.1506389208593

Writer Classes: ProtobufLogWriter

Cell Codec Class: org.apache.hadoop.hbase.regionserver.wal.WALCellCodec

Sequence=117 , region=f9c1b75be5fcb024a9cf74c47affb52e at write timestamp=Tue Sep 26 09:32:34 CST 2017

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770146, column=cf1:js

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770146, column=cf1:lsud

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770146, column=cf1:at

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770146, column=cf1:rat

Sequence=118 , region=f9c1b75be5fcb024a9cf74c47affb52e at write timestamp=Tue Sep 26 09:32:44 CST 2017

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770160, column=cf1:js

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770160, column=cf1:lsud

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770160, column=cf1:at

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748770160, column=cf1:rat

Sequence=119 , region=f9c1b75be5fcb024a9cf74c47affb52e at write timestamp=Tue Sep 26 09:32:51 CST 2017

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748767888, column=cf1:js

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748767888, column=cf1:lsud

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748767888, column=cf1:at

row=g-FMZffo5E36_sJCNx2BS4jJorpo-616451068748767888, column=cf1:rat

hbase(main):001:0> major_compact

ERROR: wrong number of arguments (0 for 1)

Here is some help for this command:

          Run major compaction on passed table or pass a region row

          to major compact an individual region. To compact a single

          column family within a region specify the region name

          followed by the column family name.

          Examples:

          Compact all regions in a table:

          hbase> major_compact ‘t1‘

          hbase> major_compact ‘ns1:t1‘

          Compact an entire region:

          hbase> major_compact ‘r1‘

          Compact a single column family within a region:

          hbase> major_compact ‘r1‘, ‘c1‘

          Compact a single column family within a table:

          hbase> major_compact ‘t1‘, ‘c1‘

hbase(main):003:0> major_compact ‘a6e65f0540bfcd1cb2740bb4b033d134‘,‘cf2‘

0 row(s) in 0.1990 seconds

hbase org.apache.hadoop.hbase.mapreduce.Export

hbase org.apache.hadoop.hbase.mapreduce.Import

hbase org.apache.hadoop.hbase.mapreduce.CopyTable

hadoop distcp

snapshot ‘test‘,‘test.snapshot‘

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot test.snapshot  -copy-from hdfs://192.168.31.30/hbase -copy-to hdfs://192.168.32.30/hbase -mappers 10 -bandwidth 50

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -overwrite -snapshot test.snapshot -copy-from hdfs://192.168.31.30/hbase -copy-to hdfs://192.168.32.30/hbase -mappers 10 -bandwidth 30 -chuser hbase -chgroup supergroup

hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -overwrite -snapshot test.snapshot  -copy-from hdfs://192.168.31.30/hbase -copy-to hdfs://192.168.32.30/hbase -mappers 10 -bandwidth 50

restore_snapshot ‘test.snapshot’

hbase \

org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles \

-Dhbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily=1024 \

hdfs://192.168.31.30/hbase/archive/datapath/tablename/filename \

tablename

delete_snapshot ‘test.snapshot’

list_snapshots

[zk: localhost:2181(CONNECTED) 0] ls /hbase

[backup-masters, draining, flush-table-proc, hbaseid, master, master-maintenance, meta-region-server, namespace, online-snapshot, replication, rs, running, splitWAL, switch, table, table-lock]

hbase(main):015:0* scan ‘hbase:meta‘,{LIMIT=>10}

hbase:acl,,1460426731436.0bbdf170c309223c0ce830 column=info:regioninfo, timestamp=1460426830411, value={ENCODED => 0bbdf170c309223c0ce830facdff9edd, NAME => ‘hbase:acl,,1460426731436.0bbdf

 facdff9edd.                                     170c309223c0ce830facdff9edd.‘, STARTKEY => ‘‘, ENDKEY => ‘‘}                                                                                

 hbase:acl,,1460426731436.0bbdf170c309223c0ce830 column=info:seqnumDuringOpen, timestamp=1461653766642, value=\x00\x00\x00\x00\x00\x00\x002                                                  

 facdff9edd.                                                                                                                                                                                 

 hbase:acl,,1460426731436.0bbdf170c309223c0ce830 column=info:server, timestamp=1461653766642, value=hadoop-034:16020                                                                         

 facdff9edd.                                                                                                                                                                                 

 hbase:acl,,1460426731436.0bbdf170c309223c0ce830 column=info:serverstartcode, timestamp=1461653766642, value=1461653610096   

$ hdfs dfs -ls hdfs:///hbase/data/abc/test/

Found 7 items

drwxr-xr-x   - hadoop supergroup        0 2017-07-19 17:45 hdfs:///hbase/data/abc/test/.tabledesc

drwxr-xr-x   - hadoop supergroup        0 2017-07-19 17:45 hdfs:///hbase/data/abc/test/.tmp

drwxr-xr-x   - hadoop supergroup        0 2019-05-30 16:07 hdfs:///hbase/data/abc/test/37b03ca897147840c3676bb7d622af2f

drwxr-xr-x   - hadoop supergroup        0 2019-05-30 16:07 hdfs:///hbase/data/abc/test/8e52bcdc2b1292fabf6bc8ea8e8be8ba

drwxr-xr-x   - hadoop supergroup        0 2019-09-07 09:36 hdfs:///hbase/data/abc/test/ba6fa3def428d0d5e53a17d30c5fa7de

drwxr-xr-x   - hadoop supergroup        0 2019-09-07 09:36 hdfs:///hbase/data/abc/test/cdf6d4ff4dc0b6cf1c23e1db133dbfe1

drwxr-xr-x   - hadoop supergroup        0 2018-04-17 23:05 hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092

$ hdfs dfs -ls hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092

Found 4 items

-rw-r--r--  3 hadoop supergroup     99 2018-04-17 23:05 hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092/.regioninfo

drwxr-xr-x  - hadoop supergroup     0 2018-04-17 23:06 hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092/.tmp

drwxr-xr-x  - hadoop supergroup     0 2018-04-17 23:06 hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092/cf1

drwxr-xr-x  - hadoop supergroup    0 2019-10-15 10:56 hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092/recovered.edits

$ hdfs dfs -du -h hdfs:///hbase/data/abc/

4.7 G    hdfs:///hbase/data/abc/test

$ hdfs dfs -du -h hdfs:///hbase/data/abc/test

287       hdfs:///hbase/data/abc/test/.tabledesc

0         hdfs:///hbase/data/abc/test/.tmp

167.1 M   hdfs:///hbase/data/abc/test/37b03ca897147840c3676bb7d622af2f

137.3 M   hdfs:///hbase/data/abc/test/8e52bcdc2b1292fabf6bc8ea8e8be8ba

2.6 G     hdfs:///hbase/data/abc/test/ba6fa3def428d0d5e53a17d30c5fa7de

870.1 M   hdfs:///hbase/data/abc/test/cdf6d4ff4dc0b6cf1c23e1db133dbfe1

1000.0 M  hdfs:///hbase/data/abc/test/fb3b4847d6cf504aea3990859e2b8092

http://192.168.1.31:8080/region.jsp?name=78d2bddb0fdc2c735c79da68658f3011

hdfs://hadoop/hbase/data/default/test/78d2bddb0fdc2c735c79da68658f3011/cf1/4facea216b15471e9d6d85cf59bd9d8a

hdfs://hadoop/hbase/data/default/test/78d2bddb0fdc2c735c79da68658f3011/cf1/404aea6371cb4722af91a07c10a3fcf3

$ hdfs dfs -ls hdfs://hadoop/hbase/data/default/test/78d2bddb0fdc2c735c79da68658f3011/cf1/4facea216b15471e9d6d85cf59bd9d8a

-rw-r--r--   3 zhangsan supergroup 6283978864 2017-09-24 12:47 hdfs://hadoop/hbase/data/default/test/78d2bddb0fdc2c735c79da68658f3011/cf1/4facea216b15471e9d6d85cf59bd9d8a

test,83--G40V6UdCnEHKSKqR_yjJo798594847946710200000795,1461323021820.d4cc7afbc2d6bf3843c121fedf4d696d.

t_user,,1461549916081.f4e17b0d99f2d77da44ccb184812c345.

PMONITOR=/usr/local/bin/process_monitor.sh

JAVA=/usr/local/jdk/bin/java

HBASE_DAEMON=/data/hbase/bin/hbase-daemon.sh

# 监控HBase Master（仅在 Master 上启动）

#* * * * * $PMONITOR "$JAVA -Dproc_master" "$HBASE_DAEMON start master"

# 监控HBase RegionServer（仅在 RegionServer 上启动）

#* * * * * $PMONITOR "$JAVA -Dproc_regionserver" "$HBASE_DAEMON start regionserver"

# 监控HBase ThriftServer2（一般和 RegionServer 共享相同机器）

#* * * * * $PMONITOR "$JAVA -Dproc_thrift2" "$HBASE_DAEMON start thrift2 --framed -threadedselector -s 10 -m 20 -w 20"

# 监控HBase RESTServer（一般和 RegionServer 共享相同机器）

#* * * * * $PMONITOR "$JAVA -Dproc_rest" "$HBASE_DAEMON start rest -p 8080"

# 设置环境变量

export H=‘192.168.1.21,192.168.1.22,192.168.1.23,192.168.1.24,192.168.1.25,192.168.1.26‘

export U=root

export P=password

export PORT=201810

# 上传/etc/hosts和/etc/profile到：192.168.31.12,192.168.31.13,192.168.31.14,192.168.31.15

mooon_upload -s=/etc/hosts,/etc/profile -d=/etc

# 检查/etc/profile文件是否一致

mooon_ssh -c=‘md5sum /etc/hosts‘

作用

说明

新生代（Young）

存活时间较短，一般存储刚生成的一些对象

又分为一个伊甸园（Eden）区，和两个Survivor区，两个Survivor区总有一个是空的。对新生代的垃圾回收叫做Minor GC，特点是次数频繁，但每次回收时间较短。

对象在Survivor每经历一次Minor GC，年龄就增长1岁，默认增长到15岁时就晋升到Old代。

老年代（Tenured）

也叫旧生成（Old），存活时间较长，主要存储在应用程序中生命周期较长的对象

对象优先在Eden上分配，但大对象直接在Old上分配。对老年代的垃圾回收叫做MajorGC或Full GC，特点是次数相对少，每次回收时间较长。

永久代（Perm）

存储meta和class的信息，JDK8已删除永久代

永久代垃圾回收比较少，效率也比较低。因为在JDK8中已无永久代，所以JVM参数“-XX:PermSize”和 “-XX:MaxPermSize”已作废，代替的参数分别为“-XX:MetaspaceSiz”和“-XX:MaxMetaspaceSize”。

回收策略

Serial

串行收集器

串行单线程处理所有垃圾回收工作，适合小数据量的回收，使用“-XX:+UseSerialGC”打开。

Parrallel New Collector

并行收集器

Serial的多线程版本，并行多线程处理，使用“-XX:+UseParallelOldGC”打开，用“-XX:ParallelGCThreads=<N>”指定垃圾回收线程数。

CMS（Concurrent Mark Sweep）

并发收集器，响应时间优先回收器

并发多线程处理，使用“-XX:+UseConcMarkSweepGC”打开，只针对老年代。

并行

Parallel

多条垃圾收集线程并行工作，应用线程处于等待状态

并发

Concurrent

垃圾收集线程与应用线程一段时间内同时工作，不是并行而是交替执行

参数名

参数说明

示例

-Xms

初始Java堆大小，JVM启动时分配的内存

-Xms256m

-Xmx

最大Java堆大小，运行时可分配的最大内存

-Xmx2048m

-Xss

Java线程堆栈大小，每个线程分配的内存大小

-Xss128m

-XX:OnOutOfMemoryError

内存溢出时的动作

-XX:OnOutOfMemoryError=‘kill -9 %p‘

-XX:+UseConcMarkSweepGC

设置并发垃圾收集器

-XX:+UseConcMarkSweepGC

-XX:+UseParallelGC

设置并行垃圾收集器，同时运行在多CPU上，独占式收集器（收集过程中进程暂停STW），不能与CMS收集器一起。使用这个参数，新生代为并行回收，老年代为串行回收。

-XX:+UseParallelGC

-XX:+UseSerialGC

设置串行垃圾收集器，也是独占式收集器。使用这个参数，新生代和老年代都为串行回收。

-XX:+UseSerialGC

-XX:+UseParNewGC

设置并发串行收集器，实际是串行垃圾回收器的多线程化，即串行垃圾回收器的多线程版本，也是独占式收集器。使用这个参数时，新生代为并行回收，老年代仍是串行回收。

-XX:+UseParNewGC

-XX:+UseParallelOldGC

新生代和老年代都使用并行收集器

-XX:+UseParallelOldGC

-XX:+UseConcMarkSweepGC

老年代回收器，停顿（STW）减少，但吞吐量会降低。

-XX:+UseConcMarkSweepGC

-XX:ParallelGCThreads

设置并行收集线程数

-XX:ParallelGCThreads=10