码迷,mamicode.com
首页 > 其他好文 > 详细

splittability A SequenceFile can be split by Hadoop and distributed across map jobs whereas a GZIP file cannot be.

时间:2017-11-01 01:11:35      阅读:181      评论:0      收藏:0      [点我收藏+]

标签:get   pow   multi   another   ado   read   other   eve   type   

splittability

 

 

 
Skip to end of metadata
 
Go to start of metadata
 

Compressed Data Storage

Keeping data compressed in Hive tables has, in some cases, been known to give better performance than uncompressed storage; both in terms of disk usage and query performance.

You can import text files compressed with Gzip or Bzip2 directly into a table stored as TextFile. The compression will be detected automatically and the file will be decompressed on-the-fly during query execution. For example:

CREATE TABLE raw (line STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘ LINES TERMINATED BY ‘\n‘;
 
LOAD DATA LOCAL INPATH ‘/tmp/weblogs/20090603-access.log.gz‘ INTO TABLE raw;

The table ‘raw‘ is stored as a TextFile, which is the default storage. However, in this case Hadoop will not be able to split your file into chunks/blocks and run multiple maps in parallel. This can cause underutilization of your cluster‘s ‘mapping‘ power.

【 A SequenceFile can be split by Hadoop and distributed across map jobs whereas a GZIP file cannot be.】

The recommended practice is to insert data into another table, which is stored as a SequenceFile. A SequenceFile can be split by Hadoop and distributed across map jobs whereas a GZIP file cannot be. For example:

CREATE TABLE raw (line STRING)
   ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘ LINES TERMINATED BY ‘\n‘;
 
CREATE TABLE raw_sequence (line STRING)
   STORED AS SEQUENCEFILE;
 
LOAD DATA LOCAL INPATH ‘/tmp/weblogs/20090603-access.log.gz‘ INTO TABLE raw;
 
SET hive.exec.compress.output=true;
SET io.seqfile.compression.type=BLOCK; -- NONE/RECORD/BLOCK (see below)
INSERT OVERWRITE TABLE raw_sequence SELECT * FROM raw;

The value for io.seqfile.compression.type determines how the compression is performed. Record compresses each value individually while BLOCK buffers up 1MB (default) before doing compression.

LZO Compression

See LZO Compression for information about using LZO with Hive.

 

splittability A SequenceFile can be split by Hadoop and distributed across map jobs whereas a GZIP file cannot be.

标签:get   pow   multi   another   ado   read   other   eve   type   

原文地址:http://www.cnblogs.com/yuanjiangw/p/7764426.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!