与hbase外部表(wizad_mdm_main)进行join出现问题:
CREATE TABLE wizad_mdm_dev_lmj_edition_result as
select *
from wizad_mdm_dev_lmj_20141120 as w
JOIN wizad_mdm_main as a ON (a.rowkey = w.guid);
解决办法:
SET hive.auto.convert.join=false; 关闭自动转化MapJoin,默认为true;
SET hive.ignore.mapjoin.hint=false; 关闭忽略mapjoin的hints(不忽略,hints有效),默认为true(忽略hints)。
然后在查询时候使用hints,/*+ mapjoin(w) */ 将小表w (wizad_mdm_dev_lmj_edition_20141120) 加入到DistributeCache,
Map task在扫描HBase中的大表阶段,就可以完成join操作:
SET mapred.job.queue.name=queue3;
SET hbase.client.scanner.caching=5000;
SET hbase.zookeeper.quorum=datanode06,datanode07,datanode08;
SET zookeeper.znode.parent=/hbase;
set hbase.regionserver.lease.period=180000;
SET hive.auto.convert.join=false;
SET hive.ignore.mapjoin.hint=false;
INSERT OVERWRITE LOCAL DIRECTORY ‘/home/wizad/lmj/‘
ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,‘
select /*+ mapjoin(w) */ cookie_id,rowkey,fixeddim_map
from wizad_mdm_dev_lmj_edition_20141120 as w JOIN wizad_mdm_main as a
ON (w.guid = a.rowkey);