码迷,mamicode.com
首页 > 其他好文 > 详细

hive优化1-列转行

时间:2020-12-22 11:49:51      阅读:0      评论:0      收藏:0      [点我收藏+]

标签:you   tab   span   into   通过   行数据   标识   div   value   

数据说明:table1的数据是 客户id,基金差额,保险差额,理财差额,。。。

create table youhua1(
    -> custid int
    -> ,jijin_change float
    -> ,baoxian_change float
    -> ,cunkuan_change float)
    -> ;

insert into youhua1 values(1,1.1,1.2,1.3),(2,2.67,2.34,2.1),(3,3.1,3.4,3.78);

select * from youhua1;
+--------+--------------+----------------+----------------+
| custid | jijin_change | baoxian_change | cunkuan_change |
+--------+--------------+----------------+----------------+
|      1 |          1.1 |            1.2 |            1.3 |
|      2 |         2.67 |           2.34 |            2.1 |
|      3 |          3.1 |            3.4 |           3.78 |
+--------+--------------+----------------+----------------+

原查询思路:

select 
     custid 
    ,"基金" as prod_name
    ,jijin_change as 差额
from youhua1

union all
select 
     custid 
    ,"保险" as prod_name
    ,baoxian_change 差额
from youhua1

union all
select 
     custid 
    ,"存款" as prod_name
    ,cunkuan_change as 差额
from youhua1;

目标数据:客户1,基金,差额(基金)
                  客户1,保险,差额(保险)

                  ...

+--------+-----------+------+
| custid | prod_name | 差额 |
+--------+-----------+------+
|      1 | 基金      |  1.1 |
|      2 | 基金      | 2.67 |
|      3 | 基金      |  3.1 |
|      1 | 保险      |  1.2 |
|      2 | 保险      | 2.34 |
|      3 | 保险      |  3.4 |
|      1 | 存款      |  1.3 |
|      2 | 存款      |  2.1 |
|      3 | 存款      | 3.78 |
+--------+-----------+------+

优化方案:

当数据量很大时,这种方式会造成多次读表。

可以采用from前置的方式;

也可以先通过concat函数拼接jijin_change、baoxian_change、cunkuan_change三个字段成一列,然后再用列转行函数lateral view explode函数处理。

行转列:一个字段的多行数据合进一个列;列转行:一个字段的一列数据拆到多个行

select custid,concat_ws(,,jijin_change,baoxian_change,cunkuan_change) from youhua1;
+--------+-----------------------------------------------------------+
| custid | concat_ws(,,jijin_change,baoxian_change,cunkuan_change) |
+--------+-----------------------------------------------------------+
|      1 | 1.1,1.2,1.3                                               |
|      2 | 2.67,2.34,2.1                                             |
|      3 | 3.1,3.4,3.78                                              |
+--------+-----------------------------------------------------------+

 

select 
         a.custid
    ,change 
from (
     select
           custid
          ,concat_ws(,,jijin_change,baoxian_change,cunkuan_change) as all_change
     from youhua1
    ) as a
lateral view explode(split(a.all_change)) table_tmp as change;

 没有加上基金、保险、存款的行标识:

可以在concat_ws之前先将基金、保险、存款的行标识和值打在一起,concat_ws("+","基金"+"jijin_change"),...

在explode时指定split(a.all_change, ‘,‘),最后select custid,split(change,"+")[0] as prod_name,split(change,"+")[1] as change

 

hive优化1-列转行

标签:you   tab   span   into   通过   行数据   标识   div   value   

原文地址:https://www.cnblogs.com/foolangirl/p/14145147.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!