标签:tar 计数 lang nio ado 复制 默认 timeout sql
数据处理平台从oracle迁移到hadoop平台后,原有的数据处理过程需要改写为sparkSql。由于sparkSql执行环境和数据的不确定,经常有资源竞争导致处理过程意外停止,数据倾斜导致任务执行失败。 为减少出错概率,需要对sparkSql进行规范与优化。
通常数据倾斜有以下几种现象:
1.检查group by 字段是否全为一个值或空, 从group中移除只用一个值得字段
2. sql使用大表反复关联小表如:
) a
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘AAC004‘) c
on a.AAC004 = c.aaa102
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘AAC011‘) d
on a.aac011 = d.aaa102
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘YAB019‘) e
on a.yab019 = e.aaa102
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘AAB019‘) f
on a.aab019 = f.aaa102
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘YAB003‘) g
on a.aab301 = g.aaa102
left join (select m.aaa102,m.aaa103 from database.aa11 m where m.aaa100 = ‘AAB301‘) h
on a.yac145 = h.aaa102
默认小于150M的表会广播(broacast),如上处理后将不能被广播, 修改为:
) a
left join database.aa11 c
on a.AAC004 = c.aaa102 and c.aaa100 = ‘AAC004‘
left join database.aa11 d
on a.aac011 = d.aaa102 and d.aaa100 = ‘AAC011‘
left join database.aa11 e
on a.yab019 = e.aaa102 and e.aaa100 = ‘YAB019‘
left join database.aa11 f
on a.aab019 = f.aaa102 and f.aaa100 = ‘AAB019‘
left join database.aa11 g
on a.aab301 = g.aaa102 and g.aaa100 = ‘YAB003‘
left join database.aa11 h
on a.yac145 = h.aaa102 and h.aaa100 = ‘AAB301‘
oracleSQL 转 SPARKSQL(hiveSql) 及常用优化
标签:tar 计数 lang nio ado 复制 默认 timeout sql
原文地址:https://www.cnblogs.com/foolaround/p/12960277.html