标签:翻译 ssi reg 导致 时间比较 load char size opera
自从16年之后,因为工作原因,项目中就没有再使用oracle了,最近最近支持一个项目,又要开始负责这块事情了。最近在跑性能测试,配置全部调好之后,不少sql还存在性能低下的问题,主要涉及执行计划的不合理,以及相关pga隐含参数的优化。可能因为几年不接触的原因,略微有些生疏需要review了。这里以最近优化过的某个比较典型的例子为例(这里只讲思路、因为涉及到核心机密,不给出最终结果,16C E5620@2.40GHz/45GB内存/fio 85/15 iops 8500/1500 配置,优化前100G临时空间都爆掉,到20分钟出结果,实际上允许完全调整的话,最多5分钟就可以运行出来),这个例子比较经典,它是一个产品转换的例子,从业务逻辑上来看,它需要对两个100w(100管理人*1w支产品)记录的结果集进行黑名单(是指A产品->B产品是否允许转换,A->B不允许,不意味着B->A不允许)以外的笛卡尔积关联,,重点回顾下一些重要的operation以及如何通过hint强制,子查询独立运行时的operation和作为主查询的子查询运行时的operation不相同可能会造成性能差异巨大。首先,不要指望写一段sql,既可以在oracle下运行、也可以在mysql下运行。原始的sql语句以及执行计划如下:
INSERT INTO C3( tenantid,c_code,l_serialno,c_txtfundcode, c_chargetype,f_sharemin,f_sharemax,l_hold,c_txtothercode, c_targetchargetype,c_custtype,c_flag,d_operatedate,d_cdate,c_cyno) SELECT a.tenantid,a.c_code,l_serialno, CASE a.agencytypes WHEN ‘AB‘ THEN a.c_prdcode ELSE COALESCE(b.c_outprdcode,a.c_prdcode) END c_txtfundcode, c_chargetype,f_sharemin,f_sharemax,l_hold, CASE a.agencytypes WHEN ‘AB‘THEN a.c_othercode ELSE COALESCE(c.c_outprdcode,a.c_othercode) END c_txtothercode, c_targetchargetype,c_custtype,c_flag,d_operatedate,20100511 d_cdate,a.c_cyno FROM (SELECT 0 l_serialno, CASE t.c_mutextype WHEN ‘A‘ THEN ‘0‘ WHEN ‘B‘ THEN ‘1‘ WHEN ‘C‘ THEN ‘2‘ ELSE t.c_mutextype END c_chargetype, CASE t.c_targetsharetype WHEN ‘A‘ THEN ‘0‘ WHEN ‘B‘ THEN ‘1‘ WHEN ‘C‘ THEN ‘2‘ ELSE t.c_targetsharetype END c_targetchargetype, ar.f_cminmutex f_sharemin,99999999999999.9 f_sharemax,0 l_hold,‘2‘ c_custtype, ‘1‘ c_flag,‘20000101‘ d_operatedate, t.* FROM (SELECT fo.tenantid, fo.c_code, fo.c_prdcode,fi.c_prdcode c_othercode, fo.c_cyno,fo.c_mutextype,fi.c_mutextype c_targetsharetype, fo.d_contractdate d_contractdate, fo.c_mutextypes c_mutextypes, fo.agencytypes, fi.d_contractdate d_othercontractdate, fi.c_mutextypes c_othersharetypes FROM (SELECT DISTINCT ss.tenantid, ss.c_code, ss.c_prdcode, ss.c_cyno,ss.c_mutextype c_mutextype, ta.c_mutextypes agencytypes, getsysvalue(ss.tenantid, ss.c_code, ss.c_cyno,ss.c_managercode, ss.c_prdcode,‘ContractDate‘, ‘20991231‘) d_contractdate, getsysvalue(ss.tenantid, ss.c_code, ss.c_cyno,ss.c_managercode, ss.c_prdcode,‘ShareTypes‘, ‘‘) c_mutextypes FROM (SELECT fi.tenantid, fi.c_code, fi.c_prdcode,c.c_cyno, s.c_mutextype, fi.c_managercode FROM FUNDINFO fi, (SELECT ‘A‘ c_mutextype FROM DUAL UNION ALL SELECT ‘B‘ c_mutextype FROM DUAL UNION ALL SELECT ‘C‘ c_mutextype FROM DUAL) s, QUALIFY c, EXPBATCH d WHERE fi.c_code = ‘F6‘ AND fi.tenantid = ‘*‘ AND fi.c_fundstatus NOT IN (‘6‘, ‘9‘) AND c.c_code = ‘F6‘ AND c.tenantid = ‘*‘ AND c.c_cyno = d.c_cyno AND c.c_code = d.c_code AND c.tenantid = d.tenantid AND INSTR(fi.c_mutextypes, s.c_mutextype) > 0 AND (fi.c_fundtype<>‘1‘ OR NOT EXISTS(SELECT 1 FROM TEXT_PARAMETER WHERE c_paramitem = ‘ChangeLimit‘ AND c_paramvalue = ‘1‘ AND c_prdcode = fi.c_prdcode AND c_code = ‘F6‘ AND tenantid = ‘*‘)) AND fi.c_prdcode = c.c_prdcode AND c.c_issaleflag = ‘1‘ ) ss, EXPBATCH ta WHERE ss.c_cyno = ta.c_cyno AND ss.c_code = ta.c_code AND ss.tenantid = ta.tenantid AND ta.c_notexpparamfiles = ‘0‘ ) fo, (SELECT DISTINCT ss.tenantid, ss.c_code, ss.c_prdcode, ss.c_cyno,ss.c_mutextype c_mutextype, getsysvalue(ss.tenantid, ss.c_code, ss.c_cyno,ss.c_managercode, ss.c_prdcode,‘ContractDate‘, 20991231) d_contractdate, getsysvalue(ss.tenantid, ss.c_code, ss.c_cyno,ss.c_managercode, ss.c_prdcode,‘ShareTypes‘, ‘‘) c_mutextypes FROM (SELECT fi.tenantid, fi.c_code, fi.c_prdcode, c.c_cyno, s.c_mutextype, fi.c_managercode FROM FUNDINFO fi, (SELECT ‘A‘ c_mutextype FROM DUAL UNION ALL SELECT ‘B‘ c_mutextype FROM DUAL UNION ALL SELECT ‘C‘ c_mutextype FROM DUAL) s, QUALIFY c, EXPBATCH d WHERE fi.c_code = ‘F6‘ AND fi.tenantid = ‘*‘ AND c.c_code = ‘F6‘ AND c.tenantid = ‘*‘ AND c.c_cyno = d.c_cyno AND c.c_code = d.c_code AND c.tenantid = d.tenantid AND INSTR(fi.c_mutextypes, s.c_mutextype) > 0 AND fi.c_fundstatus NOT IN (‘6‘, ‘9‘) AND (fi.c_fundtype <> ‘1‘ OR NOT EXISTS (SELECT 1 FROM TEXT_PARAMETER WHERE c_paramitem = ‘ChangeLimit‘ AND c_paramvalue = ‘1‘ AND c_prdcode = fi.c_prdcode AND c_code = ‘F6‘ AND tenantid = ‘*‘)) AND fi.c_prdcode = c.c_prdcode AND c.c_issaleflag = ‘1‘ ) ss, EXPBATCH ta WHERE ss.c_cyno = ta.c_cyno AND ss.c_code = ta.c_code AND ss.tenantid = ta.tenantid AND ta.c_notexpparamfiles = ‘0‘ ) fi WHERE fo.c_prdcode <> fi.c_prdcode AND fo.c_mutextype = fi.c_mutextype AND fo.c_cyno = fi.c_cyno ) t,(SELECT c_prdcode, c_cyno, MIN(a.f_cminmutex) f_cminmutex FROM ARLIMIT a WHERE c_cyno <> ‘*‘ AND c_code = ‘F6‘ AND tenantid = ‘*‘ GROUP BY c_prdcode, c_cyno UNION ALL SELECT c_prdcode, b.c_cyno, MIN(f_cminmutex) f_cminmutex FROM ARLIMIT a, EXPBATCH b WHERE a.c_cyno = ‘*‘ AND a.c_code = ‘F6‘ AND a.tenantid = ‘*‘ AND a.c_code = b.c_code AND a.tenantid = b.tenantid AND not exists (SELECT 1 FROM ARLIMIT c WHERE b.c_cyno = c.c_cyno AND a.c_prdcode = c.c_prdcode AND a.c_code = c.c_code AND a.tenantid = c.tenantid) GROUP BY c_prdcode, b.c_cyno) ar WHERE t.d_contractdate <= 20100511 AND t.d_othercontractdate <= 20100511 AND INSTR(t.c_mutextypes, t.c_mutextype) > 0 AND INSTR(t.c_othersharetypes, t.c_targetsharetype) > 0 AND t.c_prdcode = ar.c_prdcode AND t.c_cyno = ar.c_cyno AND NOT EXISTS (SELECT 1 FROM CHANGELIMIT b WHERE t.c_prdcode = b.c_prdcode AND (t.c_mutextype = b.c_mutextype OR b.c_mutextype = ‘*‘) AND (t.c_cyno = b.c_cyno OR b.c_cyno = ‘*‘) AND (t.c_othercode = b.c_othercode OR b.c_othercode = ‘*‘) AND (t.c_targetsharetype = b.c_othershare OR b.c_othershare = ‘*‘) ) ) a LEFT JOIN FUNDCODECHANGE b ON (a.c_prdcode = b.c_prdcode AND a.c_mutextype = b.c_mutextype AND a.c_code = b.c_code AND a.tenantid = b.tenantid) LEFT JOIN FUNDCODECHANGE c ON (a.c_othercode = c.c_prdcode AND a.c_targetsharetype = c.c_mutextype AND a.c_code = c.c_code AND a.tenantid = c.tenantid)
-- 此处隐藏了谓词部分 Plan Hash Value : 4256998962 -------------------------------------------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost | Time | -------------------------------------------------------------------------------------------------------------------------- | 0 | INSERT STATEMENT | | 1 | 129 | 2103 | 00:00:26 | | 1 | LOAD TABLE CONVENTIONAL | C3 | | | | | | 2 | NESTED LOOPS OUTER | | 1 | 129 | 2103 | 00:00:26 | | 3 | VIEW | | 1 | 95 | 2102 | 00:00:26 | | * 4 | FILTER | | | | | | | * 5 | HASH JOIN | | 1 | 8105 | 2101 | 00:00:26 | | * 6 | HASH JOIN | | 1 | 8078 | 1827 | 00:00:22 | | 7 | NESTED LOOPS OUTER | | 1 | 4060 | 914 | 00:00:11 | | 8 | VIEW | | 1 | 4026 | 913 | 00:00:11 | | 9 | SORT UNIQUE | | 1 | 72 | 913 | 00:00:11 | | * 10 | FILTER | | | | | | | 11 | NESTED LOOPS | | 1 | 72 | 912 | 00:00:11 | | 12 | NESTED LOOPS | | 1 | 72 | 912 | 00:00:11 | | 13 | NESTED LOOPS | | 1 | 58 | 911 | 00:00:11 | | 14 | NESTED LOOPS | | 1 | 49 | 910 | 00:00:11 | | 15 | NESTED LOOPS | | 1500 | 46500 | 309 | 00:00:04 | | 16 | VIEW | | 3 | 9 | 6 | 00:00:01 | | 17 | UNION-ALL | | | | | | | 18 | FAST DUAL | | 1 | | 2 | 00:00:01 | | 19 | FAST DUAL | | 1 | | 2 | 00:00:01 | | 20 | FAST DUAL | | 1 | | 2 | 00:00:01 | | * 21 | TABLE ACCESS FULL | FUNDINFO | 500 | 14000 | 101 | 00:00:02 | | * 22 | TABLE ACCESS BY INDEX ROWID | QUALIFY | 1 | 18 | 1 | 00:00:01 | | * 23 | INDEX RANGE SCAN | UIDX_QUALIFY | 1 | | 1 | 00:00:01 | | * 24 | INDEX RANGE SCAN | PK_BATCH | 1 | 9 | 1 | 00:00:01 | | * 25 | INDEX RANGE SCAN | PK_EXPBATCH | 1 | | 1 | 00:00:01 | | * 26 | TABLE ACCESS BY INDEX ROWID | EXPBATCH | 1 | 14 | 1 | 00:00:01 | | * 27 | INDEX RANGE SCAN | IDX_TEXTPARAMETER_VALUE | 1 | 30 | 1 | 00:00:01 | | * 28 | TABLE ACCESS BY INDEX ROWID | FUNDCODECHANGE | 1 | 34 | 1 | 00:00:01 | | * 29 | INDEX UNIQUE SCAN | PK_FUNDCODECHANGE | 1 | | 1 | 00:00:01 | | 30 | VIEW | | 1 | 4018 | 913 | 00:00:11 | | 31 | SORT UNIQUE | | 1 | 69 | 913 | 00:00:11 | | * 32 | FILTER | | | | | | | 33 | NESTED LOOPS | | 1 | 69 | 912 | 00:00:11 | | 34 | NESTED LOOPS | | 1 | 69 | 912 | 00:00:11 | | 35 | NESTED LOOPS | | 1 | 58 | 911 | 00:00:11 | | 36 | NESTED LOOPS | | 1 | 49 | 910 | 00:00:11 | | 37 | NESTED LOOPS | | 1500 | 46500 | 309 | 00:00:04 | | 38 | VIEW | | 3 | 9 | 6 | 00:00:01 | | 39 | UNION-ALL | | | | | | | 40 | FAST DUAL | | 1 | | 2 | 00:00:01 | | 41 | FAST DUAL | | 1 | | 2 | 00:00:01 | | 42 | FAST DUAL | | 1 | | 2 | 00:00:01 | | * 43 | TABLE ACCESS FULL | FUNDINFO | 500 | 14000 | 101 | 00:00:02 | | * 44 | TABLE ACCESS BY INDEX ROWID | QUALIFY | 1 | 18 | 1 | 00:00:01 | | * 45 | INDEX RANGE SCAN | UIDX_QUALIFY | 1 | | 1 | 00:00:01 | | * 46 | INDEX RANGE SCAN | PK_EXPBATCH | 1 | 9 | 1 | 00:00:01 | | * 47 | INDEX RANGE SCAN | PK_EXPBATCH | 1 | | 1 | 00:00:01 | | * 48 | TABLE ACCESS BY INDEX ROWID | EXPBATCH | 1 | 11 | 1 | 00:00:01 | | * 49 | INDEX RANGE SCAN | IDX_TEXTPARAMETER_VALUE | 1 | 30 | 1 | 00:00:01 | | 50 | VIEW | | 7073 | 190971 | 274 | 00:00:04 | | 51 | UNION-ALL | | | | | | | 52 | SORT GROUP BY | | 1 | 16 | 20 | 00:00:01 | | 53 | TABLE ACCESS BY INDEX ROWID | ARLIMIT | 1 | 16 | 20 | 00:00:01 | | * 54 | INDEX FULL SCAN | UIDX_ARLIMIT | 1 | | 20 | 00:00:01 | | 55 | SORT GROUP BY | | 7072 | 275808 | 254 | 00:00:04 | | * 56 | HASH JOIN RIGHT ANTI | | 2970075 | 115832925 | 175 | 00:00:03 | | * 57 | INDEX FULL SCAN | UIDX_ARLIMIT | 30000 | 420000 | 20 | 00:00:01 | | * 58 | HASH JOIN | | 2970075 | 74251875 | 146 | 00:00:02 | | * 59 | INDEX FULL SCAN | PK_EXPBATCH | 99 | 891 | 1 | 00:00:01 | | * 60 | TABLE ACCESS FULL | ARLIMIT | 30000 | 480000 | 137 | 00:00:02 | | * 61 | TABLE ACCESS BY INDEX ROWID | CHANGELIMIT | 1 | 28 | 1 | 00:00:01 | | * 62 | INDEX RANGE SCAN | IDX_CHANGELIMIT_FUNDCODE | 90 | | 1 | 00:00:01 | | * 63 | TABLE ACCESS BY INDEX ROWID | FUNDCODECHANGE | 1 | 34 | 1 | 00:00:01 | | * 64 | INDEX UNIQUE SCAN | PK_FUNDCODECHANGE | 1 | | 1 | 00:00:01 | --------------------------------------------------------------------------------------------------------------------------
这里不会讲基础优化,比如什么时候用hash join,什么时候nl,lio怎么看等基础优化知识。
在某些情况下,查询有left join时,使用了parallel会导致查询一直出不来,不用parallel反而立刻可以出来。
在很多场景中,parallel提升性能很有限、甚至降低了性能以及并行执行中的buffer sort。在并行的执行计划中,我们可以看到有一步是buffer sort,而且这一步占据了挺长的时间,buffer sort实际上没有排序,只是把数据加载到内存,不扫描多次表。通常在并行执行中出现这个操作。parallel为什么会性能提升有限呢,这其实涉及到oracle内部的算法是尽可能大部分场景最优化,这就会导致对于我们知道有规律的关联消除并没有被按预期消除,比如说在我们的这个优化中,每个管理人只有1w支产品,不同管理人之间是无法转换的,但是oracle不知道,即使我们采用hash分区,且在管理人字段,设置最多100个分区(读者肯定会问,为什么要使用hash分区而不是list呢?因为这里的管理人数量是不固定的,而我们使用的是oracle 11.2,如果是12c,就不存在这个问题了,因为12c支持了自动分区,见oracle 12cR1&12cR2核心高实用性新特性),也并不是每个分区一个管理人,很多分区没有数据,所以导致分区间很不均衡,采用并行执行的时候,因为很不均衡,导致执行计划倾斜严重,可能高达40%。同时因为统计信息不一定准确,可能会导致某些时候应该px send hash的成了px send broadcast,这虽然可以通过pq_distriubte hint强制,但是由于优化器会在hint基础上进一步进行查询转换(尤其涉及到子查询套子查询的时候),也可能会导致非预期的情况。实际上,对于超过三五个表以上复杂的sql,parallel效果就不佳的,这种情况,其实应该是应用里面发起多线程并行请求,采用fork-join模式来达到并行执行的效果。这一点只有掌握具体数据库比如oracle/mysql优化原理(oracle做得到的mysql不一定做得到,反过来一般不成立)和实现的架构师做得到。最后,笔者会讲到,本例是应该在设计上进行优化而不是单独的oracle和sql优化的。
and not exists (select /*+ qb_name(inn)*/* from inner_table inn where (inn.status = outer_table.status or inn.status = ‘1‘) and xxx)
对于这种情况,anti join就失效了,虽然可以让主查询使用nl加上/*+ push_subq(@inn) */来pushdown查询条件,以避免主查询使用hash join产生巨量中间结果,但是nl对于大表性能极为低下。此时可以考虑将not exists拆分为如下:
and not exists (select * from inner_table inn where (inn.status = ‘1‘) and xxx)
and not exists (select * from inner_table inn where (inn.status = outer_table.status or inn.status = ‘1‘) and xxx)
此时可能就可以做到第一个not exists采用hash anti过滤掉大部分最终结果不需要的数据集。
and exists (outer_table.status = ‘1‘ or select 1 from another_table x where x.type = outer_table.type and ... )
对于这类查询,可以考虑使用union实现,但是有些时候,union并不见得合适,比如说查询上使用了分析函数就不合适了。此时就需要考虑其他方法(因为在我们的另一个sql中,结果集还不算大,所以也就没有进一步优化)。
/*+ NO_USE_HASH_AGGREGATION */提示。
%util 90%以上就说明当前磁盘太忙,要么布局不合理,要么太慢了。
oracle 11g亿级复杂SQL优化一例(数量级性能提升)
标签:翻译 ssi reg 导致 时间比较 load char size opera
原文地址:https://www.cnblogs.com/zhjh256/p/9417329.html