码迷,mamicode.com
首页 > 系统相关 > 详细

Troubleshooting 'library cache: mutex X' waits. (文档 ID 1357946.1)

时间:2016-04-29 16:44:47      阅读:482      评论:0      收藏:0      [点我收藏+]

标签:

In this Document

  Purpose
  Troubleshooting Steps
  What is a ‘library cache: mutex X‘ wait?
  What causes ‘library cache: mutex X‘ wait?
  How to diagnose the cause.
  How to Examine the Diagnostics.
  Potential Solutions
  References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.1.0.6 and later
Information in this document applies to any platform.
***Checked for relevance on 10-June-2013*** 

PURPOSE

The purpose of the note is to help customers troubleshoot contention for wait event ‘library cache: mutex X‘.

TROUBLESHOOTING STEPS

What is a ‘library cache: mutex X‘ wait?

Library cache: mutex X is similar to a library cache wait in earlier versions.  ‘Library cache: mutex X‘ may be caused by many issues.  So it is important to find the correct cause; so that the right solution can be implemented. 

What causes ‘library cache: mutex X‘ wait?

  • Frequent Hard Parses - If the frequency of Hard Parsing is extremely high, then contention can occur on this pin.
  • High Version Counts - When Version counts become excessive, a long chain of versions needs to be examined and this can lead to contention on this event
  • Invalidations and reloads
  • Known Bugs

How to diagnose the cause.

1. Check to see if anything has changed:   
     a. increased load
     b. any change in the application, os, or middle tier
     c. any os change

2. Is there a trend to library cache: mutex X:
    a. is there a certain time of the day when this wait is seen?
    b. does something trigger this wait?

3. During the time of the issue, run AWR and ADDM.  Also obtain the baseline to compare the load, parameter changes, and any other differences. 
To gather this it is suggested to run AWR and ADDM for half an hour to an hour interval as follows:

SQL>@$ORACLE_HOME/rdbms/admin/awrrpt.sql
SQL>@$ORACLE_HOME/rdbms/admin/addmrpt.sql


4. Sometimes system state dump is necessary to match known issues. For example, if there is no obvious candidate SQL in AWR, capturing holder or waiter processes in systemstate allows you to focus in on potential problems. Run system state when processes appear hung on ‘library cache: mutex X‘:

(a) Non-Rac


sqlplus "/ as sysdba"

oradebug setmypid
oradebug unlimit
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
wait 90 seconds
oradebug dump systemstate 266
quit

(b) RAC

$ sqlplus ‘/ as sysdba‘
oradebug setmypid
oradebug unlimit
oradebug setinst all
oradebug -g all hanganalyze 4
oradebug -g all dump systemstate 266
quit

5.  Errorstacks: Another way to obtain process information is with errorstack. Assuming you can identify a blocker, taking errorstacks will provide much the same information as systemstates but with a much reduced disk footprint for trace. Once the ospid of the blocker has been found, an errorstack can be generated:

$ sqlplus
SQL> oradebug setospid <p.spid from above>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
<< wait 1min>>
oradebug dump errorstack 3
exit

In particular, the stack from the resultant trace can be used to match known issues.
The system state and errorstacks are not easily readable; so a Service Request may need to be opened to read the files.

6. Sometimes it is not feasible to run system state dump, as it may be resource intensive.  So the following sql can also be ran in interval:

select s.sid, t.sql_text
from v$session s, v$sql t
where s.event like ‘%mutex%‘
and t.sql_id = s.sql_id

Check to see what sessions are waiting on. 

7. In 11g RAC, there is another less resource intensive tool that can be used when compared with taking system state dumps:

Document 459694.1 Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes

How to Examine the Diagnostics.

1. Normally, the top wait event will be the library cache: mutex X in the problematic AWR:

技术分享

2. First look for high parsing and high version counts from AWR.
Click on *SQL Statistics under Main Report of AWR:

技术分享

Then, under SQL Statistics click on ‘SQL ordered by Parse Calls‘ and  ‘SQL ordered by Version Count‘ to view that information:

技术分享

Check for high parse calls.

技术分享

Check to see if there is high parse calls to execute.  Ideally, there should be less parse to executions.  Notice there is as many parses as executes, indicating cursors are not used well in the application.  Once the cursor is opened and parsed, it should be kept open.  Check with the application developer on how to keep the cursor opened to re execute the sqls.  

Next, check the version count of the sql:

技术分享


From this list, investigate the SQLs with the high version count. What are the reasons that these statements are not shared? Can this be addressed? 
Check V$SQL_SHARED_CURSOR to see the potential reason for the high version count using:

Document 438755.1Formated V$SQL_SHARED_CURSOR Report by SQLID or Hash Value
Document 296377.1Troubleshooting: High Version Count Issues

Potential Solutions

1. Check for high hard  parsing, as this can cause can reloads in the sql area.  Check the hard parse under the load profile:

技术分享

This load shows 26.3 hard parses per second, indicating high hard parsing.  Check to see if the application is sharing the sql.  If application is mostly using literals, see if the sqls can be shared by using bind variables. Furthermore, review the ‘Over Parsing‘ section of the following note:

Document 33089.1TROUBLESHOOTING: Possible Causes of Poor SQL Performance

Also check for high reloads in the sql area:

技术分享

If there is a high number of reloads, then look to see if cursors are being shared efficiently. If they are then check to see if the shared pool or sga_target is large enough. Remember that inefficient sharing means that the library cache will fill up with non-reuseable cursors which may then cause reuseable ones to be flushed out. These will cause reloads when they are re-executed.
If sharing is efficient and the shared pool is too small, then shareable SQL statements will age out and hard parses will be higher. In most cases however this is not the case and the problem is inefficient sharing.
The following note is helpful in tuning the shared pool:

Document 62143.1 Understanding and Tuning the Shared Pool

2. Check for invalidations under Library Cache Activity.  If the invalidation has high number, then check ddl‘s performed during the time such as truncate, drop, grants, dbms_stats, etc.

3. Check the following note for relevant bugs under ‘Known Bugs‘ and relevant version:

Document 727400.1WAITEVENT: "library cache: mutex X"

4. For 11g, make sure cursor_sharing is not similar, as it has been deprecated.  This may also cause mutex waits:

Document 1169017.1ANNOUNCEMENT: Deprecating the cursor_sharing = ‘SIMILAR’ setting

5. If the database has been migrated from 10g to 11g and mutex performance issue surfaces, please consider the 11.2.0.2.2 psu + fix for Unpublished Bug 12431716. Many mutex fixes are already included in this patch:

Document 1291879.1 Oracle Database Patch Set Update 11.2.0.2.2 Known Issues

REFERENCES

NOTE:438755.1 - High SQL Version Counts - Script to determine reason(s)
NOTE:1169017.1 - ANNOUNCEMENT: Deprecating the cursor_sharing = ‘SIMILAR‘ setting
NOTE:1291879.1 - Oracle Database Patch Set Update 11.2.0.2.2 Known Issues
NOTE:296377.1 - Troubleshooting: High Version Count Issues
NOTE:33089.1 - * TROUBLESHOOTING: Possible Causes of Poor SQL Performance
NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
NOTE:62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
NOTE:727400.1 - WAITEVENT: "library cache: mutex X"

技术分享
 
 

附件

   
 

Troubleshooting 'library cache: mutex X' waits. (文档 ID 1357946.1)

标签:

原文地址:http://blog.csdn.net/xxjj2003/article/details/51258392

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!