ORA-04031:unable to allocate 4120 bytes of shared memory ‘obj stat mem’故障解决

时间：2016-12-22 15:15:24 阅读：214 评论：0 收藏：0 [点我收藏+]

标签：ora-04031 obj stat memo version count

客户报表库，HP-ux B11.31 IA64 FOR Oracle 10.2.0.5.0 RAC系统短信过滤alert日志告警

ORA-00604: error occurred at recursive SQL level 1

ORA-04031:unable to allocate 4120 bytes of shared memory ("shared pool","select f.file#, f.block#, f....","Typecheck","kgghteInit")

询问客户SGA、share pool、主机资源等相关情况

SQL> show parameter sga

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

lock_sga boolean TRUE

pre_page_sga boolean FALSE

sga_max_size big integer 60G

sga_target big integer 0

SQL> show parameter pool

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

buffer_pool_keep string

buffer_pool_recycle string

global_context_pool_size string

java_pool_size big integer 512M

large_pool_size big integer 512M

olap_page_pool_size big integer 0

shared_pool_reserved_size big integer 644245094

shared_pool_size big integer 12G

streams_pool_size big integer 416M

SGA总大小60G，share pool总大小12G。有了这个直观的感受，随即将4031收集相关信息的脚本4031_OK-ForAll.sql发给客户，收集现在内存使用情况，脚本内容如下：

/**********************************************************************

* File: 4031.sql

* Date: 2012/01/1

* Modifications:

* 2012/02/12 Changed v1

*********************************************************************/

spool spinfo.txt

SET PAGESIZE 1024

SET LINESIZE 2000

set echo off;

set feedback off;

set heading on;

set trimout on;

set trimspool on;

COL BYTES FORMAT 999999999999999

COL CURRENT_SIZE FORMAT 999999999999999

/* Script Run TimeStamp */

set serveroutput on;

exec dbms_output.put_line(‘Script Run TimeStamp‘);

select to_char(sysdate, ‘dd-MON-yyyy hh24:mi:ss‘) "Script Run TimeStamp" from dual;

set serveroutput on;

exec dbms_output.put_line(‘Instance Startup Time‘);

/*Instance Startup time */

select to_char(startup_time, ‘dd-MON-yyyy hh24:mi:ss‘) "Instance Startup Time" from v$instance;

/* shared pool related hidden parameter */

set serveroutput on;

exec dbms_output.put_line(‘shared pool related hidden parameter ‘);

col name format a40

col value format a80;

select nam.ksppinm NAME,val.KSPPSTVL VALUE from x$ksppi nam,x$ksppsv val where nam.indx = val.indx and nam.ksppinm like ‘%shared%‘ order by 1;

/* SUB Pool Number */

set serveroutput on;

exec dbms_output.put_line(‘SUB Pool Number ‘);

col ‘Parameter‘ format a40

col ‘Session Value‘ format a40;

col ‘Instance Value‘ format a40;

select a.ksppinm "Parameter",

b.ksppstvl "Session Value",

c.ksppstvl "Instance Value"

from sys.x$ksppi a, sys.x$ksppcv b, sys.x$ksppsv c

where a.indx = b.indx and a.indx = c.indx

and a.ksppinm like ‘%_kghdsidx_count%‘;

/* Each Subpool Size */

set serveroutput on;

exec dbms_output.put_line(‘Each Subpool Size‘);

select ksmchidx poolnumer , sum(ksmchsiz) poolsize

from x$ksmsp

group by ksmchidx ;

/* Researved Shared Pool 4031 information */

set serveroutput on;

exec dbms_output.put_line(‘Researved Shared Pool 4031 information‘);

select REQUEST_FAILURES, LAST_FAILURE_SIZE from V$SHARED_POOL_RESERVED;

/* Reaserved Shared Pool Reserved 4031 information */

set serveroutput on;

exec dbms_output.put_line(‘Reaserved Shared Pool 4031 information‘);

select REQUESTS, REQUEST_MISSES, free_space, avg_free_size, free_count, max_free_size from V$SHARED_POOL_RESERVED;

/* Current SGA Buffer & Pool sizes */

set serveroutput on;

exec dbms_output.put_line(‘Current SGA Buffer Pool sizes‘);

select component, current_size from v$sga_dynamic_components;

/* Shared Pool Memory Allocations by Size */

set serveroutput on;

exec dbms_output.put_line(‘Shared Pool Memory Allocations by Size‘);

select name, bytes from v$sgastat

where pool = ‘shared pool‘ and (bytes > 999999 or name = ‘free memory‘)

order by bytes desc;

set serveroutput on;

exec dbms_output.put_line(‘show component of shared pool which is bigger than 10MB‘);

select name, round((bytes/1024/1024),0) "more than 10" from v$sgastat where pool=‘shared pool‘ and bytes > 10000000 order by bytes desc;

select sum(bytes) "SHARED POOL TOTAL SIZE" from v$sgastat where pool=‘shared pool‘;

/* Total Free of Shared Pool */

set serveroutput on;

exec dbms_output.put_line(‘Total Free(not Free) of Shared Pool ‘);

COL ‘Total Shared Pool Usage‘ FORMAT 999999999999999

select sum(bytes)/1024/1024 "Free MB in Shared Pool" from v$sgastat where pool = ‘shared pool‘ and name = ‘free memory‘;

select sum(bytes) "Not Free MB Shared Pool" from v$sgastat where pool = ‘shared pool‘ and name != ‘free memory‘;

/* current KGLH* usage */

set serveroutput on;

exec dbms_output.put_line(‘current KGLH* usage‘);

select name, bytes from v$sgastat where pool = ‘shared pool‘ and name in (‘KGLHD‘,‘KGHL0‘);

/* Hisotry KGLH* usage */

set serveroutput on;

exec dbms_output.put_line(‘Hisotry KGLH* usage‘);

select bytes/1024/1024 , s.snap_id, begin_interval_time START_TIME

from dba_hist_sgastat g, dba_hist_snapshot s

where name=‘KGLHD‘

and pool=‘shared pool‘

and trunc(begin_interval_time) >= ‘30-DEC-2011‘

and s.snap_id = g.snap_id

order by 2;

set serveroutput on;

exec dbms_output.put_line(‘Hisotry KGLH0* usage‘);

select bytes/1024/1024 , s.snap_id, begin_interval_time START_TIME

from dba_hist_sgastat g, dba_hist_snapshot s

where name=‘KGLH0‘

and pool=‘shared pool‘

and trunc(begin_interval_time) >= ‘30-DEC-2011‘

and s.snap_id = g.snap_id

order by 2;

/* History of Shared pool allocations in a speciifed Day*/

set serveroutput on;

exec dbms_output.put_line(‘history of Shared pool allocations in a speciifed Day‘);

col name format a30

select n,

max(decode(to_char(begin_interval_time, ‘hh24‘), 1,bytes, null)) "1",

max(decode(to_char(begin_interval_time, ‘hh24‘), 2,bytes, null)) "2",

max(decode(to_char(begin_interval_time, ‘hh24‘), 3,bytes, null)) "3",

max(decode(to_char(begin_interval_time, ‘hh24‘), 4,bytes, null)) "4",

max(decode(to_char(begin_interval_time, ‘hh24‘), 5,bytes, null)) "5",

max(decode(to_char(begin_interval_time, ‘hh24‘), 6,bytes, null)) "6",

max(decode(to_char(begin_interval_time, ‘hh24‘), 7,bytes, null)) "7",

max(decode(to_char(begin_interval_time, ‘hh24‘), 8,bytes, null)) "8",

max(decode(to_char(begin_interval_time, ‘hh24‘), 9,bytes, null)) "9",

max(decode(to_char(begin_interval_time, ‘hh24‘), 10,bytes, null)) "10",

max(decode(to_char(begin_interval_time, ‘hh24‘), 11,bytes, null)) "11",

max(decode(to_char(begin_interval_time, ‘hh24‘), 12,bytes, null)) "12",

max(decode(to_char(begin_interval_time, ‘hh24‘), 13,bytes, null)) "13",

max(decode(to_char(begin_interval_time, ‘hh24‘), 14,bytes, null)) "14",

max(decode(to_char(begin_interval_time, ‘hh24‘), 15,bytes, null)) "15",

max(decode(to_char(begin_interval_time, ‘hh24‘), 16,bytes, null)) "16",

max(decode(to_char(begin_interval_time, ‘hh24‘), 17,bytes, null)) "17",

max(decode(to_char(begin_interval_time, ‘hh24‘), 18,bytes, null)) "18",

max(decode(to_char(begin_interval_time, ‘hh24‘), 19,bytes, null)) "19",

max(decode(to_char(begin_interval_time, ‘hh24‘), 20,bytes, null)) "20",

max(decode(to_char(begin_interval_time, ‘hh24‘), 21,bytes, null)) "21",

max(decode(to_char(begin_interval_time, ‘hh24‘), 22,bytes, null)) "22",

max(decode(to_char(begin_interval_time, ‘hh24‘), 23,bytes, null)) "23",

max(decode(to_char(begin_interval_time, ‘hh24‘), 24,bytes, null)) "24"

from (select ‘"‘||name||‘"‘ n, begin_interval_time, bytes from dba_hist_sgastat a, dba_hist_snapshot b

where pool=‘shared pool‘ and a.snap_id=b.snap_id

and to_char(begin_interval_time,‘hh24:mi‘) between ‘01:00‘ and ‘24:00‘

and to_char(begin_interval_time,‘dd-mon‘) = to_char(sysdate-1, ‘dd-mon‘))

group by n;

/* Each Subpool sumary usage for free memory , may slow ,it depends on custoemr database workload */

set serveroutput on;

exec dbms_output.put_line(‘Each Subpool sumary usage for free memory‘);

col subpool format a20

col name format a40

SELECT

subpool

, name

, SUM(bytes)

, ROUND(SUM(bytes)/1048576,2) MB

FROM (

SELECT

‘shared pool (‘||DECODE(TO_CHAR(ksmdsidx),‘0‘,‘0 - Unused‘,ksmdsidx)||‘):‘ subpool

, ksmssnam name

, ksmsslen bytes

FROM

x$ksmss

WHERE

ksmsslen > 0

AND LOWER(ksmssnam) LIKE LOWER(‘%free memory%‘)

)

GROUP BY

subpool

, name

ORDER BY

subpool ASC

, SUM(bytes) DESC ;

/* Memory fragment and chunk allocation like 0-1K,1-2K, may slow ,it depends on custoemr database workload */

set serveroutput on;

exec dbms_output.put_line(‘Memory fragment and chunk allocation like 0-1K,1-2K‘);

col SubPool format 999

col mb format 999,999

col name heading "Name"

SELECT ksmchidx "SubPool",

‘sga heap(‘ || ksmchidx || ‘,0)‘ sga_heap,

ksmchcom chunkcomment,

DECODE(ROUND(ksmchsiz / 1000),

‘0-1K‘,

‘1-2K‘,

‘2-3K‘,

‘3-4K‘,

‘4-5K‘,

‘5-6k‘,

‘6-7k‘,

‘7-8k‘,

‘8-9k‘,

‘9-10k‘,

‘> 10K‘

) "size",

COUNT(*),

ksmchcls status,

SUM(ksmchsiz) BYTES

FROM x$ksmsp

WHERE ksmchcom = ‘free memory‘

GROUP BY ksmchidx,

ksmchcls,

‘sga heap(‘ || ksmchidx || ‘,0)‘,

ksmchcom,

ksmchcls,

DECODE(ROUND(ksmchsiz / 1000),

‘0-1K‘,

‘1-2K‘,

‘2-3K‘,

‘3-4K‘,

‘4-5K‘,

‘5-6k‘,

‘6-7k‘,

‘7-8k‘,

‘8-9k‘,

‘9-10k‘,

‘> 10K‘);

select to_char(sysdate,‘YYYY-MM-DD HH24:MI:SS‘) "Script END TimeStamp" from dual;

spool off;

执行后结果为：

NAME VALUE

---------------------------------------- --------------------------------------------------------------------------------

__shared_pool_size 12884901888

_all_shared_dblinks

_dm_max_shared_pool_pct 1

_enable_shared_pool_durations FALSE

_io_shared_pool_size 4194304

_shared_pool_max_size 0

_shared_pool_minsize_on FALSE

_shared_pool_reserved_min_alloc 4400

_shared_pool_reserved_pct 5

_shared_server_spare_param1

_shared_server_spare_param2

_shared_server_spare_param3

_skgxp_shared_port 0

hi_shared_memory_address 0

max_shared_servers

shared_memory_address 0

shared_pool_reserved_size 644245094

shared_pool_size 12884901888

shared_server_sessions

shared_servers 0

ERROR:

ORA-04031: unable to allocate 48 bytes of shared memory ("shared pool","BEGIN DBMS_OUTPUT.ENABLE(NUL...","parameters","kglpda")

REQUEST_FAILURES LAST_FAILURE_SIZE

---------------- -----------------

5679 4200

shared pool中内存大于_SHARED_POOL_RESERVED_MIN_ALLOC 将放入shared pool保留池,保留池维护一个单独的freelist,lru，并且不会在lru列表存recreatable类型chunks，普通shared pool的释放与shared pool保留池无关。

REQUEST_FAILURES>0且LAST_FAILURE_SIZE(最后请求内存大小)<_SHARED_POOL_RESERVED_MIN_ALLOC,表示在 shared pool中缺少连续内存,或者库里面有大量的硬解析造成的。一般是绑定变量问题。也就是version count过高。我们顺着这个思路往下继续查看；

也就是说两个sql（7ng34ruy5awxq、2g9nykfyk0a95）有重大影响；

SQL> select sql_id,child_number,BIND_MISMATCH from v$sql_shared_cursor where sql_id=‘2g9nykfyk0a95‘ and BIND_MISMATCH=‘Y‘ and rownum<10;

SQL_ID        CHILD_NUMBER B
------------- ------------ -
2g9nykfyk0a95            4 Y
2g9nykfyk0a95            5 Y
2g9nykfyk0a95           30 Y
2g9nykfyk0a95            0 Y
2g9nykfyk0a95           54 Y
2g9nykfyk0a95           23 Y
2g9nykfyk0a95           27 Y
2g9nykfyk0a95           35 Y
2g9nykfyk0a95           46 Y

9 rows selected.

SQL> select count(*) from v$sql_shared_cursor where sql_id=‘2g9nykfyk0a95‘ and BIND_MISMATCH=‘Y‘ ;

  COUNT(*)
----------
        23

SQL>select position,LAST_CAPTURED,datatype_string from v$sql_bind_capture where sql_id=‘2g9nykfyk0a95‘ and rownum<50

POSITION LAST_CAPTURE DATATYPE_STRING

---------- ------------ ------------------------------

1 VARCHAR2(128)

2 VARCHAR2(128)

3 VARCHAR2(128)

4 VARCHAR2(32)

5 VARCHAR2(128)

6 VARCHAR2(32)

7 TIMESTAMP

8 TIMESTAMP

9 VARCHAR2(32)

10 VARCHAR2(32)

11 VARCHAR2(32)

12 VARCHAR2(128)

13 VARCHAR2(128)

14 VARCHAR2(32)

15 VARCHAR2(32)

16 VARCHAR2(32)

17 VARCHAR2(32)

18 VARCHAR2(32)

19 VARCHAR2(32)

20 VARCHAR2(32)

21 VARCHAR2(32)

22 VARCHAR2(32)

23 VARCHAR2(32)

24 VARCHAR2(32)

25 VARCHAR2(32)

26 VARCHAR2(32)

27 VARCHAR2(32)

28 VARCHAR2(32)

29 VARCHAR2(32)

30 VARCHAR2(32)

31 NUMBER

1 VARCHAR2(128)

2 VARCHAR2(128)

3 VARCHAR2(128)

4 VARCHAR2(32)

5 VARCHAR2(128)

6 VARCHAR2(32)

7 TIMESTAMP

8 TIMESTAMP

9 VARCHAR2(32)

10 VARCHAR2(32)

11 VARCHAR2(32)

12 VARCHAR2(128)

13 VARCHAR2(128)

14 VARCHAR2(32)

15 VARCHAR2(32)

16 VARCHAR2(32)

17 VARCHAR2(2000)

18 VARCHAR2(32)

49 rows selected.

禁用相关应用后，接着往下分析，看看还有没有别的地方引起的该4031问题；

NAME BYTES

---------------------------------------- ----------------

obj stat memo 6235601184

free memory 1919164576

object level 1148667072

gcs resources 980982312

gcs shadows 426063424

sql area 401151376

db_block_hash_buckets 188743680

kglsim object batch 179096400

kglsim heap 173694528

CCursor 139486384

Cursor Stats 131995544

ges resource 110674136

library cache 101986224

PCursor 88613936

ges enqueues 87640800

sql area:PLSQL 78693424

ASH buffers 52428800

trace buffer 40927232

KQR L PO 36581592

Checkpoint queue 32776192

state objects 30602616

event statistics per sess 26095360

FileOpenBlock 15936504

ges big msg buffers 15936168

sessions 15163528

KCL name table 12582912

kgllk hash table 10231808

KGLS heap 10113784

simulator hash buckets 8404992

dbwriter coalesce buffer 8392704

gcs res hash bucket 8388608

ges reserved msg buffers 8240008

Heap0: KGL 7905976

object queue 7894320

row cache 7511248

transaction 6885376

KQR L SO 5958168

enqueue 5886080

parameter table block 5331280

procs: ksunfy 5120000

FileIdentificatonBlock 4571216

call 4535640

KCB Table Scan Buffer 4194816

kglsim hash table bkts 4194304

KSFD SGA I/O b 4190328

buffer handles 3600008

DML lock 3541016

KQR M SO 3300680

gcs affinity 3241728

ges process array 3181272

ges resource hash table 2883584

PL/SQL DIANA 2771128

trace buf hdr xtend 2736864

PL/SQL MPCODE 2626400

ges regular msg buffers 2622008

KTI SGA freea 2498560

KGSK scheduler 2358624

ktlbk state objects 2108880

object queue hash buckets 2101248

enqueue resources 1953128

replication session stats 1939520

SGA - SWRF Metric CHBs 1857960

db_files 1777912

kks stbkt 1572864

KEWS sesstat values 1432600

Wait History 1322400

pso tbs: ksunfy 1300000

Sort Segment 1272848

osp allocation 1195984

mvobj part des 1110240

KSXR receive buffers 1036000

SUBPOOL NAME SUM(BYTES) MB

-------------------- ---------------------------------------- ---------- ----------

shared pool (1): free memory 259671360 247.64

shared pool (2): free memory 252015608 240.34

shared pool (3): free memory 277114712 264.28

shared pool (4): free memory 275504440 262.74

shared pool (5): free memory 281692368 268.64

shared pool (6): free memory 268028160 255.61

shared pool (7): free memory 304926192 290.8

期间我们看见obj stat memo排名第一，消耗内存资源5.9G。七个子池，大概share pool剩余空间1.83GB

SQL> SELECT * FROM

2 (SELECT NAME, BYTES/(1024*1024) MB

3 FROM V$SGASTAT

4 WHERE POOL = ‘shared pool‘

5 ORDER BY BYTES DESC)

6 WHERE ROWNUM <= 10;

NAME MB

-------------------------- ----------

obj stat memo 5955.40194

free memory 2503.24937

object level 1097.0509

gcs resources 935.537636

gcs shadows 406.325745

db_block_hash_buckets 180

kglsim object batch 170.799637

kglsim heap 165.64801

Cursor Stats 125.88076

ges resource 104.992348

SQL>

SQL> select * from v$sgastat where name = ‘obj stat memo‘;

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 6244703856

SQL>

SQL> jselect * from v$sgastat where name = ‘obj stat memo‘;

SP2-0734: unknown command beginning "jselect * ..." - rest of line ignored.

SQL> select * from v$sgastat where name = ‘obj stat memo‘;

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 6245586216

SQL> /

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 6245930952

且obj stat memo一直在增长，无法释放。尝试使用flush share_pool但是obj无视

SQL> alter system flush shared_pool;

SQL> select * from v$sgastat where name = ‘obj stat memo‘;

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 6343766208

无解只好求助support进行搜索，找到一篇

ORA-04031 With Leak in "OBJ STAT MEMO" Allocations Seen in V$SGASTAT on 10.2.0.5 (文档 ID 1350050.1)描述一致；

CAUSE

On 10.2.0.5 an architectural change was made to switch off the publishing of "obj stat del channel" messages by default. This can lead to excessive growth of "obj stat memo" memory allocation.

SOLUTION

On 10.2.0.5, and only for 10.2.0.5, we have introduced the hidden parameter :

_disable_objstat_del_broadcast

If you are seeing ORA-04031 related to the symptoms reported then this parameter can be set to FALSE and by doing so we will no longer see the growth of "obj stat memo" that potentially leads to ORA-04031.

This parameter has been instructed by development to be used as the solution to ORA-04031 with the symptoms reported. There is no patch fix and no patch fix will be made. The hidden parameter will not cause any problems to the database and it must not be accidentally left within the init/spfile when/if the database is upgraded as startup would fail with :

将原隐患参数设置false；

ALTER SYSTEM SET "_disable_objstat_del_broadcast"=FALSE SCOPE=BOTH;

_disable_objstat_del_broadcast为false并不会对数据库造成影响，可以修改，但是请注意假如之后数据库进行升级，需要在参数文件中取消该参数，否则数据库无法正常启动。

因为一直没有释放，将数据库实例重启后，监控该资源情况，得到释放；

SQL>select * from v$sgastat where name = ‘obj stat memo‘;

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 102600

SQL> /

POOL NAME BYTES

------------ -------------------------- ----------

shared pool obj stat memo 143640