Introduction
~~~~~~~~~~~~
This short article aims to explain how to get a stack trace from a
core dump produced by any of the Oracle products on Unix platforms.
By following the steps below you can provide Oracle Support with vital
information to help identify the cause of a problem.
Please note that it is important to include information about the
tool being used, any code involved, the operation being performed,
environment etc.. in addition to the details below.
What is a ‘core dump‘ ?
~~~~~~~~~~~~~~~~~~~~~~~
A core dump is an image copy of a processes state at the instant
it ‘aborted‘. It is produced in the form of a file called ‘core‘
usually located in the current directory.
What causes a core dump ?
~~~~~~~~~~~~~~~~~~~~~~~~~
There are many situations which can cause a core dump to be produced,
but it is usually because the process has attempted to do something
which the operating system does not like. The most common causes
of this are:
The program tried to access memory outside its allowed range.
The program tried to obtain a resource which was either
exhausted or unavailable.
An attempt was made to execute illegal instructions.
An attempt was made to read unaligned data
In Unix systems the offending process is sent one of a number of
signals which force a core dump to be produced. It is also possible
for a user to produce a core dump by sending one of these signals
to a process manually.
What should I do if I get a core dump ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
As with any problem you should first note down the FULL version
numbers of the product, the RDBMS, PL/SQL (if used) and any
related products.
You should also note the EXACT command you were running when
this occurred. Eg: If it was a SQL*Forms problem and you were
using ‘mrunform30‘, write this down. This command will be referred
to as ‘program‘ below.
Now follow the instructions below in order:
1) Check if you have a ‘core‘ file, it should be in the directory where
the command was issued, or in CORE_DUMP_DEST/core_NNNNNN
if it is the ‘oracle‘ executable. "oracle" can also produce
core files in $ORACLE_HOME/dbs/core_NNNNN or $ORACLE_HOME/dbs .
2) Log in as ORACLE and cd to the directory containing the core file.
Then issue:
file core
This should identify the "program" name to use in the next step,
e.g.: oracle
3) Log in as ORACLE and change in to the $ORACLE_HOME/bin
directory. Enter the command:
file program
and write the result down letter for letter. If the word ‘dynamic‘
or ‘dynamically linked‘ appears in the output of this command
then please make a note of this as there are a few platforms on
which Oracle does NOT support dynamic linking and this may be
the cause of your problem.
4) Now enter:
chmod +r program
to add read permission to the program.
5) Log out , then log in as the user who encountered the error.
The next step will vary slightly depending on which version of
Unix you are using. One of the following commands should exist
on your machine - try each in order until you find one that exists.
In some cases you may be asked for stacks from all threads so
use the "thread" version of the command if it exists. If the core
file contains multiple threads see Note:118252.1.
An alternative to the commands below is to use the stackx.sh
script from Note:362791.1 . That script will try to find a
suitable debugger and extract the stack tracee for you .
Common Debuggers and commands to show a symbolic stack trace:
Command NB Exit command Stack Trace command
------- -- ------------ -------------------
dbx quit where
xdb (HPUX 10) quit t
gdb (HPUX 11) q bt
dde (HPUX 11) q bt
sdb q t
adb $q (or Ctrl-D) $c
debug (PTX only) quit stack
gdb (Linux) quit bt
or thread apply all where
pstack (HPUX, Linux, Solaris)
Change to the directory where the core dump is located and enter
the commands as in the relevant example below. If you are not
sure which program produced the ‘core‘ file then on some Unix
platforms the command ‘file core‘ will tell you the executable
name that the core file is from (this does not work on ALL
Unix platforms, see note below.)
.
Example commands:
DBX: $ script /tmp/mystack
$ dbx $ORACLE_HOME/bin/<program> core
(dbx) where
... << Stack should appear here
(dbx) quit
$ exit
XDB: $ script /tmp/mystack
$ xdb $ORACLE_HOME/bin/<program> core
(xdb) t
... << Stack should appear here
(xdb) quit
$ exit
SDB: $ script /tmp/mystack
$ sdb $ORACLE_HOME/bin/<program> core
(sdb) t
... << Stack should appear here
(sdb) q
$ exit
(NOTE: In the ‘adb‘ commands below literally type the $c & $q)
ADB: $ script /tmp/mystack
$ adb $ORACLE_HOME/bin/<program> core
$c << NB: adb has no prompt so just enter $c
...
$q
$ exit
DEBUG: $ script /tmp/mystack
$ debug -c core $ORACLE_HOME/bin/<program>
debug> stack
... << Stack should appear here
debug> quit
$ exit
GDB: $ script /tmp/mystack
$ gdb $ORACLE_HOME/bin/<program> core
(gdb) bt
... << Stack should appear here
(gdb) thread apply all where
... << Stacks for all threads here
(gdb) quit
$ exit
DDE: $ script /tmp/mystack
$ dde -ui line core $ORACLE_HOME/bin/<program>
dde> bt
...
dde> q
$exit
PSTACK: $ script /tmp/mystack
$ pstack core
$ exit
Assuming this worked then the stack trace should be shown in the
file ‘/tmp/mystack‘. Upload this to Oracle Support.
6) If the debug command failed to give a stack trace then try using
a different debugger from the list above (if available).
If all debuggers fail then there is probably a problem with
either the permissions or the file type - see the section below
and then contact Oracle Support with all the details you have so far.
Common reasons for not getting a sensible stack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Filesize Limits:
Note that on some machines there may be a kernel parameter or
user limit which controls the maximum size of core file that
can be produced - you can usually check this by typing:
limit in the C shell
OR ulimit -a in the Bourne / Korn shells.
If this limit is too small the core file will be useless -
raise the limit and reproduce the problem.
Stripped Executable
Some program executables are stripped of symbol information.
This makes the stack trace useless. If ‘file program‘ shows
the word ‘stripped‘ or ‘nm program‘ shows no output then it
is likely that the executable is stripped of symbolic information.
In this case the problem tool must be relinked without being
stripped - on most Unix platforms this involves ensuring there is
no ‘-s‘ option on the link line. Contact Oracle Support with
details of the link line used to link the tool.
HP Unix
Some platforms like HP Unix need a special object file linking
in at link time to ensure symbols in shared objects can be
reported by the debug tool. Typically this involves relinking the
tool including /usr/lib/end.o on the link line. The location of
this special file may be different depending on your HPUX
version. ‘xdb‘ generally tells you the location of this file
if it was not linked into the executable.
If ‘file core‘ does not return the executable name:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Try using the ‘strings‘ command:
csh> setenv LANG C
or
ksh> LANG=C;export LANG ...to get rid of non-ASCII characters return by ‘strings‘
> strings -a core | more
The first part of the output may reveal the executable name.
References
NOTE:118252.1 - How to Process an Express Core File Using dbx, dbg, dde, gdb or ladebug NOTE:362791.1 - STACKX User Guide |
BULLETIN
