#定位系统性能瓶颈# strace & ltrace

比如若仅仅说明s b r k系统调用，那么就会忽略非常多应用程序使用的m a l l o c库函数。
本书除了必需要区分两者时，都将使用术语函数（ f u n c t i o n）来指代系统调用和库函数两者。

CPU硬件决定了这些（这就是为什么它被称作"保护模式"）。系统调用是这些规则的一个例外。

其原理是进程先用适当的值填充寄存器，然后调用一个特殊的指令。这个指令会跳到一个事先定义的内核中的一个位置（当然，这个位置是用户进程可读可是不可写的）。在Intel CPU中，这个由中断0x80实现。

硬件知道一旦你跳到这个位置，你就不是在限制模式下执行的用户。而是作为操作系统的内核--所以你就能够为所欲为。

进程能够跳转到的内核位置叫做sysem_call。这个过程检查系统调用号。这个号码告诉内核进程请求哪种服务。然后。它查看系统调用表(sys_call_table)找到所调用的内核函数入口地址。

接着。就调用函数，等返回后。做一些系统检查，最后返回到进程（或到其它进程，假设这个进程时间用尽）。假设你希望读这段代码，它在<内核源代码文件夹>/kernel/entry.S，Entry(system_call)的下一行。

为防止和正常的返回值混淆，系统调用并不直接返回错误码，而是将错误码放入一个名为errno的全局变量中。

假设一个系统调用失败，你能够读出errno的值来确定问题所在。
errno不同数值所代表的错误消息定义在errno.h中。你也能够通过命令"man 3 errno"来察看它们。
须要注意的是，errno的值仅仅在函数错误发生时设置。假设函数不错误发生，errno的值就无定义，并不会被置为0。另外。在处理errno前最好先把它的值存入还有一个变量，由于在错误处理过程中。即使像printf()这种函数出错时也会改变errno的值。

man strace
DESCRIPTION
       In the simplest case strace runs the specified command until it exits.  It intercepts and  records  the  system
       calls  which  are called by a process and the signals which are received by a process.  The name of each system
       call, its arguments and its return value are printed on standard error or to the file  specified  with  the 
       -o option.

       strace  is  a  useful diagnostic, instructional, and debugging tool.  System administrators, diagnosticians and
       trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily
       available  since  they  do not need to be recompiled in order to trace them.  Students, hackers and the overly-
       curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary
       programs.   And  programmers  will  find  that  since  system  calls  and signals are events that happen at the
       user/kernel interface, a close examination of this boundary is very useful for bug isolation,  sanity  checking
       and attempting to capture race conditions.

       Each  line  in the trace contains the system call name, followed by its arguments in parentheses and its return
       value.  An example from stracing the command ‘‘cat /dev/null’’ is:

       open("/dev/null", O_RDONLY) = 3

       Errors (typically a return value of -1) have the errno symbol and error string appended.

       open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)

       Signals are printed as a signal symbol and a signal string.  An excerpt from stracing and interrupting the com-
       mand ‘‘sleep 666’’ is:

       sigsuspend([] <unfinished ...>
       --- SIGINT (Interrupt) ---
       +++ killed by SIGINT +++

       If  a  system  call is being executed and meanwhile another one is being called from a different thread/process
       then strace will try to preserve the order of those events and mark the ongoing call as being unfinished.  When
       the call returns it will be marked as resumed.

       [pid 28772] select(4, [3], NULL, NULL, NULL <unfinished ...>
       [pid 28779] clock_gettime(CLOCK_REALTIME, {1130322148, 939977000}) = 0
       [pid 28772] <... select resumed> )      = 1 (in [3])

       Interruption  of a (restartable) system call by a signal delivery is processed differently as kernel terminates
       the system call and also arranges its immediate reexecution after the signal handler completes.

       read(0, 0x7ffff72cf5cf, 1)              = ? ERESTARTSYS (To be restarted)
       --- SIGALRM (Alarm clock) @ 0 (0) ---
       rt_sigreturn(0xe)                       = 0
       read(0, ""..., 1)                       = 0
      Arguments are printed in symbolic form with a passion.  This example shows  the  shell  performing  ‘‘>>xyzzy’’
       output redirection:

       open("xyzzy", O_WRONLY|O_APPEND|O_CREAT, 0666) = 3

       Here  the  three  argument form of open is decoded by breaking down the flag argument into its three bitwise-OR
       constituents and printing the mode value in octal by tradition.  Where traditional or native usage differs from
       ANSI  or  POSIX,  the  latter forms are preferred.  In some cases, strace output has proven to be more readable
       than the source.

       Structure pointers are dereferenced and the members are displayed as appropriate.  In all cases  arguments  are
       formatted  in the most C-like fashion possible.  For example, the essence of the command ‘‘ls -l /dev/null’’ is
       captured as:

       lstat("/dev/null", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0

       Notice how the ‘struct stat’ argument is dereferenced and how each member is displayed symbolically.   In  par-
       ticular,  observe how the st_mode member is carefully decoded into a bitwise-OR of symbolic and numeric values.
       Also notice in this example that the first argument to lstat is an input to the  system  call  and  the  second
       argument  is  an  output.   Since output arguments are not modified if the system call fails, arguments may not
       always be dereferenced.  For example, retrying the ‘‘ls -l’’ example with a non-existent file produces the fol-
       lowing line:

       lstat("/foo/bar", 0xb004) = -1 ENOENT (No such file or directory)

       In this case the porch light is on but nobody is home.

       Character  pointers are dereferenced and printed as C strings.  Non-printing characters in strings are normally
       represented by ordinary C escape codes.  Only the first strsize (32 by default) bytes of strings  are  printed;
       longer  strings have an ellipsis appended following the closing quote.  Here is a line from ‘‘ls -l’’ where the
       getpwuid library routine is reading the password file:

       read(3, "root::0:0:System Administrator:/"..., 1024) = 422

       While structures are annotated using curly braces, simple pointers and arrays are printed using square brackets
       with  commas  separating  elements.   Here is an example from the command ‘‘id’’ on a system with supplementary
       group ids:

       getgroups(32, [100, 0]) = 2

       On the other hand, bit-sets are also shown using square brackets but set  elements  are  separated  only  by  a
       space.  Here is the shell preparing to execute an external command:

       sigprocmask(SIG_BLOCK, [CHLD TTOU], []) = 0

       Here  the  second  argument  is a bit-set of two signals, SIGCHLD and SIGTTOU.  In some cases the bit-set is so
       full that printing out the unset elements is more valuable.  In that case, the bit-set is prefixed by  a  tilde
       like this:

       sigprocmask(SIG_UNBLOCK, ~[], NULL) = 0

       Here the second argument represents the full set of all signals.