内核调试神器SystemTap — 简单介绍与使用（一）

a linux trace/probe tool.

官网：https://sourceware.org/systemtap/

简单介绍

SystemTap是我眼下所知的最强大的内核调试工具，有些家伙甚至说它无所不能：）

(1) 发展历程

Debuted in 2005 in Red Hat Enterprise Linux 4 Update 2 as a technology preview.

After four years in development, System 1.0 was released in 2009.

As of 2011 SystemTap runs fully supported in all Linux distributions.

(2) 官方介绍

SystemTap provides free software(GPL) infrastructure to simplify the gathering of information about the

running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates

the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot

sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for

a live running kernel plus user-space application. We are publishing samples, as well as enlarging the internal

"tapset" script library to aid reuse and abstraction.

Among other tracing/probing tools, SystemTap is the tool of choice for complex tasks that may require live analysis,

programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing

jobs.

Current project members include Red Hat, IBM, Hitachi, and Oracle.

(3) 获取源代码

git clone git://sourceware.org/git/systemtap.git

安装

(1) Ubuntu发行版

1. 安装systemtap包

apt-get install systemtap

2. 安装依赖包

gcc：C语言编译器

elfutils：提供分析调试信息的库函数

linux-headers-generic：编译内核模块所需的内核头文件以及模块配置信息

3. 安装内核调试信息(kernel-debuginfo)

kernel-debuginfo提供了调试内核所需的符号表，假设没有安装的话SystemTap的威力就会大打折扣，

仅仅能提供kprobes系列的功能。

下载地址：http://ddebs.ubuntu.com/pool/main/l/linux/

下载相应的内核版本号，我的是linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

下载后安装：dpkg -i linux-image-3.11.0-12-generic-dbgsym_3.11.0-12.19_amd64.ddeb

4. 验证

stap -ve ‘probe kernel.function("do_fork") { print("hello world\n") exit() }‘

假设没有提示错误，就是成功安装了。

(2) CentOS/RedHat发行版

使用yum安装下列rpm包就可以：

systemtap：SystemTap包

gcc：C语言编译器

elfutils：提供库函数来分析调试信息

kernel-devel：编译内核模块所需的内核头文件及模块配置信息

kernel-debuginfo：提供所需的内核调试信息来定位内核函数和变量的位置

使用

一些样例SystemTap的简单样例。

(1) stap

通常直接使用stap运行用SystemTap语法编写的脚本就可以。

stap - systemtap script translator/driver

stap test.stp // .stp后缀的文件是用SystemTap语法编写的脚本

脚本主要元素：probe point + probe handler

stap [options] FILE // Run script in file 

stap [options] -e SCRIPT // Run given script.

stap [options] -l PROBE // List matching probes.

stap [options] -L PROBE // List matching probes and local variables.

经常使用选项

-h：帮助

-g：guru模式，嵌入式C代码须要

-m：指定编译成的模块名称

 -v：add verbosity to all passes

-k：不删除暂时文件夹

-p NUM：stop after pass NUM 1-5, instead of 5 (parse, elaborate, translate, compile, run)

-b：bulk (percpu file) mode, 使用RelayFS将数据从内核空间传输到用户空间

-o FILE：输出到指定文件，而不是stdout

-c CMD：start the probes, run CMD, and exit when it finishes

stap是SystemTap的前端。当出现下面情况时退出：

1. The user interrupts the script with a CTRL-C.

2. The script executes the exit() function.

3. The script encounters a sufficient number of soft errors.

4. The monitored command started with the stap program‘s -c option exits.

(2) staprun

假设我们的输入不是.stp脚本，而是一个用stap生成的模块。那么就用staprun来运行。

staprun - systemtap runtime

staprun [OPTIONS] MODULE [MODULE-OPTIONS]

staprun的作用：

The staprun program is the back-end of the Systemtap tool. It expects a kernel module produced by

the front-end stap tool.

Splitting the systemtap tool into a front-end and a back-end allows a user to compile a systemtap script

on a development machine that has the kernel debugging information (need to compile the script) and

then transfer the resulting kernel module to a production machine that doesn‘t have any development

tools or kernel debugging information installed.

staprun is a part of the SystemTap package, dedicated to module loading and unloading and kernel-to-user

data transfer.

经常使用选项

-o FILE：Send output to FILE. 

-D：Run in background. This requires ‘-o‘ option.

(3) 监測内核函数

一个简单脚本，每当内核函数do_fork()被调用时。显示调用它的进程名、进程ID、函数參数。

global proc_counter

probe begin {
	print("Started monitoring creation of new processes...Press ^C to terminate\n")
	printf("%-25s %-10s %-s\n", "Process Name", "Process ID", "Clone Flags")
}

probe kernel.function("do_fork") {
	proc_counter++
	printf("%-25s %-10d 0x%-x\n", execname(), pid(), $clone_flags)
}

probe end {
	printf("\n%d processes forked during the observed period\n", proc_counter)
}

(4) 监測系统调用

一个简单脚本。显示4秒内open系统调用的信息：调用进程名、进程ID、函数參数。

probe syscall.open
{
	printf("%s(%d) open(%s)\n", execname(), pid(), argstr)
}

probe timer.ms(4000) # after 4 seconds
{
	exit()
}

(5) 监測源文件里全部函数入口和出口

括号内的探測点描写叙述包括三个部分：

function name part：函数名

@file name part：文件名称

function line part：所在行号

比如：

probe kernel.function("*@net/socket.c") {}
probe kernel.function("*@net/socket.c").return {}

这里指定函数名为随意（用*表示），指定文件名称为net/socket.c。探測函数的入口和返回。

还能够用“：行号”来指定行号。

(6) 查找匹配的内核函数和变量

查找名字中包括nit的内核函数：

stap -l ‘kernel.function("*nit*")‘

查找名字中包括nit的内核函数和变量：

stap -L ‘kernel.function("*nit*")‘

(7) 自带的用例集

/root/systemtap/testsuite/systemtap.examples/。包括了很多用例脚本。

主要有几个方面：

network、io、interrupt、locks、memory、process、virtualization等

(8) 监控全部进程的收发包情况

global recv, xmit

probe begin {
	printf("Starting network capture...Press ^C to terminate\n")
}

probe netdev.receive {
	recv[dev_name, pid(), execname()] <<< length
}

probe netdev.transmit {
	xmit[dev_name, pid(), execname()] <<< length
}

probe end {
	printf("\nCapture terminated\n\n")
	printf("%-5s %-15s %-10s %-10s %-10s\n", 
		"If", "Process", "Pid", "RcvPktCnt", "XmtPktCnt")
	
	foreach([dev, pid, name] in recv) {
		recvcnt = @count(recv[dev, pid, name])
		xmtcnt =  @count(xmit[dev, pid, name])
		printf("%-5s %-15s %-10d %-10d %-10d\n", dev, name, pid, recvcnt, xmtcnt)
	}
}

(9) Systemtap usage stories and interesting demos

https://sourceware.org/systemtap/wiki/WarStories

官网提供的非常多样例。