标签：hdfs add hdf target col stdout url for width
|
1.flume简介
Flume是Cloudera提供的日志收集系统，Flume支持在日志系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。
Flume是一个分布式、可靠、和高可用的海量日志采集、聚合和传输的系统。
2.安装和使用说明:
2.1 安装
a. 下载: http://archive.cloudera.com/cdh/3/ flume-0.9.0+1.tar.gz
接着解压.暂时用$flume代表解压路径.
c. 下载: http://archive.cloudera.com/cdh/3/ zookeeper-3.3.1.tar.gz
d. 安装zookeeper
yum install hadoop-zookeeper –y
yum install hadoop-zookeeper-server –y
修改/zookeeper-3.3.1/conf/ zoo_sample.cfg重命名为zoo.cfg
执行如下命令：
export ZOOKEEPER_HOME=/home/hadoop/zookeeper-3.3.1
export FLUME_HOME=/home/hadoop/flume-0.9.0+1
export PATH=.:$FLUME_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH
2.2 使用
执行>flume
输出如下：
usage: flume command [args...]
commands include:
dump Takes a specified source and dumps to console
node Start a Flume node/agent (with watchdog)
master Start a Flume Master server (with watchdog)
version Dump flume build version information
node_nowatch Start a flume node/agent (no watchdog)
master_nowatch Start a Flume Master server (no watchdog)
class <class> Run specified fully qualified class using Flume environment (no watchdog)
ex: flume com.cloudera.flume.agent.FlumeNode
classpath Dump the classpath used by the java executables
shell Start the flume shell
启动flume的master节点执行:bin/flume master
通过flume打开文件
输入命令
$ flume dump ‘tail("/home/hadoop/log/bb.txt")‘
输出：
<ignore_js_op>
 通过flume导入文件到hdfs
可打开http://10.1.27.30:35871/flumemaster.jsp 即可看到整理节点的情况
从上面URL打开的选项卡config,输入节点配置，然后点提交查询内容
如下：
<ignore_js_op>
 Source为数据源，可有多种输入源,sink为接收器,当启动master节点时，会把文件写入到hdsf里
启动配置好的节点：bin/flume node –n master
通过flume读取syslog-ng
<ignore_js_op>
 分别启动节点host和collector节点
<ignore_js_op>
 <ignore_js_op>
 3.附录:
Flume Event
Sources console
Stdin console
text("filename")
One shot text file source. One line is one event
tail("filename")
Similar to Unix’s tail -F. One line is one event. Stays open for more data and follows filename if file rotated.
multitail("file1"[, "file2"[, …]])
Similar to tail source but follows multiple files.
asciisynth(msg_count,msg_size)
A source that synthetically generates msg_count random messages of size msg_size. This converts all characters into printable ASCII characters.
syslogUdp(port)
Syslog over UDP port, port. This is syslog compatible.
syslogTcp(port)
Syslog over TCP port, port. This is syslog-ng compatible.
Flume Event Sinks
默认端口如下：
TCP ports are used in all situations.
|
标签：hdfs add hdf target col stdout url for width
原文地址：https://www.cnblogs.com/huanghanyu/p/13495445.html
