标签：
我们的输入文件 hello0, 内容如下:
xiaowang 28 shanghai@_@zhangsan 38 beijing@_@someone 100 unknown
逻辑上有3条记录, 它们以@_@分隔. 我们将分别用旧版MapReduce API 和新版MapReduce API实现自定义TextInputFormat，然后在hive配置使用， 加载数据.
首先用旧版API
1, 自定义Format6继承自TextInputFormat
package MyTestPackage; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.FileSplit; import org.apache.hadoop.mapred.InputSplit; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.JobConfigurable; import org.apache.hadoop.mapred.LineRecordReader; import org.apache.hadoop.mapred.RecordReader; import org.apache.hadoop.mapred.Reporter; import org.apache.hadoop.mapred.TaskAttemptContext; import org.apache.hadoop.mapred.TextInputFormat; public class Format6 extends TextInputFormat { @Override public RecordReader getRecordReader (InputSplit split, JobConf job, Reporter reporter) throws IOException { byte[] recordDelimiterBytes = "@_@".getBytes(); return new LineRecordReader(job, (FileSplit)split, recordDelimiterBytes); } }
2.导出为MyInputFormat.jar, 放到 hive/lib中
3.在HIVE DDL中配置使用
create table hive_again2(name varchar(50), age int, city varchar(30)) stored as INPUTFORMAT ‘MyTestPackage.Format6‘ OUTPUTFORMAT ‘org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat‘;
成功load数据-
新API
1, 自定义Format5继承自TextInputFormat
package MyTestPackage; import org.apache.hadoop.mapreduce.InputSplit; import org.apache.hadoop.mapreduce.RecordReader; import org.apache.hadoop.mapreduce.TaskAttemptContext; import org.apache.hadoop.mapreduce.lib.input.LineRecordReader; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; public class Format5 extends TextInputFormat { @Override public RecordReader createRecordReader (InputSplit split, TaskAttemptContext tac) { byte[] recordDelimiterBytes = "@_@".getBytes(); return new LineRecordReader(recordDelimiterBytes); } }
2.导出为MyInputFormat.jar, 放到 hive/lib中
3.在HIVE DDL中配置使用
create table hive_again1(name varchar(50), age int, city varchar(30)) stored as INPUTFORMAT ‘MyTestPackage.Format5‘ OUTPUTFORMAT ‘org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat‘;
出错了!
把Format5放到MapReduce中debug一切正常(http://www.cnblogs.com/silva/p/4490532.html), 为什么给hive用就不行了呢? 没明白.. 有知道的同学请指点.谢了!
HIVE自定义TextInputFormat (旧版MapReduceAPI ok, 新版MapReduceAPI实现有BUG？)
标签：
原文地址：http://www.cnblogs.com/silva/p/4491281.html
