HDFS 读取、写入、遍历目录获取文件全路径

标签：hadoop   hdfs   读取写入遍历目录   

1、从HDFS中读取数据

Configuration conf = getConf();
  Path path = new Path(pathstr); 
  FileSystem fs = FileSystem.get(conf);
   FSDataInputStream fsin= fs.open(path ); 
   BufferedReader br =null;
   String line ;
   try{
    br = new BufferedReader(new InputStreamReader(fsin));
       while ((line = br.readLine()) != null) {
         System.out.println(line);
        } 
   }finally{
    br.close();
   }

  Configuration conf = getConf();
  Path path = new Path(mid_sort); 
  FileSystem fs = FileSystem.get(conf); 
  FSDataOutputStream out = fs.create(resultpath);
  out.write(sb.toString().getBytes());
  out.close();

/**
  * 得到一个目录(不包括子目录)下的所有名字匹配上pattern的文件名
  * @param fs
  * @param folderPath
  * @param pattern 用于匹配文件名的正则
  * @return
  * @throws IOException
  */
 public static List<Path> getFilesUnderFolder(FileSystem fs, Path folderPath, String pattern) throws IOException {
  List<Path> paths = new ArrayList<Path>();
  if (fs.exists(folderPath)) {
   FileStatus[] fileStatus = fs.listStatus(folderPath);
   for (int i = 0; i < fileStatus.length; i++) {
    FileStatus fileStatu = fileStatus[i];
    if (!fileStatu.isDir()) {//只要文件
     Path oneFilePath = fileStatu.getPath();
     if (pattern == null) {
      paths.add(oneFilePath);
     } else {
      if (oneFilePath.getName().contains(pattern)) {
       paths.add(oneFilePath);
      }
     }  
    }
   }
  }
  return paths;
 }

HDFS 读取、写入、遍历目录获取文件全路径,布布扣,bubuko.com

HDFS 读取、写入、遍历目录获取文件全路径

标签：hadoop   hdfs   读取写入遍历目录   

原文地址：http://blog.csdn.net/smile0198/article/details/37573081