码迷,mamicode.com
首页 > 其他好文 > 详细

FileInputFormat导读getSplits

时间:2015-01-09 14:24:52      阅读:203      评论:0      收藏:0      [点我收藏+]

标签:

/**
   * Generate the list of files and make them into FileSplits.
   * @param job the job context
   * @throws IOException
   */
  public List<InputSplit> getSplits(JobContext job) throws IOException {
    Stopwatch sw = new Stopwatch().start();
    //获得一个InputSplit能够包含的最小值
    long minSize = Math.max(getFormatMinSplitSize(), getMinSplitSize(job));
    //获得一个InputSplit能够包含的最大值
    long maxSize = getMaxSplitSize(job);

    // generate splits
    List<InputSplit> splits = new ArrayList<InputSplit>();
    List<FileStatus> files = listStatus(job);
    /*
     * 由此可知,如果有一百万个小文件,就会循环一百万次,并且至少生成一百万个InputSplit,就至少含有一百万个map任务
     * 如果一个InputSplit的默认大小是一个block大小,即64M
     * 一个20M的文件会产生一个InputSplit,一个Map任务
     * 一个80M的文件会产生两个InputSplit,两个Map任务
     * 两个分别为20M的文件总共产生两个InputSplit,两个Map任务
     * 一个20M、一个70M的文件总共会产生三个InputSplit,三个Map任务
     */
    for (FileStatus file: files) {
      Path path = file.getPath();
      long length = file.getLen();
      if (length != 0) {
        BlockLocation[] blkLocations;
        if (file instanceof LocatedFileStatus) {
          blkLocations = ((LocatedFileStatus) file).getBlockLocations();
        } else {
          FileSystem fs = path.getFileSystem(job.getConfiguration());
          blkLocations = fs.getFileBlockLocations(file, 0, length);
        } if (isSplitable(job, path)) {
          //拿到hdfs默认的block块大小
          long blockSize = file.getBlockSize();
          //计算一个InputSplit的大小
          long splitSize = computeSplitSize(blockSize, minSize, maxSize);

          long bytesRemaining = length;
          while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, splitSize,
                        blkLocations[blkIndex].getHosts(),
                        blkLocations[blkIndex].getCachedHosts()));
            bytesRemaining -= splitSize;
          }

          if (bytesRemaining != 0) {
            int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
            splits.add(makeSplit(path, length-bytesRemaining, bytesRemaining,
                       blkLocations[blkIndex].getHosts(),
                       blkLocations[blkIndex].getCachedHosts()));
          }
        } else { // not splitable
          splits.add(makeSplit(path, 0, length, blkLocations[0].getHosts(),
                      blkLocations[0].getCachedHosts()));
        }
      } else {
        //Create empty hosts array for zero length files
        splits.add(makeSplit(path, 0, length, new String[0]));
      }
    }
    // Save the number of input files for metrics/loadgen
    job.getConfiguration().setLong(NUM_INPUT_FILES, files.size());
    sw.stop();
    if (LOG.isDebugEnabled()) {
      LOG.debug("Total # of splits generated by getSplits: " + splits.size() + ", TimeTaken: " + sw.elapsedMillis());
    }
    return splits;
  }

FileInputFormat导读getSplits

标签:

原文地址:http://my.oschina.net/sniperLi/blog/365483

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!