采用ToolRunner执行Hadoop基本面分析程序

A utility to help run Tools.

ToolRunner can be used to run classes implementing Tool interface. It works in conjunction with GenericOptionsParser to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. The application-specific options are passed along without being modified.

run

public static int run(Configuration conf,
                      Tool tool,
                      String[] args)
               throws Exception

Runs the given Tool by Tool.run(String[]), after parsing with the given generic arguments. Uses the given Configuration, or builds one if null. Sets the Tool‘s configuration with the possibly modified version of the conf.

Parameters:

conf - Configuration for the Tool.

tool - Tool to run.

args - command-line arguments to the tool.

Returns:

exit code of the Tool.run(String[]) method.

Throws:

Exception

run

public static int run(Tool tool,
                      String[] args)
               throws Exception

Runs the Tool with its Configuration. Equivalent to run(tool.getConf(), tool, args).

Parameters:

tool - Tool to run.

args - command-line arguments to the tool.

Returns:

exit code of the Tool.run(String[]) method.

Throws:

Exception

它们均是静态方法。即能够通过类名调用。

除此以外，另一个方法：

static void printGenericCommandUsage(PrintStream out) 
          Prints generic command-line argurments and usage information.

4、ToolRunner完毕下面2个功能：

（1）为Tool创建一个Configuration对象。

（2）使得程序能够方便的读取參数配置。

ToolRunner完整源码例如以下：

package org.apache.hadoop.util;

import java.io.PrintStream;

import org.apache.hadoop.conf.Configuration;

/**
 * A utility to help run {@link Tool}s.
 * 
 * <p><code>ToolRunner</code> can be used to run classes implementing 
 * <code>Tool</code> interface. It works in conjunction with 
 * {@link GenericOptionsParser} to parse the 
 * <a href="{@docRoot}/org/apache/hadoop/util/GenericOptionsParser.html#GenericOptions">
 * generic hadoop command line arguments</a> and modifies the 
 * <code>Configuration</code> of the <code>Tool</code>. The 
 * application-specific options are passed along without being modified.
 * </p>
 * 
 * @see Tool
 * @see GenericOptionsParser
 */
public class ToolRunner {
 
  /**
   * Runs the given <code>Tool</code> by {@link Tool#run(String[])}, after 
   * parsing with the given generic arguments. Uses the given 
   * <code>Configuration</code>, or builds one if null.
   * 
   * Sets the <code>Tool</code>'s configuration with the possibly modified 
   * version of the <code>conf</code>.  
   * 
   * @param conf <code>Configuration</code> for the <code>Tool</code>.
   * @param tool <code>Tool</code> to run.
   * @param args command-line arguments to the tool.
   * @return exit code of the {@link Tool#run(String[])} method.
   */
  public static int run(Configuration conf, Tool tool, String[] args) 
    throws Exception{
    if(conf == null) {
      conf = new Configuration();
    }
    GenericOptionsParser parser = new GenericOptionsParser(conf, args);
    //set the configuration back, so that Tool can configure itself
    tool.setConf(conf);
    
    //get the args w/o generic hadoop args
    String[] toolArgs = parser.getRemainingArgs();
    return tool.run(toolArgs);
  }
  
  /**
   * Runs the <code>Tool</code> with its <code>Configuration</code>.
   * 
   * Equivalent to <code>run(tool.getConf(), tool, args)</code>.
   * 
   * @param tool <code>Tool</code> to run.
   * @param args command-line arguments to the tool.
   * @return exit code of the {@link Tool#run(String[])} method.
   */
  public static int run(Tool tool, String[] args) 
    throws Exception{
    return run(tool.getConf(), tool, args);
  }
  
  /**
   * Prints generic command-line argurments and usage information.
   * 
   *  @param out stream to write usage information to.
   */
  public static void printGenericCommandUsage(PrintStream out) {
    GenericOptionsParser.printGenericCommandUsage(out);
  }
  
}

Unless explicitly turned off, Hadoop by default specifies two resources, loaded in-order from the classpath:

1. core-default.xml : Read-only defaults for hadoop.
2. core-site.xml: Site-specific configuration for a given hadoop installation.

static{
    //print deprecation warning if hadoop-site.xml is found in classpath
    ClassLoader cL = Thread.currentThread().getContextClassLoader();
    if (cL == null) {
      cL = Configuration.class.getClassLoader();
    }
    if(cL.getResource("hadoop-site.xml")!=null) {
      LOG.warn("DEPRECATED: hadoop-site.xml found in the classpath. " +
          "Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, "
          + "mapred-site.xml and hdfs-site.xml to override properties of " +
          "core-default.xml, mapred-default.xml and hdfs-default.xml " +
          "respectively");
    }
    addDefaultResource("core-default.xml");
    addDefaultResource("core-site.xml");
  }

for (Entry<String, String> entry : conf)｛
.....
｝

package org.apache.hadoop.util;

import org.apache.hadoop.conf.Configurable;

public interface Tool extends Configurable {
  
  int run(String [] args) throws Exception;
}

package org.apache.hadoop.conf;

public interface Configurable {

  void setConf(Configuration conf);

  Configuration getConf();
}

package org.apache.hadoop.conf;

public class Configured implements Configurable {

  private Configuration conf;
  public Configured() {
    this(null);
  }
 
  public Configured(Configuration conf) {
    setConf(conf);
  }

  public void setConf(Configuration conf) {
    this.conf = conf;
  }

  public Configuration getConf() {
    return conf;
  }

}