Gradient-Boosted Trees (GBTs) are ensembles of decision trees. GBTs iteratively train decision trees in order to minimize a loss function. Like decision trees, GBTs handle categorical features, do not require feature scaling, and are able to capture non-linearities and feature interactions.

　　spark.mllib supports GBTs for binary classification and for regression, using both continuous and categorical features. spark.mllib implements GBTs using the existing decision tree implementation. Please see the decision tree guide for more information on trees.

　　Note: GBTs do not yet support multiclass classification. For multiclass problems, please use decision trees or Random Forests.

   Gradient-Boosted Trees vs. Random Forests 
　　Both Gradient-Boosted Trees (GBTs) and Random Forests are algorithms for learning ensembles of trees, but the training processes are different. There are several practical trade-offs:

　　GBTs train one tree at a time, so they can take longer to train than random forests. Random Forests can train multiple trees in parallel.

　　On the other hand, it is often reasonable to use smaller (shallower) trees with GBTs than with Random Forests, and training smaller trees takes less time.

　　Random Forests can be less prone to overfitting. Training more trees in a Random Forest reduces the likelihood of overfitting, but training more trees with GBTs increases the likelihood of overfitting. (In statistical language, Random Forests reduce variance by using more trees, whereas GBTs reduce bias by using more trees.)

　　Random Forests can be easier to tune since performance improves monotonically with the number of trees (whereas performance can start to decrease for GBTs if the number of trees grows too large).

　　In short, both algorithms can be effective, and the choice should be based on the particular dataset.

//定义GBTClassifier，注意在Spark中输出（预测列）都有默认的设置，可以不自己设置
GBTClassifier gbtClassifier=new GBTClassifier()
                            .setLabelCol("indexedLabel")//输入label
                            .setFeaturesCol("indexedFeatures")//输入features vector
                            .setMaxIter(MaxIter)//最大迭代次数
                            .setImpurity("entropy")//or "gini"
                            .setMaxDepth(3)//决策树的深度
                            .setStepSize(0.3)//范围是(0, 1]
                            .setSeed(1234); //可以设一个随机数种子点

　(1) loss: See the section above for information on losses and their applicability to tasks (classification vs. regression). Different losses can give significantly different results, depending on the dataset. 
　　(2) numIterations: This sets the number of trees in the ensemble. Each iteration produces one tree. Increasing this number makes the model more expressive, improving training data accuracy. However, test-time accuracy may suffer if this is too large. 
Gradient boosting can overfit when trained with more trees. In order to prevent overfitting, it is useful to validate while training. The method runWithValidation has been provided to make use of this option. It takes a pair of RDD’s as arguments, the first one being the training dataset and the second being the validation dataset. 
　　The training is stopped when the improvement in the validation error is not more than a certain tolerance (supplied by the validationTol argument in BoostingStrategy). In practice, the validation error decreases initially and later increases. There might be cases in which the validation error does not change monotonically, and the user is advised to set a large enough negative tolerance and examine the validation curve using evaluateEachIteration (which gives the error or loss per iteration) to tune the number of iterations. 
　　(3) learningRate: This parameter should not need to be tuned. If the algorithm behavior seems unstable, decreasing this value may improve stability.

Validation while training 
　　Gradient boosting can overfit when trained with more trees. In order to prevent overfitting, it is useful to validate while training. The method runWithValidation has been provided to make use of this option. It takes a pair of RDD’s as arguments, the first one being the training dataset and the second being the validation dataset. 
　　The training is stopped when the improvement in the validation error is not more than a certain tolerance (supplied by the validationTol argument in BoostingStrategy). In practice, the validation error decreases initially and later increases. There might be cases in which the validation error does not change monotonically, and the user is advised to set a large enough negative tolerance and examine the validation curve using evaluateEachIteration (which gives the error or loss per iteration) to tune the number of iterations.

//Spark 2.0 GBDT完整代码
package my.spark.ml.practice.classification;

import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.ml.Pipeline;
import org.apache.spark.ml.PipelineModel;
import org.apache.spark.ml.PipelineStage;
import org.apache.spark.ml.classification.GBTClassifier;
import org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator;
import org.apache.spark.ml.feature.IndexToString;
import org.apache.spark.ml.feature.StringIndexer;
import org.apache.spark.ml.feature.StringIndexerModel;
import org.apache.spark.ml.feature.VectorIndexer;
import org.apache.spark.ml.feature.VectorIndexerModel;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class myGDBT {

    public static void main(String[] args) {
        SparkSession spark=SparkSession
                .builder()
                .appName("CoFilter")
                .master("local[4]")
                .config("spark.sql.warehouse.dir",
                        "file///:G:/Projects/Java/Spark/spark-warehouse" )
                .getOrCreate();         

        String path="C:/Users/user/Desktop/ml_dataset/classify/horseColicTraining2libsvm.txt";
        String path2="C:/Users/user/Desktop/ml_dataset/classify/horseColicTest2libsvm.txt";
        //屏蔽日志
        Logger.getLogger("org.apache.spark").setLevel(Level.ERROR);//WARN
        Logger.getLogger("org.eclipse.jetty.server").setLevel(Level.OFF);   

        Dataset<Row> training=spark.read().format("libsvm").load(path);
        Dataset<Row> test=spark.read().format("libsvm").load(path2);        

        StringIndexerModel indexerModel=new StringIndexer()
                .setInputCol("label")
                .setOutputCol("indexedLabel")
                .fit(training);
        VectorIndexerModel vectorIndexerModel=new VectorIndexer()
                .setInputCol("features")
                .setOutputCol("indexedFeatures")
                .fit(training);
        IndexToString converter=new IndexToString()
                .setInputCol("prediction")
                .setOutputCol("convertedPrediction")
                .setLabels(indexerModel.labels());
        //调试参数MaxIter,learningRate,maxDepth，也对两种不纯度进行了测试                
       for (int MaxIter = 30; MaxIter < 40; MaxIter+=10)
          for (int maxDepth = 2; maxDepth < 3; maxDepth+=1)
              for (int impurityType = 1; impurityType <2; impurityType+=1)
                 for (int setpSize = 1; setpSize< 10; setpSize+=1) {    
                    long begin = System.currentTimeMillis();//训练开始时间
                    String impurityType_=null;//不纯度类型选择
                    if (impurityType==1) {
                        impurityType_="gini";
                    }
                    else  {
                        impurityType_="entropy";
                    }
                    double setpSize_=0.1*setpSize;
                    GBTClassifier gbtClassifier=new GBTClassifier()
                            .setLabelCol("indexedLabel")
                            .setFeaturesCol("indexedFeatures")
                            .setMaxIter(MaxIter)
                            .setImpurity(impurityType_)//.setImpurity("entropy")
                            .setMaxDepth(maxDepth)
                            .setStepSize(setpSize_)//范围是(0, 1]
                            .setSeed(1234);                     

                    PipelineModel pipeline=new Pipeline().setStages
                            (new PipelineStage[]
                                    {indexerModel,vectorIndexerModel,gbtClassifier,converter})
                            .fit(training);     
                    long end=System.currentTimeMillis();        

                    //一定要在测试数据集上做验证
                    Dataset<Row> predictDataFrame=pipeline.transform(test);     

                    double accuracy=new MulticlassClassificationEvaluator()
                            .setLabelCol("indexedLabel")
                            .setPredictionCol("prediction")
                            .setMetricName("accuracy").evaluate(predictDataFrame);          
                    String str_accuracy=String.format(" accuracy = %.4f ", accuracy);
                    String str_time=String.format(" trainig time = %d ", (end-begin));
                    String str_maxIter=String.format(" maxIter = %d ", MaxIter);
                    String str_maxDepth=String.format(" maxDepth = %d ", maxDepth);
                    String str_stepSize=String.format(" setpSize = %.2f ", setpSize_);
                    String str_impurityType_=" impurityType = "+impurityType_;
                    System.out.println(str_maxIter+str_maxDepth+str_impurityType_+
                            str_stepSize+str_accuracy+str_time);

                }//Params Cycle         
    }   
}

/*下面的参数分析只是针对这个小数据集，实际不同数据会有很大差别，仅仅是一种非常的简单的测试而已*/
/**迭代次数影响：随着次数的增加，开始在测试上准确度会提高，训练时间呈线性增长。
maxIter = 1  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7313  trainig time = 1753 
 maxIter = 11  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7463  trainig time = 2820 
 maxIter = 21  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7612  trainig time = 5043 
 maxIter = 31  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7761  trainig time = 7217 
 maxIter = 41  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7761  trainig time = 9932 
 maxIter = 51  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7761  trainig time = 12337 
 maxIter = 61  maxDepth = 1  impurityType = entropy setpSize = 0.10  accuracy = 0.7761  trainig time = 15091 
 */
/**

 Gradient Tree Boosting or Gradient Boosted Regression Trees (GBRT) is a generalization of boosting to arbitrary differentiable loss functions. GBRT is an accurate and effective off-the-shelf procedure that can be used for both regression and classification problems. Gradient Tree Boosting models are used in a variety of areas including Web search ranking and ecology.

from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import GradientBoostingClassifier

#加载一个Demo数据集
X, y = make_hastie_10_2(random_state=0)
X_train, X_test = X[:2000], X[2000:]
y_train, y_test = y[:2000], y[2000:]

#定义参数，训练分类器
clf = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0,
max_depth=1, random_state=0).fit(X_train, y_train)

#测试集上评估
clf.score(X_test, y_test)
Out[7]: 0.91300000000000003

[HTF2009] Hastie, R. Tibshirani and J. Friedman, “Elements of 
Statistical Learning Ed. 2”, Springer, 2009. 
[R2007] Ridgeway,Generalized Boosted Models: A guide to the gbm package”, 2007 
还没有时间看这两个文献，希望有时间再学习学习。 
The parameter learning_rate strongly interacts with the parameter n_estimators, the number of weak learners to fit. Smaller values of learning_rate require larger numbers of weak learners to maintain a constant training error. Empirical evidence suggests that small values of learning_rate favor better test error. [HTF2009] recommend to set the learning rate to a small constant (e.g. learning_rate <= 0.1) and choose n_estimators by early stopping. For a more detailed discussion of the interaction between learning_rate and n_estimators see [R2007].

#GDBT python参数测试代码
from sklearn.datasets import make_hastie_10_2
from sklearn.ensemble import GradientBoostingClassifier

X, y = make_hastie_10_2(random_state=0)
X_train, X_test = X[:2000], X[2000:]

y_train, y_test = y[:2000], y[2000:]

‘‘‘
n_estimators_ =[10,100,300,500,1000]
learning_rate_=[0.05,0.10,0.2,0.5,1.0]
for i in range(5):
    for j in range(5): 
        clf = GradientBoostingClassifier(n_estimators=n_estimators_[i],        learning_rate=learning_rate_[j],        max_depth=1,random_state=0).fit(X_train, y_train)

        print ("n_estimators = "+str(n_estimators_[i])        +"  learning_rate = "+str(learning_rate_[j])+         "  score = "+str(clf.score(X_test, y_test)))
‘‘‘
n_estimators_ =[10,100,300,500,1000,2000,5000]
learning_rate_=[0.05]
for i in range(7):
    for j in range(1): 
        clf = GradientBoostingClassifier(n_estimators=n_estimators_[i],        learning_rate=learning_rate_[j],        max_depth=1,random_state=0).fit(X_train, y_train)

        print ("n_estimators = "+str(n_estimators_[i])        +"  learning_rate = "+str(learning_rate_[j])+         "  score = "+str(clf.score(X_test, y_test)))