scikit-learn：3.3. Model evaluation: quantifying the quality of predictions

參考：http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter

三种方法评估模型的预測质量：

• Estimator score method: Estimators都有 score method作为默认的评估标准，不属于本节内容。详细參考不同estimators的文档。

• Scoring parameter: Model-evaluation toolsusing cross-validation (such as cross_validation.cross_val_score and grid_search.GridSearchCV) rely on an internal scoring strategy. 本节讨论The scoring parameter: defining model evaluation rules.（參考第一小节）
• Metric functions: The metrics module 能较全面评价预測质量，本节讨论Classification metrics, Multilabel ranking metrics, Regression metrics and Clustering metrics.（參考二、三、四、五小节）

最后介绍 Dummy estimators 。提供随机推測的策略，能够作为预測质量评价的baseline。

（參考第六小节）

See also

For “pairwise” metrics, between samples and not estimators or predictions, see the Pairwise metrics, Affinities and Kernels section.

详细内容有时间再写。。

。

1、

Model selection and evaluation using tools, such as grid_search.GridSearchCV and cross_validation.cross_val_score, take a scoring parameter that controls what metric they apply to the estimators evaluated.

1）提前定义的标准

全部的scorer都是越大越好。因此mean_absolute_error and mean_squared_error（測量预測点离模型的距离）是负值。

给个样例：

>>> from sklearn import svm, cross_validation, datasets
>>> iris = datasets.load_iris()
>>> X, y = iris.data, iris.target
>>> model = svm.SVC()
>>> cross_validation.cross_val_score(model, X, y, scoring=‘wrong_choice‘)
Traceback (most recent call last):
ValueError: ‘wrong_choice‘ is not a valid scoring value. Valid options are [‘accuracy‘, ‘adjusted_rand_score‘, ‘average_precision‘, ‘f1‘, ‘f1_macro‘, ‘f1_micro‘, ‘f1_samples‘, ‘f1_weighted‘, ‘log_loss‘, ‘mean_absolute_error‘, ‘mean_squared_error‘, ‘median_absolute_error‘, ‘precision‘, ‘precision_macro‘, ‘precision_micro‘, ‘precision_samples‘, ‘precision_weighted‘, ‘r2‘, ‘recall‘, ‘recall_macro‘, ‘recall_micro‘, ‘recall_samples‘, ‘recall_weighted‘, ‘roc_auc‘]
>>> clf = svm.SVC(probability=True, random_state=0)
>>> cross_validation.cross_val_score(clf, X, y, scoring=‘log_loss‘) 
array([-0.07..., -0.16..., -0.06...])

3）自己定义scoring标准

following two rules:

• It can be called with parameters (estimator, X, y), where estimator is the model that should be evaluated, X is validation data, and y is the ground truth target for X (in the supervised case) or None (in the unsupervised case).
• It returns a floating point number that quantifies the estimator prediction quality on X, with reference to y. Again, by convention higher numbers are better, so if your scorer returns loss, that value should be negated.

2、

2）accuracy score：

The accuracy_score function computes the accuracy, 默认是计算预測正确的比例，假设设置normalize=False。计算预測正确的绝对数量。给个样例就明确：

>>> import numpy as np
>>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2

对于multilabel classification，仅仅有所有的labels所有预測对。该sample才算预測对。

给个样例就明确：

>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5

• See Test with permutations the significance of a classification score for an example of accuracy score usage using permutations of the dataset.

3）confusion matrix：

The confusion_matrix function evaluates classification accuracy by computing the confusion matrix. 给个样例：

>>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])

再參考：

• See Confusion matrix for an example of using a confusion matrix to evaluate classifier output quality.
• See Recognizing hand-written digits for an example of using a confusion matrix to classify hand-written digits.
• See Classification of text documents using sparse features for an example of using a confusion matrix to classify text documents.

4）classification report：

The classification_report function builds a text report showing the main classification metrics. 给个样例：

>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 0]
>>> y_pred = [0, 0, 2, 2, 0]
>>> target_names = [‘class 0‘, ‘class 1‘, ‘class 2‘]
>>> print(classification_report(y_true, y_pred, target_names=target_names))
             precision    recall  f1-score   support

    class 0       0.67      1.00      0.80         2
    class 1       0.00      0.00      0.00         1
    class 2       1.00      1.00      1.00         2

avg / total       0.67      0.80      0.72         5

• See Recognizing hand-written digits for an example of classification report usage for hand-written digits.
• See Classification of text documents using sparse features for an example of classification report usage for text documents.
• See Parameter estimation using grid search with cross-validation for an example of classification report usage for grid search with nested cross-validation.

5）hamming loss：

If  is the predicted value for the -th label of a given sample,  is the corresponding true value, and  is the number of classes or labels, then the Hamming loss  between two samples is defined as:

6）jaccard similarity coefficient score：

The Jaccard similarity coefficient of the -th samples, with a ground truth label set  and predicted label set , is defined as

Several functions allow you to analyze the precision, recall and F-measures score:

Note that the precision_recall_curve function is restricted to the binary case. The average_precision_score function works only in binary classification and multilabel indicator format.

8）hinge loss：

9）log loss：

10）matthews correlation coefficient：

11）receiver operating characteristic（ROC）：

12）zero one loss：

3、

In multilabel learning, each sample can have any number of ground truth labels associated with it. The goal is to give high scores and better rank to the ground truth labels.

1）coverage error：

2）label ranking average precision：

4、

The sklearn.metrics module implements several loss, score, and utility functions to measure regression performance. 

Some of those have been enhanced to handle the multioutput case: mean_absolute_error, mean_squared_error, median_absolute_error and r2_score.

1）explained variance score：

If  is the estimated target output,  the corresponding (correct) target output, and  is Variance, the square of the standard deviation, then the explained variance is estimated as follow:

2）mean absolute error：

If  is the predicted value of the -th sample, and  is the corresponding true value, then the mean absolute error (MAE) estimated over  is defined as

3）mean squared error：

If  is the predicted value of the -th sample, and  is the corresponding true value, then the mean squared error (MSE) estimated over  is defined as

4）R^2 score、the coefficient of determination：

If  is the predicted value of the -th sample and  is the corresponding true value, then the score R2 estimated over  is defined as

5、

The sklearn.metrics module implements several loss, score, and utility functions. For more information see the Clustering performance evaluation section for instance clustering, and Biclustering evaluation for biclustering.

6、Dummy estimators

>>> from sklearn.datasets import load_iris
>>> from sklearn.cross_validation import train_test_split
>>> iris = load_iris()
>>> X, y = iris.data, iris.target
>>> y[y != 1] = -1
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

>>> from sklearn.dummy import DummyClassifier
>>> from sklearn.svm import SVC
>>> clf = SVC(kernel=‘linear‘, C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test) 
0.63...
>>> clf = DummyClassifier(strategy=‘most_frequent‘,random_state=0)
>>> clf.fit(X_train, y_train)
DummyClassifier(constant=None, random_state=0, strategy=‘most_frequent‘)
>>> clf.score(X_test, y_test)  
0.57...

>>> clf = SVC(kernel=‘rbf‘, C=1).fit(X_train, y_train)
>>> clf.score(X_test, y_test)  
0.97...