Classifier evaluation

sklearn-evluation has two main modules for evaluating classifiers: sklearn_evaluation.plot and sklearn_evaluation.table, let’s see an example of how to use them.

First, let’s load some data and split it in training and test set.

In [1]: data = datasets.make_classification(200, 10, n_informative=5,
   ...:                                     class_sep=0.65)
   ...: 

In [2]: X = data[0]

In [3]: y = data[1]

# shuffle and split training and test sets
In [4]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

Now, we are going to train the data using one of the scikit-learn classifiers.

In [5]: est = RandomForestClassifier(n_estimators=5)

In [6]: est.fit(X_train, y_train)
Out[6]: RandomForestClassifier(n_estimators=5)

Most of the functions require us to pass the class predictions for the test set (y_pred), the scores assigned (y_score) and the ground truth classes (y_true), let’s define such variables.

In [7]: y_pred = est.predict(X_test)

In [8]: y_score = est.predict_proba(X_test)

In [9]: y_true = y_test

We can start evaluating our model, the following example shows how to plot a confusion matrix.

In [10]: plot.confusion_matrix(y_true, y_pred)
Out[10]: <AxesSubplot:title={'center':'Confusion matrix'}, xlabel='Predicted label', ylabel='True label'>

Some classifiers (such as sklearn.ensemble.RandomForestClassifier) have feature importances, we can plot then passing the estimator object to the feature_importances function.

In [11]: plot.feature_importances(est, top_n=5)
Out[11]: <AxesSubplot:title={'center':'Feature importances'}>

A feature importances function is also available in the table module.

In [12]: print(table.feature_importances(est))
+----------------+--------------+-----------+
| feature_name   |   importance |      std_ |
+================+==============+===========+
| Feature 7      |    0.165648  | 0.0759156 |
+----------------+--------------+-----------+
| Feature 5      |    0.134308  | 0.072749  |
+----------------+--------------+-----------+
| Feature 1      |    0.131682  | 0.0682108 |
+----------------+--------------+-----------+
| Feature 4      |    0.128283  | 0.0728022 |
+----------------+--------------+-----------+
| Feature 8      |    0.0991082 | 0.0556539 |
+----------------+--------------+-----------+
| Feature 2      |    0.0874345 | 0.0662462 |
+----------------+--------------+-----------+
| Feature 10     |    0.0841662 | 0.0429043 |
+----------------+--------------+-----------+
| Feature 6      |    0.0740909 | 0.0789648 |
+----------------+--------------+-----------+
| Feature 3      |    0.0709322 | 0.0567116 |
+----------------+--------------+-----------+
| Feature 9      |    0.0243471 | 0.0131475 |
+----------------+--------------+-----------+

Now, let’s see how to generate two of the most common plots for evaluating classifiers: Precision-Recall and ROC.

In [13]: plot.precision_recall(y_true, y_score)
Out[13]: <AxesSubplot:title={'center':'Precision-Recall'}, xlabel='Recall', ylabel='Precision'>

In [14]: plot.roc(y_true, y_score)
Out[14]: <AxesSubplot:title={'center':'ROC'}, xlabel='False Positive Rate', ylabel='True Positive Rate'>