Classifier evaluation

sklearn-evluation has two main modules for evaluating classifiers: sklearn_evaluation.plot and sklearn_evaluation.table, let’s see an example of how to use them.

First, let’s load some data and split it in training and test set.

In [1]: data = datasets.make_classification(200, 10, n_informative=5,
   ...:                                     class_sep=0.65)
   ...: 

In [2]: X = data[0]

In [3]: y = data[1]

# shuffle and split training and test sets
In [4]: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

Now, we are going to train the data using one of the scikit-learn classifiers.

In [5]: est = RandomForestClassifier(n_estimators=5)

In [6]: est.fit(X_train, y_train)
Out[6]: RandomForestClassifier(n_estimators=5)

Most of the functions require us to pass the class predictions for the test set (y_pred), the scores assigned (y_score) and the ground truth classes (y_true), let’s define such variables.

In [7]: y_pred = est.predict(X_test)

In [8]: y_score = est.predict_proba(X_test)

In [9]: y_true = y_test

We can start evaluating our model, the following example shows how to plot a confusion matrix.

In [10]: plot.confusion_matrix(y_true, y_pred)
Out[10]: <AxesSubplot:title={'center':'Confusion matrix'}, xlabel='Predicted label', ylabel='True label'>
../_images/cm.png

Some classifiers (such as sklearn.ensemble.RandomForestClassifier) have feature importances, we can plot then passing the estimator object to the feature_importances function.

In [11]: plot.feature_importances(est, top_n=5)
Out[11]: <AxesSubplot:title={'center':'Feature importances'}>
../_images/fi.png

A feature importances function is also available in the table module.

In [12]: print(table.feature_importances(est))
+----------------+--------------+-----------+
| feature_name   |   importance |      std_ |
+================+==============+===========+
| Feature 1      |    0.201174  | 0.175533  |
+----------------+--------------+-----------+
| Feature 10     |    0.169756  | 0.0793474 |
+----------------+--------------+-----------+
| Feature 6      |    0.153439  | 0.110896  |
+----------------+--------------+-----------+
| Feature 4      |    0.127926  | 0.119564  |
+----------------+--------------+-----------+
| Feature 9      |    0.109839  | 0.0974525 |
+----------------+--------------+-----------+
| Feature 5      |    0.0896095 | 0.0664194 |
+----------------+--------------+-----------+
| Feature 3      |    0.0418469 | 0.0242582 |
+----------------+--------------+-----------+
| Feature 7      |    0.0363967 | 0.032281  |
+----------------+--------------+-----------+
| Feature 8      |    0.0360637 | 0.016887  |
+----------------+--------------+-----------+
| Feature 2      |    0.0339492 | 0.041633  |
+----------------+--------------+-----------+

Now, let’s see how to generate two of the most common plots for evaluating classifiers: Precision-Recall and ROC.

In [13]: plot.precision_recall(y_true, y_score)
Out[13]: <AxesSubplot:title={'center':'Precision-Recall'}, xlabel='Recall', ylabel='Precision'>
../_images/pr.png
In [14]: plot.roc(y_true, y_score)
Out[14]: <AxesSubplot:title={'center':'ROC'}, xlabel='False Positive Rate', ylabel='True Positive Rate'>
../_images/roc1.png