Analyzing results from notebooks

The .ipynb format is capable of storing tables and charts in a standalone file. This makes it a great choice for model evaluation reports. NotebookCollection allows you to retrieve results from previously executed notebooks to compare them.

import papermill as pm
import jupytext

from sklearn_evaluation import NotebookCollection

Let’s first generate a few notebooks, we have a train.py script that trains a single model, let’s convert it to a jupyter notebook:

nb = jupytext.read('train.py')
jupytext.write(nb, 'train.ipynb')

We use papermill to execute the notebook with different parameters, we’ll train 4 models: 2 random forest, a linear regression and a support vector regression:

# models with their corresponding parameters
params = [{
    'model': 'sklearn.ensemble.RandomForestRegressor',
    'params': {
        'n_estimators': 50
    }
}, {
    'model': 'sklearn.ensemble.RandomForestRegressor',
    'params': {
        'n_estimators': 100
    }
}, {
    'model': 'sklearn.linear_model.LinearRegression',
    'params': {
        'normalize': True
    }
}, {
    'model': 'sklearn.svm.LinearSVR',
    'params': {}
}]

# ids to identify each experiment
ids = [
    'random_forest_1', 'random_forest_2', 'linear_regression',
    'support_vector_regression'
]

# output files
files = [f'{i}.ipynb' for i in ids]

# execute notebooks using papermill
for f, p in zip(files, params):
    pm.execute_notebook('train.ipynb', output_path=f, parameters=p)

To use NotebookCollection, we pass a a list of paths, and optionally, ids for each notebook (uses paths by default).

The only requirement is that cells whose output we want to extract must have tags, each tag then becomes a key in the notebook collection. For instructions on adding tags, see this.

Extracted tables add colors to certain cells to identify the best and worst metrics. By default, it assumes that metrics are errors (smaller is better). If you are using scores (larger is better), pass scores=True, if you have both, pass a list of scores:

nbs = NotebookCollection(paths=files, ids=ids, scores=['r2'])

To get a list of tags available:

list(nbs)

['model_name', 'feature_names', 'model_params', 'plot', 'metrics', 'river']

model_params contains a dictionary with model parameters, let’s get them (click on the tabs to switch):

# pro-tip: then typing the tag, press the "Tab" key for autocompletion!
nbs['model_params']

random_forest_1 random_forest_2 linear_regression support_vector_regression

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'mse',
    'max_depth': None,
    'max_features': 'auto',
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_impurity_split': None,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 50,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'mse',
    'max_depth': None,
    'max_features': 'auto',
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_impurity_split': None,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 100,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'copy_X': True,
    'fit_intercept': True,
    'n_jobs': None,
    'normalize': True,
    'positive': False,
}

{
    'C': 1.0,
    'dual': True,
    'epsilon': 0.0,
    'fit_intercept': True,
    'intercept_scaling': 1.0,
    'loss': 'epsilon_insensitive',
    'max_iter': 1000,
    'random_state': None,
    'tol': 0.0001,
    'verbose': 0,
}

plot has a y_true vs y_pred chart:

nbs['plot']

random_forest_1 random_forest_2 linear_regression support_vector_regression

On each notebook, metrics outputs a data frame with a single row with mean absolute error (mae) and mean squared error (mse) as columns.

For single-row tables, a “Compare” tab shows all results at once:

nbs['metrics']

Compare random_forest_1 random_forest_2 linear_regression support_vector_regression

	random_forest_1	random_forest_2	linear_regression	support_vector_regression
mae	2.195940	2.187443	3.148256	5.637194
mse	10.660879	10.239911	20.724023	43.524603
r2	0.859129	0.864692	0.726157	0.424875

	mae	mse	r2
0	2.19594	10.660879	0.859129

	mae	mse	r2
0	2.187443	10.239911	0.864692

	mae	mse	r2
0	3.148256	20.724023	0.726157

	mae	mse	r2
0	5.637194	43.524603	0.424875

We can see that the second random forest is performing the best in both metrics.

river contains a multi-row table where with error metrics broken down by the CHAS indicator feature. Multi-row tables do not display the “Compare” tab:

nbs['river']

random_forest_1 random_forest_2 linear_regression support_vector_regression

	mae	mse	r2
CHAS
0.0	2.232430	11.068710	0.858899
1.0	1.555333	3.501173	0.868473

	mae	mse	r2
CHAS
0.0	2.201354	10.542307	0.865609
1.0	1.943222	4.931181	0.814753

	mae	mse	r2
CHAS
0.0	3.145562	21.137297	0.730547
1.0	3.195546	13.468775	0.494026

	mae	mse	r2
CHAS
0.0	5.819687	45.465308	0.420420
1.0	2.433424	9.454464	0.644829

If we only compare two notebooks, the output is a bit different:

# only compare two notebooks
nbs_two = NotebookCollection(paths=files[:2], ids=ids[:2], scores=['r2'])

Comparing single-row tables includes a diff column with the error difference between experiments. Error reductions are showed in green, increments in red:

nbs_two['metrics']

Compare random_forest_1 random_forest_2

	random_forest_1	random_forest_2	diff	diff_relative	ratio
mae	2.195940	2.187443	-0.008497	-0.39%	0.996131
mse	10.660879	10.239911	-0.420968	-4.11%	0.960513
r2	0.859129	0.864692	0.005563	0.64%	1.006475

	mae	mse	r2
0	2.19594	10.660879	0.859129

	mae	mse	r2
0	2.187443	10.239911	0.864692

When comparing multi-row tables, the “Compare” tab appears, showing the difference between the tables:

nbs_two['river']

Compare random_forest_1 random_forest_2

	mae	mse	r2
CHAS
0.0	-0.031076	-0.526403	0.006710
1.0	0.387889	1.430008	-0.053720

	mae	mse	r2
CHAS
0.0	2.232430	11.068710	0.858899
1.0	1.555333	3.501173	0.868473

	mae	mse	r2
CHAS
0.0	2.201354	10.542307	0.865609
1.0	1.943222	4.931181	0.814753

When displaying dictionaries, a “Compare” tab shows with a diff view:

nbs_two['model_params']

Compare random_forest_1 random_forest_2

f	1	{	f	1	{
	2	'bootstrap': True,		2	'bootstrap': True,
	3	'ccp_alpha': 0.0,		3	'ccp_alpha': 0.0,
	4	'criterion': 'mse',		4	'criterion': 'mse',
	5	'max_depth': None,		5	'max_depth': None,
	6	'max_features': 'auto',		6	'max_features': 'auto',
	7	'max_leaf_nodes': None,		7	'max_leaf_nodes': None,
	8	'max_samples': None,		8	'max_samples': None,
	9	'min_impurity_decrease': 0.0,		9	'min_impurity_decrease': 0.0,
	10	'min_impurity_split': None,		10	'min_impurity_split': None,
	11	'min_samples_leaf': 1,		11	'min_samples_leaf': 1,
	12	'min_samples_split': 2,		12	'min_samples_split': 2,
	13	'min_weight_fraction_leaf': 0.0,		13	'min_weight_fraction_leaf': 0.0,
t	14	'n_estimators': 50,	t	14	'n_estimators': 100,
	15	'n_jobs': None,		15	'n_jobs': None,
	16	'oob_score': False,		16	'oob_score': False,
	17	'random_state': None,		17	'random_state': None,
	18	'verbose': 0,		18	'verbose': 0,
	19	'warm_start': False,		19	'warm_start': False,
	20	}		20	}

Legends
Colors  Added  Changed Deleted	Links (f)irst change (n)ext change (t)op

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'mse',
    'max_depth': None,
    'max_features': 'auto',
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_impurity_split': None,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 50,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'mse',
    'max_depth': None,
    'max_features': 'auto',
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_impurity_split': None,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 100,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

Lists (and sets) are compared based on elements existence:

nbs_two['feature_names']

Compare random_forest_1 random_forest_2

Both	Only in random_forest_1	Only in random_forest_2
AGE
B
CHAS
CRIM
DIS
INDUS
LSTAT
NOX
PTRATIO
RAD
RM
TAX
ZN

['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']

['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT']

Using the mapping interface

NotebookCollection has a dict-like interface, you can retrieve data from individual notebooks:

nbs['model_params']['random_forest_1']

{'bootstrap': True,
 'ccp_alpha': 0.0,
 'criterion': 'mse',
 'max_depth': None,
 'max_features': 'auto',
 'max_leaf_nodes': None,
 'max_samples': None,
 'min_impurity_decrease': 0.0,
 'min_impurity_split': None,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 50,
 'n_jobs': None,
 'oob_score': False,
 'random_state': None,
 'verbose': 0,
 'warm_start': False}

nbs['plot']['random_forest_2']