MRCpy.MRC

class MRCpy.MRC(loss='0-1', s=0.3, deterministic=True, random_state=None, fit_intercept=True, solver='subgrad', max_iters=10000, n_max=100, k_max=20, eps=0.0001, phi='linear', **phi_kwargs)

Minimax Risk Classifier

The class MRC implements the method Minimimax Risk Classifiers (MRC) proposed in [1] using the default constraints. It implements two kinds of loss functions, namely 0-1 and log loss.

The method MRC approximates the optimal classification rule by an optimization problem of the form

\[\mathcal{P}_{\text{MRC}}: \min_{h\in T(\mathcal{X},\mathcal{Y})} \max_{p\in\mathcal{U}} \ell(h,p)\]

where we consider an uncertainty set \(\mathcal{U}\) of potential probabilities. These untertainty sets of distributions are given by constraints on the expectations of a vector-valued function \(\phi : \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}^m\) referred to as feature mapping.

This is a subclass of the super class BaseMRC.

See Examples of use for futher applications of this class and its methods.

See also

For more information about MRC, one can refer to the following resources:

[1] Mazuelas, S., Zanoni, A., & Pérez, A. (2020). Minimax Classification with 0-1 Loss and Performance Guarantees. Advances in Neural Information Processing Systems, 33, 302-312.

[2] Mazuelas, S., Shen, Y., & Pérez, A. (2020). Generalized Maximum Entropy for Supervised Classification. arXiv preprint arXiv:2007.05447.

[3] Bondugula, K., Mazuelas, S., & Pérez, A. (2021). MRCpy: A Library for Minimax Risk Classifiers. arXiv preprint arXiv:2108.01952.

Parameters

lossstr {‘0-1’, ‘log’}, default = ‘0-1’

Type of loss function to use for the risk minimization. 0-1 loss quantifies the probability of classification error at a certain example for a certain rule. Log-loss quantifies the minus log-likelihood at a certain example for a certain rule.

sfloat, default = 0.3

Parameter that tunes the estimation of expected values of feature mapping function. It is used to calculate \(\lambda\) (variance in the mean estimates for the expectations of the feature mappings) in the following way

\[\lambda = s * \text{std}(\phi(X,Y)) / \sqrt{\left| X \right|}\]

where (X,Y) is the dataset of training samples and their labels respectively and \(\text{std}(\phi(X,Y))\) stands for standard deviation of \(\phi(X,Y)\) in the supervised dataset (X,Y).

deterministicbool, default = True

Whether the prediction of the labels should be done in a deterministic way (given a fixed random_state in the case of using random Fourier or random ReLU features).

random_stateint, RandomState instance, default = None

Random seed used when ‘fourier’ and ‘relu’ options for feature mappings are used to produce the random weights.

fit_interceptbool, default = True

Whether to calculate the intercept for MRCs If set to false, no intercept will be used in calculations (i.e. data is expected to be already centered).

solver{‘cvx’, ’subgrad’, ’cg’}, default = ’subgrad’

Method to use in solving the optimization problem. Default is ‘cvx’. To choose a solver, you might want to consider the following aspects:

’cvx’

Solves the optimization problem using the CVXPY library. Obtains an accurate solution while requiring more time than the other methods. Note that the library uses the GUROBI solver in CVXpy for which one might need to request for a license. A free license can be requested here

’subgrad’

Solves the optimization using a subgradient approach. The parameter max_iters determines the number of iterations for this approach. More iteration lead to an accurate solution while requiring more time.

’cg’

Solves the optimization using an algorithm based on constraint generation. This algorithm provides efficient learning especially for scenarios with large number of features.

See also

For more information about the constraint generation algorithm for 0-1 MRC, one can refer to the following resource:

[1] Bondugula, K., Mazuelas, S., & Pérez, A. (2023). Efficient Learning of Minimax Risk Classifiers in High Dimensions. The 39th Conference on Uncertainty in Artificial Intelligence, 206-215.

max_itersint, default = 10000

Maximum number of iterations to use for finding the solution of optimization when using the subgradient approach.

n_maxint, default = 100

Maximum number of features selected in each iteration in case of ’cg’ solver.

k_maxint, default = 20

Maximum number of iterations in case of ’cg’ solver.

epsfloat, default = 1e-4

Dual constraints’ violation threshold for ’cg’ solver.

phistr or BasePhi instance, default = ‘linear’

Type of feature mapping function to use for mapping the input data. The currenlty available feature mapping methods are ‘fourier’, ‘relu’, ‘threshold’ and ‘linear’. The users can also implement their own feature mapping object (should be a BasePhi instance) and pass it to this argument. Note that when using ‘fourier’ or ‘relu’ feature mappings, training and testing instances are expected to be normalized. To implement a feature mapping, please go through the Feature Mappings section.

‘linear’

It uses the identity feature map referred to as Linear feature map. See class BasePhi.

‘fourier’

It uses Random Fourier Feature map. See class RandomFourierPhi.

‘relu’

It uses Rectified Linear Unit (ReLU) features. See class RandomReLUPhi.

‘threshold’

It uses Feature mappings obtained using a threshold. See class ThresholdPhi.

**phi_kwargsAdditional parameters for feature mappings.

Groups the multiple optional parameters for the corresponding feature mappings(phi).

For example in case of fourier features, the number of features is given by n_components parameter which can be passed as argument MRC(loss='log', phi='fourier', n_components=500)

The list of arguments for each feature mappings class can be found in the corresponding documentation.

Examples

Simple example of using MRC with default seetings: 0-1 loss and linear feature mapping. We first load the data and split it into train and test sets. We fit the model with the training samples using fit function. Then, we predict the class of some test samples with predict. We can also obtain the probabilities of each class with predict_proba. Finally, we calculate the score of the model over the test set using score.

>>> from MRCpy import MRC
>>> from MRCpy.datasets import load_mammographic
>>> from sklearn import preprocessing
>>> from sklearn.model_selection import train_test_split
>>> # Loading the dataset
>>> X, Y = load_mammographic(return_X_y=True)
>>> # Split the dataset into training and test instances
>>> X_train, X_test, Y_train, Y_test =
train_test_split(X, Y, test_size=0.2, random_state=0)
>>> # Standarize the data
>>> std_scale = preprocessing.StandardScaler().fit(X_train, Y_train)
>>> X_train = std_scale.transform(X_train)
>>> X_test = std_scale.transform(X_test)
>>> # Fit the MRC model
>>> clf = MRC().fit(X_train, Y_train)
>>> # Prediction. The predicted values for the first 10 test instances are:
>>> clf.pre (X_test[:10, :])
[1 0 0 0 0 1 0 1 0 0]
>>> # Predicted probabilities.
>>> # The predicted probabilities for the first 10 test instances are:
>>> clf.predict_proba(X_test[:10, :])
[[2.80350905e-01 7.19649095e-01]
[9.99996406e-01 3.59370941e-06]
[8.78592959e-01 1.21407041e-01]
[8.78593719e-01 1.21406281e-01]
[8.78595619e-01 1.21404381e-01]
[1.58950511e-01 8.41049489e-01]
[9.99997060e-01 2.94047920e-06]
[4.01753510e-01 5.98246490e-01]
[8.78595322e-01 1.21404678e-01]
[6.35793570e-01 3.64206430e-01]]
>>> # Calculate the score of the predictor
>>> # (mean accuracy on the given test data and labels)
>>> clf.score(X_test, Y_test)
0.7731958762886598

Attributes

is_fitted_bool

Whether the classifier is fitted i.e., the parameters are learnt or not.

tau_array-like of shape (n_features) or float

Mean estimates for the expectations of feature mappings.

lambda_array-like of shape (n_features) or float

Variance in the mean estimates for the expectations of the feature mappings.

mu_array-like of shape (n_features) or float

Parameters learnt by the optimization.

nu_float

Parameter learnt by the optimization.

mu_l_array-like of shape (n_features) or float

Parameters learnt by solving the lower bound optimization of MRC.

upper_float

Optimized upper bound of the MRC classifier.

lower_float

Optimized lower bound of the MRC classifier.

upper_params_dict

Dictionary that stores the optimal points and best value for the upper bound of the function.

params_dict

Dictionary that stores the optimal points and best value for the lower bound of the function.

Methods

compute_lambda(X, Y)	Compute deviation in the mean estimate tau using the given training instances.
compute_phi(X)	Compute the feature mapping corresponding to instances given for learning the classifiers (in case of training) and prediction (in case of testing).
compute_tau(X, Y)	Compute mean estimate tau using the given training instances.
error(X, Y)	Return the mean error obtained for the given test data and labels.
fit(X, Y[, X_])	Fit the MRC model.
get_lower_bound()	Obtains the lower bound on the expected loss for the fitted classifier.
get_params([deep])	Get parameters for this estimator.
get_upper_bound()	Returns the upper bound on the expected loss for the fitted classifier.
minimax_risk(X, tau_, lambda_, n_classes)	Solves the minimax risk problem for different types of loss (0-1 and log loss).
predict(X)	Predicts classes for new instances using a fitted model.
predict_proba(X)	Conditional probabilities corresponding to each class for each unlabeled input instance
score(X, y[, sample_weight])	Return the mean accuracy on the given test data and labels.
set_params(**params)	Set the parameters of this estimator.

__init__(loss='0-1', s=0.3, deterministic=True, random_state=None, fit_intercept=True, solver='subgrad', max_iters=10000, n_max=100, k_max=20, eps=0.0001, phi='linear', **phi_kwargs)

compute_lambda(X, Y)

Compute deviation in the mean estimate tau using the given training instances.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Training instances used for solving the minimax risk optimization problem.

Yarray-like of shape (n_samples, 1), default = None

Labels corresponding to the training instances used only to compute the expectation estimates.

compute_phi(X)

Compute the feature mapping corresponding to instances given for learning the classifiers (in case of training) and prediction (in case of testing).

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Instances to be converted to features.

compute_tau(X, Y)

Compute mean estimate tau using the given training instances.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Training instances used for solving the minimax risk optimization problem.

Yarray-like of shape (n_samples, 1), default = None

Labels corresponding to the training instances used only to compute the expectation estimates.

error(X, Y)

Return the mean error obtained for the given test data and labels.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Test instances for which the labels are to be predicted by the MRC model.

Yarray-like of shape (n_samples, 1), default = None

Labels corresponding to the testing instances used to compute the error in the prediction.

Returns

errorfloat

Mean error of the learned MRC classifier

fit(X, Y, X_=None)

Fit the MRC model.

Computes the parameters required for the minimax risk optimization and then calls the minimax_risk function to solve the optimization.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Training instances used in

• Calculating the expectation estimates that constrain the uncertainty set for the minimax risk classification

• Solving the minimax risk optimization problem.

n_samples is the number of training samples and n_dimensions is the number of features.

Yarray-like of shape (n_samples, 1), default = None

Labels corresponding to the training instances used only to compute the expectation estimates.

X_array-like of shape (n_samples2, n_dimensions), default = None

These instances are optional and when given, will be used in the minimax risk optimization. These extra instances are generally a smaller set and give an advantage in training time.

Returns

self

Fitted estimator

get_lower_bound()

Obtains the lower bound on the expected loss for the fitted classifier.

Returns

lower_boundfloat

Lower bound of the error for the fitted classifier.

get_params(deep=True)

Get parameters for this estimator.

Parameters

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns

paramsdict

Parameter names mapped to their values.

get_upper_bound()

Returns the upper bound on the expected loss for the fitted classifier.

Returns

upper_boundfloat

Upper bound of the expected loss for the fitted classifier.

minimax_risk(X, tau_, lambda_, n_classes)

Solves the minimax risk problem for different types of loss (0-1 and log loss). The solution of the default MRC optimization gives the upper bound of the error.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Training instances used for solving the minimax risk optimization problem.

tau_array-like of shape (n_features * n_classes)

Mean estimates for the expectations of feature mappings.

lambda_array-like of shape (n_features * n_classes)

Variance in the mean estimates for the expectations of the feature mappings.

n_classesint

Number of labels in the dataset.

Returns

self

Fitted estimator

predict(X)

Predicts classes for new instances using a fitted model.

Returns the predicted classes for the given instances in X using the probabilities given by the function predict_proba.

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Test instances for which the labels are to be predicted by the MRC model.

Returns

y_predarray-like of shape (n_samples)

Predicted labels corresponding to the given instances.

predict_proba(X)

Conditional probabilities corresponding to each class for each unlabeled input instance

Parameters

Xarray-like of shape (n_samples, n_dimensions)

Testing instances for which the prediction probabilities are calculated for each class.

Returns

hy_xndarray of shape (n_samples, n_classes)

Probabilities \((p(y|x))\) corresponding to the predictions for each class.

score(X, y, sample_weight=None)

Return the mean accuracy on the given test data and labels.

In multi-label classification, this is the subset accuracy which is a harsh metric since you require for each sample that each label set be correctly predicted.

Parameters

Xarray-like of shape (n_samples, n_features)

Test samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs)

True labels for X.

sample_weightarray-like of shape (n_samples,), default=None

Sample weights.

Returns

scorefloat

Mean accuracy of self.predict(X) w.r.t. y.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters

**paramsdict

Estimator parameters.

Returns

selfestimator instance

Estimator instance.

Examples using MRCpy.MRC

Example: Predicting COVID-19 patients outcome using MRCs

Example: Predicting COVID-19 patients outcome using MRCs