HPOBench (0.0.10) + scikit-learn (1.5.2)
HPOBench set metrics with default:
# hpobench/dependencies/ml/ml_benchmark_template.py
metrics = dict(acc=accuracy_score,bal_acc=balanced_accuracy_score,f1=f1_score,precision=precision_score,
)metrics_kwargs = dict(acc=dict(),bal_acc=dict(),f1=dict(average="macro", zero_division=0),precision=dict(average="macro", zero_division=0),
)
# set metrics with defalut param
self.scorers = dict()
for k, v in metrics.items():self.scorers[k] = make_scorer(v, **metrics_kwargs[k])
as for f1_score and precision_score, pos label is set to1 as defalut:
# sklearn\metrics\_classification.py
def f1_score(y_true,y_pred,*,labels=None,pos_label=1, # look hereaverage="binary",sample_weight=None,zero_division="warn",
)
...
def precision_score(y_true,y_pred,*,labels=None,pos_label=1, # look hereaverage="binary",sample_weight=None,zero_division="warn",
):
...
在实验中，验证(validation)步骤，计算y_pred 时（此时不涉及具体指标的计算）：
when call _check_set_wise_labels, raise ValueError.
Note that present_labels = [‘1’,‘2’] determined by dataset.
# sklearn/metrics/_scorer.py:372
def _score(self, method_caller, estimator, X, y_true, **kwargs):...y_pred = method_caller(estimator, response_method.__name__, X, pos_label=pos_label)...# sklearn/metrics/_scorer.py:89
def _cached_call(cache, estimator, response_method, *args, **kwargs):...result, _ = _get_response_values(estimator, *args, response_method=response_method, **kwargs)# sklearn/utils/_response.py:113
def _get_response_values(estimator,X,response_method,pos_label=None,return_response_method_used=False,
):...if is_classifier(estimator):prediction_method = _check_response_method(estimator, response_method)classes = estimator.classes_target_type = type_of_target(classes)if target_type in ("binary", "multiclass"):if pos_label is not None and pos_label not in classes.tolist():raise ValueError(f"pos_label={pos_label} is not a valid label: It should be "f"one of {classes}")elif pos_label is None and target_type == "binary":pos_label = classes[-1]y_pred = prediction_method(X)
理论上，f1_score / precision_score 的参数说明中已经写清楚：
pos_label : int, float, bool or str, default=1The class to report if `average='binary'` and the data is binary,otherwise this parameter is ignored.
在设置average = “macro”时，pos_label 本应该被忽略，然而_get_response_values方法并没有很好地处理这一点。
Quickly skip value error:
metrics_kwargs = dict(acc=dict(),bal_acc=dict(),f1=dict(average="macro", zero_division=0, pos_label=None),precision=dict(average="macro", zero_division=0, pos_label=None),
)