I noticed an issue with the current API design - the sign (pun intended).
Namely, experiments can be minimization or maximization, e.g., sklearn experiments with the accuracy score, vs the MSE; or more classical optimization.
Currently, the API implicitly assumes minimization everywhere. But this becomes confusing ot the user as the SklearnCvExperiment expects metrics, so it raises the expectation that it handles the sign.
It is unclear where to handle this and is left to the user - there are two natural places of which one or both could be used:
- the experiment - a tag
lower_is_better: bool or direction: {"max", "min", "neither"} - or an __init__ arg
- the optimizer, which could get an
__init__ arg, or an arg to run
It seems more natural to have the sign "minmize" or "maximize" as part of the optimization problem, hence the experiment.
My favourite solution is the tag, from a user perspective, although I wonder what this would imply about the optimizer. I am confident that we could assume minimization and let the boilerplate handle the sign, although this has to be thought through.
FYI @SimonBlanke
I noticed an issue with the current API design - the sign (pun intended).
Namely, experiments can be minimization or maximization, e.g.,
sklearnexperiments with the accuracy score, vs the MSE; or more classical optimization.Currently, the API implicitly assumes minimization everywhere. But this becomes confusing ot the user as the
SklearnCvExperimentexpects metrics, so it raises the expectation that it handles the sign.It is unclear where to handle this and is left to the user - there are two natural places of which one or both could be used:
lower_is_better: boolordirection: {"max", "min", "neither"}- or an__init__arg__init__arg, or an arg torunIt seems more natural to have the sign "minmize" or "maximize" as part of the optimization problem, hence the experiment.
My favourite solution is the tag, from a user perspective, although I wonder what this would imply about the optimizer. I am confident that we could assume minimization and let the boilerplate handle the sign, although this has to be thought through.
FYI @SimonBlanke