This is now possible in TuringBot 4.0. Custom search metrics can use dataset columns in aggregations, including a column of observation weights.
Add a column with the weight of each observation (say w) and use one of these as the custom search metric:
- Weighted mean squared error (weighted least squares):
sum(w*pow(actual-predicted,2))/sum(w)
- Weighted RMS error:
sqrt(sum(w*pow(actual-predicted,2))/sum(w))
- Weighted mean absolute error:
sum(w*abs(actual-predicted))/sum(w)
- Weighted mean relative error:
sum(w*abs((actual-predicted)/actual))/sum(w)
- 1 − weighted R² (minimizing it maximizes the weighted R²):
sum(w*pow(actual-predicted,2))/sum(w*pow(actual-sum(w*actual)/sum(w),2))
For measurements with known uncertainties σ, the usual weights are w = 1/σ².
So that w is only used as a weight and not as an input of the formulas, use a custom formula that leaves it out, like y = f(x1, x2). The weights retain their original units even when "Normalize the dataset" is enabled. From the command line or the Python library, pass the same expression with --custom-metric or custom_metric=.
