Normalize the dataset

calimero22

Hi All
When I normalize the dataset, how can i use the formulas found by TuringBot ?
For example: my dataset is:
x y
1 5
2 10
3 15
4 21

Without normalization, i get the formula:
y=round(5.13396*x)
And by replacing "x" with 1, 2, 3, 4 i get again the y values: 5, 10, 15, 21. OK

If i Use the normalization, i get the formula:
y=x-(0.00707311/x)
By replacing "x" with 1, 2, 3, 4 i don't get the y values: 5, 10, 15, 21.

Thank you very much.
Giovanni Di Maria

admin

When the "normalize the dataset" option is checked, what is used in the optimization in place of x is

x -> (x - average) / (standard_deviation)

The averages and standard deviations for each input variable can be consulted Analysis tab.

For instance, if the average for x is 1.66702, the standard deviation is 2.76995, and the formula that was found was y=x-(0.00707311/x), you can write it in terms of the non-normalized variable as

y = (x*2.76995 + 1.66702) - 0.00707311/(x*2.76995 + 1.66702)

ljuillen

admin
shall it be called "standardisation" instead of "normalisation" ?
For me, "normalisation" should be x -> (x - minimum_x ) / maximum_x - minimum_x
Both would be very interesting to be available !

cf.
https://www.shiksha.com/online-courses/articles/normalization-and-standardization/
https://www.simplilearn.com/normalization-vs-standardization-article

tim

admin Doing this manually for more complex formula and for all input variables is not very practical. Would be good to add this functionality to the software. Concretely, if a user choses to use normalize_dataset setting, then software provides also the formula written in terms of non-normalized variables to be used against production data after the model is built. Thanks.

calimero22

Perfect.
Thank you
Giovanni

xntang88

Good to know this.

ljuillen

ljuillen
in some publications the name "scaling" is used to set the range of a variable x between x_min and x_max !!!