Clarification on delay Function Calculation

Pit

Hello TuringBot Support Team,

I am seeking clarification on the delay function in TuringBot, as I have encountered discrepancies between manual calculations and the software's output. Here is a detailed example illustrating the issue:

Example:
vg_0 vg_1 vg_2 vg_3 vg_4 vg_5 vg_6 target row
...
5 31 3 1 5 0 5 2 1603
2 4 0 4 6 0 6 8 1604
5 7 0 7 6 0 6 6 1605
2 11 1 1 6 0 6 4 1606
5 14 1 4 6 0 6 6 1607
2 18 1 8 6 0 6 1 1608
2 25 2 5 6 0 6 4 1609
5 28 2 8 6 0 6 1 1610
2 2 0 2 7 0 7 7 1611
5 5 0 5 7 0 7 1 1612
2 9 0 9 7 0 7 9 1613
5 12 1 2 7 0 7 8 1614
2 16 1 6 7 0 7 9 1615
5 19 1 9 7 0 7 9 1616
2 23 2 3 7 0 7 5 1617
5 26 2 6 7 0 7 1 1618
2 30 3 0 7 0 7 1 1619

Calculation Functions:

Size Error CV_Error Function
1 1.71349 1.67078 5
6 1.67702 1.73258 delay(vg_6,4)+4
10 1.64133 1.61073 5-vg_0/(30+(-7)vg_1)
20 1.60751 1.65307 5-((-2)+delay(vg_6,1)-delay(vg_2,7))/(30+(-7)vg_1)

Data and Predictions:

vg_0 vg_1 vg_2 vg_3 vg_4 vg_5 vg_6 target row solution_1 solution_6 solution_10 solution_20
...
5 31 3 1 5 0 5 2 1603 5 9 5.026737967914438 4.978609625668449
2 4 0 4 6 0 6 8 1604 5 9 4 5
5 7 0 7 6 0 6 6 1605 5 6 5.263157894736842 5.052631578947368
2 11 1 1 6 0 6 4 1606 5 4 5.042553191489362 5.063829787234043
5 14 1 4 6 0 6 6 1607 5 9 5.073529411764706 4.970588235294118
2 18 1 8 6 0 6 1 1608 5 6 5.020833333333333 4.989583333333333
2 25 2 5 6 0 6 4 1609 5 9 5.013793103448275 4.972413793103448
5 28 2 8 6 0 6 1 1610 5 6 5.030120481927711 4.969879518072289
2 2 0 2 7 0 7 7 1611 5 6 4.875 5.25
5 5 0 5 7 0 7 1 1612 5 9 6 5.6
2 9 0 9 7 0 7 9 1613 5 9 5.060606060606061 4.909090909090909
5 12 1 2 7 0 7 8 1614 5 4 5.092592592592593 5.037037037037037
2 16 1 6 7 0 7 9 1615 5 4 5.024390243902439 4.987804878048781
5 19 1 9 7 0 7 9 1616 5 4 5.048543689320389 4.932038834951456
2 23 2 3 7 0 7 5 1617 5 9 5.015267175572519 5.015267175572519
5 26 2 6 7 0 7 1 1618 5 4 5.032894736842105 5
2 30 3 0 7 0 7 1 1619 5 6 5.011111111111111 4.961111111111111

Issue:
Solution_6 = delay(vg_6,4) + 4:
Expected value for the last row should be 7 + 4 = 11, but TuringBot shows 6.

Solution 10:
The calculations appear to be correct.

Solution 20:
Discrepancies from manual calculations appear randomly.

Questions:
How exactly is the delay function calculated in TuringBot?
What algorithm or method is used?
Are there any settings that could affect the output ?

Additional Details:
Input: Integers - six columns of variables.
Targets: Integers.
Using only integer constants, delay function, addition, multiplication, and division.
Version: TuringBot 2.19.
Search metrics: Hybrid (CC_RMS).
Train/test split: 80/20, with the test sample being the last points.
Without cross validation problem still exists.
Understanding these details will help me determine if there is an issue with my usage or a potential bug.

Thank you for your assistance.

admin

Thanks for the detailed report. Indeed there was an error in how the delay and moving average functions were being printed when the search contained more than 1 input variable. This has been resolved in version 2.20.

Pit

Hello,

Thank you for your response and for addressing the issue in version 2.20. However, I would like to inform you that the problem persists even after updating to the latest version. I have tested it on various datasets, and the discrepancies in the delay function calculations continue to occur.

I am currently running TuringBot on Windows 10, so this issue might be specific to this environment. Could you please look into this further? If there's any additional information or logs I can provide to assist in diagnosing the issue, please let me know.

Best Regards

admin

Hello @Pit, following our discussion by email, I have made a test with your data by running a search with delay enabled and comparing the errors reported by TuringBot with a recalculation using Python. The errors matched perfectly.

I think that the confusion may caused by the usage of cross-validation with random rows in your search. By disabling cross-validation, you should also be able to make validate the numbers.

I hope that resolves the issue!

Pit

Hello,

Thank you for your response. However, I would like to clarify that the issue I'm encountering is purely a mathematical mismatch between the provided results and the expected outcomes based on the function 5 - delay(vg_5, 17).

The discrepancy persists regardless of whether I use cross-validation with random rows, any other setting (e.g., "Test sample - Chosen randomly" or "The last point"), or if I do not use cross-validation at all. The mismatch is consistent across multiple configurations and has been verified on three different computers running Windows 10.

It appears that certain columns in the solutions_6 output are incorrect, though not all of them. This inconsistency suggests the issue might not be related to cross-validation or the algorithm itself but rather to the output generation.

Additionally, I noticed that in the results for delay(vg_6, 4), TuringBot returns values even for the first 4 rows, where, according to the logic of delay, the values should be NaN. Results should not be generated where there is no previous data available. As such, for the first 4 rows, the function should return NaN, which introduces further inconsistencies.

It’s easy to manually calculate these simple discrepancies, but I have also verified them using Python, which further reveals other inconsistencies.

For convenience, here are the links to the data files so that someone can verify the issue on their own Windows machine:

test.csv https://drive.google.com/file/d/1vU_OPjTGMDU4LXgJDigG1NuCzQB6rO53/view?usp=drive_link

solutions.txt https://drive.google.com/file/d/1YYJGbIN5R6A9vblyk1FVmSaz5Vbe5Xgh/view?usp=drive_link

data_and_predictions.txt https://drive.google.com/file/d/1Rp8Jbe33Ph2HH6sSeXs8UkmWiopYN349/view?usp=drive_link

settings_for_test.cfg https://drive.google.com/file/d/15PrhYiex_XBfI-rWjRo2u8MOOVgkdECh/view?usp=drive_link
Thanks

admin

Okay, I see it now!

The delay function used to be outputted wrong when the search contained more than 1 input variable. This was a clear bug that was fixed in version 2.20, and now the errors reported by the interface or in a solutions.txt export match a recalculation using Python exactly. It was possible to verify this using your test.csv above.
The predictions in the file created by "Export data+predictions" are indeed incorrect when the solution uses delay or moving_average. Additionally, the file includes rows that shouldn't be there, as you discovered. These predictions don't match those used in the search, and the extra rows aren't utilized in the search process either. This issue is due to a bug in the function that generates the "Export data+predictions" output itself. It will be fixed in the upcoming release 2.21 next week.

Pit

Thank you for the thorough explanation. I eagerly await the upcoming update and will recheck the results once it’s available.

Pit

Small correction: The previous link didn't work. The proper link to settings_for_test should now work. The metrics used here are Hybrid CC+RMS.
https://drive.google.com/file/d/1yMCG9s86ppdas0YcLfXMJjNFKAhJxAMr/view?usp=sharing

admin

v2.21 has been released including a fix for the issue with "Export data+predictions" when the solution includes delay or moving_average:

https://turingbotsoftware.com/changelog.html#v2.21

Pit

It seems that the issue with the delay function has been resolved with the release of version 2.21. Thank you for addressing the bug related to the "Export data+predictions" function when solutions involved delay or moving_average. Thanks