By Giovanni Di Maria, creator of EDM Electronics Design Master
A person’s movements can be represented using mathematical formulas. With the help of TuringBot, it is possible to translate into numbers all the movements that this person makes. If the movements are simple, the formulas will also be simple. Conversely, complex movements can be described by longer and more complex formulas and equations. Thanks to the ability to find mathematical formulas for any type of action, robotic systems, simulations, 3D prototyping, video games and much more can be implemented.
Turning real life into numbers and formulas
To describe the movement of a person (or an animal, plant, or object) on the computer, it is necessary to follow different phases and mathematically represent the set of actions that he performs. Some steps are quite complicated. This article is dedicated to the numerical representation of the movements of a man performing the “Jumping Jack” physical exercise. The steps to follow are as follows:
- video capture;
- cutting the video and keeping only the useful frames;
- identification of the parts of the body in motion;
- determination of the coordinates of the various segments;
- symbolic regression and curve fitting of data;
- application of the formulas and creation of a 3D model;
- creation of final animation and comparison with the original video.
Video capture and frame selection
For this step, you need to create a video of the person in motion. It need not be very long. 3 or 4 seconds is enough, even at 15 FPS. The important thing is to acquire the sequence of useful frames, in which the athlete moves for a complete period, ie from when the human body assumes a certain position to when it returns to the same position.
You probably need to use a variety of software. When creating the video, the important thing is that the athlete is well-lit and visible, and, above all, the camera angle must not change. You need to trim the video to take only the useful sequence. For this example, the video consists of 29 frames. The various elements, therefore, are represented by as many positions in the two-dimensional spatial domain.
Identification of the moving parts of the body
Our work aims to simulate on the computer the movement of a graphic object composed simply of some segments. We are interested in the mathematical aspect of the problem and not the graphic one. The digital athlete must be created using some segments that represent the arms, legs, etc. Below we can see the athlete together with his virtual “avatar”, made up of many segments which, subsequently, will be mathematically processed. For convenience, the forearm and hand have been treated as a single segment, since the video shows them both with a single line.
The virtual athlete is composed of the following elements:
- a circle “H” for the head, that is a circle with position X, Y, and radius R;
- by two segments “Arm1” and “Arm2” for the arms;
- by two segments “Forearm1” and “Forearm2” for the forearms;
- from the “Body” segment for the body;
- by two segments “Thigh1” and “Thigh2” for the thighs;
- by two segments “Leg1” and “Leg2” for the legs;
- by two segments “Foot1” and “Foot2” for the feet.
Determination of the coordinates of the various segments
The determination of all the coordinates of the points of the subject is the most important and tiring phase of the whole operation. A good result depends on the precision with which the point values were taken. For each frame you need:
- determine the X and Y coordinate of the start of a segment;
- determine the end X and Y coordinates of the same segment.
The head is drawn with a circle, so its center (and its position) is represented by a single point, with X and Y coordinates. To evaluate the position of each point we can use any graphics program. For instance, ImageJ shows in real time the position of the mouse (in pixels) on the status bar and has many excellent automated selection functions that simplify a lot of work. We use this information to obtain the coordinates of the various segments.
It is a bit of a long job, where maximum precision and a lot of patience are required. At the end of the calculations, it is possible to insert the results in a spreadsheet. Remember that these coordinates are relative. To transform them into absolute coordinates it is sufficient to subtract a fixed value.
With all this data available it is possible, for example, to draw the displacement graphs of some elements of the body, to discover the presence of any anomalies. For example, the following graph shows the trend of the position on the X-axis and the Y-axis, during the entire movement of all the frames.
And now, TuringBot in action
All these coordinates must be transformed into mathematical formulas, even if the simplest and safest way would be to insert them into an array and process the movements within a loop. But the adoption of equations and formulas makes the operation much more elegant and faster, as well as allowing the writing of a shorter and more compact source code, with less memory consumption. It is understood that if the athlete’s movements change, the formulas must also follow the same changes. It is therefore the right time to try to represent the trends of human movements with mathematical formulas. The various nodes of the human body are represented by many points, of which it is necessary to memorize the two coordinates X and Y. It is therefore necessary to generate a considerable amount of mathematical formulas that allow one to obtain a small arithmetic miracle. Some points are in common, so they use the same formulas.
For each coordinate of each point, it is necessary to create a text file for TuringBot, containing two columns. The data is, of course, taken from the giant spreadsheet. We show below the example for the X coordinate of the head, but the other files follow the same rule.
x y
1 190
2 190
3 190
4 189
5 189
6 188
7 188
8 187
9 186
10 186
11 185
12 184
13 184
14 184
15 184
16 185
17 185
18 186
19 186
20 187
21 188
22 188
23 188
24 189
25 190
26 190
27 191
28 192
29 192
The TuringBot setting is as follows:
- Search metric: RMS error;
- Train/test split: No cross-validation;
- Test sample: Chosen randomly;
- Integer constants only: Disabled;
- Bound search mode: Deactivated;
- Maximum formula complexity: 60;
- Allowed functions: + * / sin cos exp log sqrt abs floor ceil round
For some systems, care must be taken with trigonometric functions. They work in radians in TuringBot, while for other graphics software, they may work in degrees, complicating the conversion process. In our specific case, the following conversion functions were created in OpenSCAD:
function cos_deg (x) = cos (x * 180 / 3.1415);
function sin_deg (x) = sin (x * 180 / 3.1415);
A relatively low search time is sufficient for each formula, let’s say 5-10 minutes or less is more than enough. It is very interesting to observe TuringBot during the research. Its formulas are getting closer and closer to the final goal, as can be seen in the screen below.
Below there are all the formulas that describe the positions of all the coordinates of the segments that make up the virtual athlete.
Implementation of the model
After the creation of the formulas by TuringBot, it is finally possible to implement the athlete’s mathematical model. The figure below shows the use of OpenSCAD with the created equations.
The following short list shows all software used in the project. The user can use other software, according to their needs.
- video recorder of a smartphone or camera;
- Avidemux;
- ImageMagick (convert.exe);
- ImageJ;
- Calc (LibreOffice);
- and of course… TuringBot.
It is very interesting to observe the animation of the real and virtual subjects at the same time.
Conclusions
By symbolic regression, it is possible to convert any fact and action into a mathematical formula. These techniques are used in movies, video games, and 2D and 3D simulations. Behind a very short film, even lasting just one second, there is a lot of effort and dozens of hours of work. The adoption of mathematical formulas for the description of the points, in 2D and 3D space, gives a great touch of elegance to the project. Depending on the software used, it may be necessary to “overturn” the coordinates of the subject. The whole procedure described was very exciting and fun, even if very tiring. And the satisfaction was immense when after all the calculations made and the implementation of the code, the digital figure acquired a life of its own and started jumping. Let us remember once again, the athlete’s movement is not produced by coordinates stored in an array, but is created using mathematical formulas. All this massive processing would not have been possible without the use of TuringBot. This is math!
