shown in figure 7. The regressor takes in an image
and outputs two numbers, representing the angle of
both pendulums. Similar to the pendulum-tracking
experiments, the trajectories are constructed by the
outputs of the regressor across 10 continuous images.
We also provide a simulator of the joint dynamics of
both pendulums. The model is thus trained adversarially when the discriminator tries to distinguish
the outputs of the regressor and the simulated trajectories.
Our trained model achieves an average correlation
of 99. 2 percent between the predicted angles and the
ground truth angles for detecting both pendulums.
Note that the regressor will not converge to tracking
only one pendulum with both outputs. Although
such a situation may occur early in training, the dis-
criminator quickly learns to distinguish the correlat-
ed joint trajectories (if the regressor outputs two same
numbers) from the independent joint trajectories
(where the two numbers are independent), and the
adversarial loss forces the regressor to track both pen-
dulums.
Overall, the real-world pendulum experiment
shows that using adversarial constraint learning it is
possible to train a neural network to extract object
information from real images using only a simulator
of physics that the object obeys.
Pose Estimation
In this experiment, we benchmark the proposed
model on pose estimation, which has a larger output
space. We aim to learn a regression network, map-
Figure 6. Example Predictions on the Test Data.
Top: frames from video used in the pendulum experiment. Bottom: the network is trained to predict angles that cannot be distinguished
from the simulated dynamics, encouraging it to track the metal ball over time.
– 1. 5
- 1
–0.5
0
0.5
1
1. 5
1 6 11 16 21 26 31 36 41 46 51
Ground Truth Predictors
