Getting started with motion controllers
What's this for
Wanna capture hand/prop movements? Now this is possible with the support for motion controllers added to iPi Recorder. Essentially, motion controller provides one 3D orientation value which can be used to track rotations of a single bone, whether it is a body part or attached prop.
Now, what you need to try this. First, a motion controller (or several ones, iPi Recorder currently has no limitation on the number of controllers). Three models are supported, all of them are easy to buy:
- PlayStation Move
- Wii Remote with MotionPlus
- Wii Remote Plus, which has MotionPlus integrated
Important. If you don't already have Wii stuff, we strongly recommend to use PlayStation Move, because of its built-in magnetometer. In short, the presence of magnetometer gives a lot more stable orientation value. Without it, orientation tends to drift around vertical axis, and this leads to less accurate measurements.
Also, your PC should be equipped with a Bluetooth radio module to connect with motion controllers. If you do not have a built-in one, you'll need an USB Bluetooth adapter. You'll need only one Bluetooth adapter for any reasonable number of motion controllers (at least 4). For PS Move, you will definitely need an USB to mini-USB cable to charge your motion controller and for pairing (see next section).
Warning. On Windows 8 and 8.1, PS Move does not work properly using the standard Bluetooth drivers included with the OS. For these OS versions, it is recommended to install Motion in Joy drivers.
Warning. Some of Wii Remote Plus controllers may not work properly using standard Bluetooth drivers. For them, it is recommended to use Toshiba Bluetooth stack.
Tip. To connect more than 4 PS Move controllers, multiple Bluetooth adapters are usually needed. To use multiple Bluetooth adapters, it is necessary to install Motion in Joy drivers.
Getting into Recorder
To get your motion controller in the list of devices, it must be connected to a PC via Bluetooth. But before it can connect to your Bluetooth adapter, the controller must be paired with it. That is where button ADD NEW helps. So, your first step is to press this button, select device type you have, and then follow the instructions specific for this device.
For PS Move, pairing is one-time process unless you reconnect motion controller to another PC or gaming console. Each time you connect controller to another host, it should be paired. After pairing, the Bluetooth connection is established automatically each time you power on the motion controller. To power on PS Move, press the PS button. To power off, hold down the PS button for a long time (about 10 seconds).
For Wii Remote, pairing should be done each time after Bluetooth connection is lost, powering off either the motion controller or the PC. This is another disadvantage of Wii Remote. To power off Wii Remote, hold down the Power button for a couple of seconds.
After you have successfully paired the controller with Bluetooth adapter, it should be visible in device list. Mark it selected and go to Setup stage. Here for the first time you will see the controller visualized as a rotating blue "brick". And also you should see a message over it saying it is not calibrated. Calibration of a motion controller (not confuse with calibration of cameras) is needed for more accurate measurements, so it is highly recommended to get through it every time you see this message. To run calibration process, press CALIBRATION button in the on-screen controls.
After a motion controller is calibrated, you can use it in recording action videos.
Tip. You can use motion controller to start/stop recording (including background evaluation). Just hold PS Move's Start button, or Wiimote's A button for a few seconds, until you feel a vibration.
Tip. There is no need to enable motion controllers when recording calibration videos, because their data will not be used in the calibration process.
Tip. Under certain conditions (generally indoors with dim lighting), PS Move's light sphere can be used as a light marker for calibration. Here is short instruction:
- Ensure PS Move is visible in device list.
- Unselect it, when going to Setup stage.
- To change the color of the sphere, press button combination on the controller:
- Select + Move — White
- Select + Cross — Blue
- Select + Square — Magenta
- Select + Triangle — Green
- Select + Circle — Red
- Select + T (Trigger) — Cyan
- Select + Start — Yellow
- Select + PS — Off
- Proceed as usual with recording a calibration video. Do not change the color of the sphere during recording.
Tips on action video
There are two main requirements:
- At some moment during the record (for example, at the beginning), you should aim with (each) motion controller at one of the cameras. This is needed to correctly orient the motion controller within the scene coordinate space in iPi Mocap Studio.
- For correct orientation of prop models in iPi Mocap Studio, hold the controller as if its side with buttons is a back side of a prop in a "standard" position — i.e. facing the actor's face when the arm with prop is stretched forward. Of course, you can use a non-"standard" grip of the controller if such a grip for a prop is implied by the scenario of the action.
Now, the tips:
- If you have already made some records without explicitly aiming at the camera, these videos may not be going to trash. Look thoroughly through all the frames — it is very likely that at some point the actor aimed at some camera by accident, at least in horizontal direction (this is sufficient).
- You can use special accessories for motion controllers like shooting attachments. In that case, the relative position of the controller to the prop is good known, so you just hold it as needed by the scenario (i.e. the requirement #2 is not relevant for this case).
- You can change the grip of the motion controller during the action. The iPi Mocap Studio allows you to process every frame range with the same grip independently.
iPi Mocap Studio has got a new tab called Motion Controllers. It is visible only when open project contains data from one or more motion controllers. Dealing with this data makes sense only after the tracking of the body is complete. In case of several controllers, data from each of the controllers is processed separately in any convenient order. Numbering of controllers correspond to their order in iPi Recorder when a video was recorded.
When a project with motion data is first created, you will see this data as one or more colored "bricks" floating somewhere under the actor's model. To give more sense to this picture, for each controller you should specify a bone it is attached to. See the Attach to combobox. Currently it contains only LHand and RHand bones. This list will be extended as the support for other motion controller scenarios is implemented. When attached, the controller's "brick" is displayed near the corresponding bone of actor's model.
For a nicer look, you have an option to replace the default brick with several prop models like sword and pistol. Currently, there are only few standard prop models. This list may be extended in the future, and the support for importing custom prop models is also planned. You can also change the main color of a prop model, and toggle the display of prop's coordinate axes.
To make some use of motion controller's data, it first needs to be matched with the scene coordinate space. Assuming that scene's ground plane is horizontal, controller's pitch and bank angles already correspond to the scene. However, heading angle of controller's orientation is measured from some "arbitrary" origin direction (in terms of the scene). For Playstation Move it's always the North direction, for Wii Remote — a direction it was oriented to when recording started. To match coordinates, knowing the scene coordinates for some reference orientation of the controller is needed. That is where aiming at a camera helps.
So you've done with the body tracking, and have attached the controller to a corresponding hand. Now find some frame where the actor is aiming with this controller at some camera (remember action video requirements?) Ensure the body position has been tracked in this frame. Select the target camera as the default, pressing corresponding toolbar button. Pressing the Match with Camera X button does matching. After matching, the prop model for the controller should be oriented perfectly as it was intended in the video. Unless the actor did not pay attention to a proper grip of the controller, especially regarding it's bank rotation.
You can also manually adjust position and orientation of the controller by using the Move tool. Activate the tool, click on the controller's model in the scene, and then move and rotate the model to correspond to a position and orientation of an actual controller in the video.
And finally — applying the motion controller's data to the hand's rotation. First, determine you region of interest. Ensure that within this region the actor maintain a fixed grip of the controller. Frames with different grip should be processed separately. Next, select some frame within the region where the hand's orientation may be adjusted manually to match the actor without much effort. Make these adjustments. When ready, just press the Apply Motion Data button — and in all frames within ROI, the hand's orientation will be set automatically.
Tutorials and demos
- A great sample of hand tracking from Seed Crystal Games studio:
- PS3 Move Gloves for Hand by CManson:
- DIY solutions for attaching PS Move to a head:
- Live video of pairing procedure for PS Move: