User Guide for Dual Depth Sensor Configuration

Revision as of 18:24, 9 December 2011

System Requirements

• Computer (desktop or laptop):
  • CPU: x86 compatible (Intel Pentium 4 or higher, AMD Athlon or higher), dual- or quad- core is preferable
  • Operating system: Windows 7 (x86 or x64), Windows Vista (x86 or x64), Windows XP SP3 (x86 or x64)
  • Video card: Direct3D 10-capable (Shader Model 4.0) gaming-class graphics card
    • for more info see Cameras and accessories#Video_Card
  • USB: at least two USB 2.0 or USB 3.0 controllers
    • for more info see USB controllers
  • Storage system: HDD or SSD or RAID with write speed at least 55 Mb/sec
• Two Microsoft Kinect Sensors with separate power adapters
  • If you bought "XBox 360 + Kinect" bundle then you need to order power adapter
• Optional: active USB 2.0 extension cables
• Optional: Kinect Adjustable Tripod Stands
• Minimum required space: 3m by 3m (10 by 10 feet)

Note that before you start working with two Kinects it's highly recommended to get appropriate results with single Kinect solution: Quick Start Guide with MS Kinect Sensor.

Software Installation

iPi Recorder

Download and run the setup package of the latest version of iPi Recorder. You will be presented with the following dialog.

1. Select needed components
2. Read and accept the license agreement by checking appropriate checkbox
3. Press the Install button to begin installation

Note. Most of the components require administrative privileges because they install device drivers or write to Program Files and other system folders. On Windows Vista/7 you will be presented with UAC prompts when appropriate during installation. If you plan to use iPi Recorder under user which has no administrative rights, you can pre-install other components separately using administrator's account.

Important!

1. You can plug only one MS Kinect / ASUS Xtion / PrimeSense Carmine sensor to one USB controller. 1 USB controller bandwidth is not enough to record from 2 sensors.
2. You can plug not more than 2 Sony PS Eye cameras to one USB controller, otherwise you will not be able to capture at 60 fps with 640 x 480 resolution.

For more info see USB controllers.

Upon installation is complete, iPi Recorder will launch automatically. Continue with user's guide to get a knowledge of using the software.

Components

If some of the components is already installed, it has no checkbox and is marked with ALREADY INSTALLED label. You should not install all optional components in advance, without necessity. All of them can be installed separately at later time. Components descriptions below contain corresponding download links.

• Microsoft .NET Framework 4 - Client. This is required component and cannot be unchecked.
  This is basic infrastructure for running .NET programs. iPi Recorder is a .NET program.
  • Web installer: http://www.microsoft.com/en-us/download/details.aspx?id=17113
  • Standalone installer: http://www.microsoft.com/en-us/download/details.aspx?id=24872

• Playstation3 Eye Webcam :: WinUSB Drivers Registration. Check if you plan to work with Sony PS Eye cameras.
  Device drivers for PS Eye camera.
  • 64-bit OS: http://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x64.msi
  • 32-bit OS: http://files.ipisoft.com/drivers/PlayStation3_Eye_iPi-x86.msi

• ASUS Xtion / PrimeSense Carmine :: OpenNI Redistributable and ASUS Xtion / PrimeSense Carmine :: PrimeSense Sensor. Check if you plan to work with ASUS Xtion, or ASUS Xtion Live, or PrimeSense Carmine depth sensors.
  Device drivers and software libraries for ASUS Xtion / ASUS Xtion Live / PrimeSense Carmine.
  • 64-bit OS: http://files.ipisoft.com/3dparty/openni-win64-1.5.4.0-dev.msi, http://files.ipisoft.com/3dparty/sensor-win64-5.1.2.1-redist.msi
  • 32-bit OS: http://files.ipisoft.com/3dparty/openni-win32-1.5.4.0-dev.msi, http://files.ipisoft.com/3dparty/sensor-win32-5.1.2.1-redist.msi

• (Windows 7, 8) Microsoft Kinect :: MS Kinect SDK 1.8. Check if you plan to work with Microsoft Kinect depth sensors.
  Device drivers and software libraries for Microsoft Kinect. Requires Windows 7 and later.
  • http://www.microsoft.com/en-us/download/details.aspx?id=40278

• (Windows XP, Vista) Microsoft Kinect :: PrimeSense psdrv3.sys Driver Registration. Check if you plan to work with Microsoft Kinect depth sensors.
  Alternative device drivers for Microsoft Kinect.
  • 64-bit OS: http://files.ipisoft.com/drivers/KinectPsdrv3_iPi-x64.msi
  • 32-bit OS: http://files.ipisoft.com/drivers/KinectPsdrv3_iPi-x86.msi

  Note. iPi Recorder does support working with Kinect sensors on Windows 7 using PrimeSense driver. If you have installed and used it for Kinect with iPi Recorder 1.x, you can continue using it with iPi Recorder 2.

• iPi Recorder 2.x.x.x. This is required component and cannot be unchecked.
  iPi Recorder itself.

Important! Each Kinect Sensor must be plugged in through separate USB controllers. If both Kinects are plugged in through one USB controller then the second Kinect will not work. How it looks in the Device Manager:

Good		Bad
Both Kinects are plugged correctly		Both Kinects are plugged through one and the same USB controller

Run iPi Recorder and make sure that there are two items “MS Kinect Sensor (OpenNI)” in “Camera Selector” dialog:

Select both items “MS Kinect Sensor (OpenNI)” and press “Ok” button. As a result you should see something like this:

Make sure that you can see correct colored “depth video” from both sensors and

1. that you can change motor position for each sensor using appropriate slider and “Motor of” combobox
2. that you can switch between RGB and “depth” video using “View RGB/Depth” button
3. that actual frame rate is 30 fps (or 29.9) and frames drop is in range from 0 to 5 for each sensor

Note that values of Tilt and Roll are updated with a time lag. During recording these values cannot be updated assuming that in this case sensor should be static.

Recording video from two Depth Sensors

Environment

It is convenient to put each Kinect Sensor on a chair or a table. The picture below will help you to understand sensor field of view and possible distance from actor:

Side view	Top view

Below in #Calibration section you will find information on 2 recommended sensors' mutual configuratioins.

Actor Clothing

Current version uses only depth information to track motions. So clothes requirements are:

• no restrictions on clothes colors (just avoid shiny fabrics)
• please use slim clothes to reduce noise in resulted animation

Recording Process

Important! Each Kinect Sensor must be plugged in through separate USB controllers. If both Kinects are plugged in through one USB controller then the second Kinect will not work. How it looks in the Device Manager:

Good		Bad
Both Kinects are plugged correctly		Both Kinects are plugged through one and the same USB controller

Run iPi Recorder and mark both MS Kinect Sensors in the Camera Selector dialog:

Press “Ok” button. As a result you should see something like this:

Make sure that you can see correct colored “depth video” from both sensors and

1. that you can change motor position for each sensor using appropriate slider and “Motor of” combobox
2. that you can switch between RGB and “depth” video using “View RGB/Depth” button
3. that actual frame rate is 30 fps (or 29.9) and frames drop is in range from 0 to 5 for each sensor

Note that you can turn off RGB streaming using appropriate checkbox in the Advanced Options section of Camera Selector dialog. Also you can change the position of cameras output and motor controls in the interface in case they are positioned in the wrong order:

Please don't forget to set output directory for recorded video:

Calibration

There are two possible arrangements of the two sensors:

1. angle between sensors is between 60 and 90 degrees;
2. angle between sensors in near to 180 degrees that means that sensors are placed opposite to each other.

The first configuration

Top view	Side view

Preparations

For calibration purposes flat rectangular veneer/plywood/cardboard/pasteboard is used:

• horizontal size should be at least 0.5m (1m - 1.3m is recommended);
• vertical size should be at least 0.7m (1m - 1.5m is recommended);
• hold it vertically in front of yourself on outstretched arms;
• make sure that this calibration board is good visible in both sensors (the amount of yellow is low in the area of the veneer/plywood/cardboard/pasteboard).

Good		Bad
good size good depth data		too many yellow (no depth data) points too small too close to human body

Important! Yellow color in the depth map means "depth here is unknown". Thus it is important to minimize the amount of yellow points using appropriate materials and clothes.

Recording Calibration Video

After you finished preparations and sensors setup you're ready to record the calibration video:

• Start recording by pressing the "Start Video Recording" button.
• Initial two seconds of video should contain only background and static objects. Presence of actor or any moving objects in the initial two seconds of recorded video is unacceptable.
• Go to the capture area with prepared calibration plane (flat rectangular veneer/plywood/cardboard/pasteboard).
• Turn the veneer/plywood/cardboard/pasteboard to the left and to the right thus view angle is changing but the veneer/plywood/cardboard/pasteboard is still visible in both sensors.
• Stop the recording by pressing the "Stop Video Recording" button.

Note that is important to have variety in angles from one hand and good visibility of the calibration plane in each camera from the other hand. It is not so easy to reach good results for 90 degrees angle thus 60-80 degrees angles could be more practical.

Good		Bad
the calibration plane is good visible in both sensors		the calibration plane is not visible in the left sensor

Please open the captured video in the iPi Studio and check the following:

• At least two initial seconds of video contains only background and static objects (no actor, no calibration plane)
• The floor (ground) is good visible on the depth maps from both sensors
• The calibration plane is good visible on the depth maps most the time
• The calibration plane is moving and has at least 30 degrees variety in the horizontal view angle
• The amount of yellow points is relatively small

Processing Calibration Video

• Please make sure that the ground plane is determined correctly for both sensors after calibration video opening in the iPi Studio (to switch between sensors use buttons "Camera 1" and "Camera 2" in the top toolbar).

• After that it is convenient to turn off visualization of background points (View -> Hide Background).

• Set the beginning of Region of Interest (ROI) to the point where calibration plane is good visible in both sensors.

• Adjust the end of ROI:
  • the length of ROI should be at leat 5 seconds (150 frames)
  • ROI should contain good variety of horizontal view angles for the calibration plane

• Go to the Calibration tab, select "Angle between sensors is 30 - 90 degrees" option and click "Calibrate Based on 3D Plane" button

• Please wait for the end of calibration process

• Evaluate the result:
  • turn on "Show Depth From All Sensors" option in the "View" menu
  • rotate the scene and check how 3D-points from different sensors fit with each other

• Save the result:
  • go to the Scene tab
  • click "Save scene..." button
  • choose file name for your scene parameters

Video tutorial from TrueBones

The second configuration

Top view	Side view

Preparations

In this case flat rectangular veneer/plywood/cardboard/pasteboard is also used. But this box is held in one near-straight arm side by side with body. Each Kinect sensor sees different sides of board. Thus thickness of the board should not exceed 3 cm.

Recording Calibration Video

It is also important to have variety in angles from one hand and good visibility of the calibration plane in each camera from the other hand:

Good		Bad
the calibration plane is good visible in both sensors		the calibration plane is not visible in the left sensor

Processing Calibration Video

It's mostly similar to processing of the first configuration. The only difference is that before pressing "Calibrate Based on 3D Plane" button you should choose "Angle between sensors is 100 - 180 degrees" radio button.

Recording Actor's Performance

After completing Setup and Background recording steps, press “Record” button to begin video recording.

As soon as recorder starts, go to the capture area and stand in a T-pose:

After that you can act desired motions. If you make several takes of one actor, you do not need to record T-pose again.

To stop recording, press “Stop” button.

Performance Tips

Recommended layout of an action video

• Enter the actor.
• Strike a T-pose.
• Action

T-pose

It is preferable to have actor strike a “T-pose” before the actual action. The software will need T-pose for building actor appearance model during tracking. If you make several takes of one actor you do not need to re-record T-pose before each take.

When using the depth sensors, it is recommended to face the palms down, as it corresponds to the default orientation of the model's hand bones. When using color cameras, it is recommended to face the palms forward, as it helps the software in determining the right color for the model's hands.

Takes

Take is a concept originating from cinematography. In a nutshell, take is a single continuous recorded performance.

Usually it is a good idea to record multiple takes of the same motion, because a lot of things can go wrong for purely artistic reasons.

Iterations

A common problem with motion capture is “clipping” in resulting 3D character animation. For example, arms entering the body of animated computer-generated character. Many CG characters have various items and attachments like a bullet-proof vest, a fantasy armor or a helmet. It can be easy for an actor to forget about the shape of the CG model.

For this reason, you may need to schedule more than one motion capture session for the same motions. Recommended approach is:

• Record the videos
• Process the videos in iPiStudio
• Import your target character into iPiStudio and review the resulting animation
• Give feedback to the actor
• Schedule another motion capture session if needed

Ian Chisholm's hints on motion capture

Ian Chisholm is a machinima director and actor and the creator of critically acclaimed Clear Skies machinima series. Below are some hints from his motion capture guide based on his experience with motion capture for Clear Skies III.

Three handy hints for acting out mocap:

1. Don’t weave and bob around like you’re in a normal conversation – it looks terrible when finally onscreen. You need to be fairly (but not completely) static when acting.

1. If you are recording several lines in one go, make sure you have lead in and lead out between each one, i.e. stand still! Otherwise, the motions blend into each other and it’s hard to pick a start and end point for each take.

1. Stand a bit like a gorilla – have your arms out from your sides:
   
   Well, obviously not quite that much. But anyway, if you don’t, you’ll find the arms clip slightly into the models and they look daft.

If you have a lot of capture to do, you need to strike a balance between short and long recordings. Aim for 30 seconds to 2 minutes. Too long is a pain to work on later due to the fiddlyness of setting up takes, and too short means you are forever setting up T-poses.

Takes

Because motion capture is not a perfect art, and neither is acting, it’s best to perform multiple takes. I found that three was the best amount for most motion capture. Take less if it’s a basic move, take more if it’s complex and needs to be more accurate. It will make life easier for you in the processing stage if you signal the break between takes – I did this by reaching out one arm and holding up fingers to show which take it was.

Naming conventions

As it’s the same actor looking exactly the same each and every time, and there is no sound, and the capture is in lowres 320*200, you really need to name the files very clearly so that you later know which act, scene, character, and line(s) the capture is for.

My naming convention was based on act, scene, character, page number of the scene, line number, and take number. You end up with something unpleasant to read like A3S1_JR_P2_L41_t3 but it’s essential when you’ve got 1500 actions to record.

Processing Video from Two Depth Sensors

Processing of performance video is generally the same as for single Kinect solution: Quick Start Guide with MS Kinect Sensor#Processing Video from Depth Sensor.

The main difference is that after loading captured video into iPi Studio you should load your scene parameters:

• switch to the Scene tab
• click "Load scene..." button
• select file with scene parameters which has been saved during calibration process

That's all. After that you're ready for the processing:

• Position timeline slider to the frame where actor is in T-pose
• Adjust actor height using appropriate slider on tab “Actor”
• Select “Move” tool on toolbar.
• Move actor model to left or right to match roughly actor silhouette on video.
• etc.

Next Steps

• Clean-up
• Animation export and motion transfer

Materials

Video

• From Greenlaw:

• From argus1000:

• From TrueBones:

Users' Feedback

• Tentacle:

Brilliant work guys.
We just tested our setup with two Kinects at 90 degrees.
As imagined, most of the occlusion issues are solved.

• Greenlaw:

I'm seeing that dKS tracking is just slightly slower
than with single Kinect tracking, but the time difference is negligible.
With a single Kinect, I was getting about 0.67 sec per frame;
with dKS I'm seeing about 0.7 sec per frame.
I think that's very good for my graphics card.
(My graphics card is a Nvidia GTX 460,
which these days is probably considered a mid-range card.)

At a glance, the dKS tracking results appears to be more stable
than with a single Kinect.

• and one more:

Okay, I have several tracked dKS motions now and
I just finished applying iPi Studio's Configurable Jitter Removal
to a few clips. I have to say, this mocap looks incredibly solid!
The results are far superior to the data
I was getting using a single Kinect.

• TrueBones

I have successfully calibrated the 180 degree angle with iPiSoft.
It is great, it calibrates at 180 degrees producing
a full 360 Hologram of myself in iPiSoft.
The tracking is rock solid and the results are out of this world.
I am utterly astonished.
http://www.youtube.com/watch?v=WTyzFGJNGEo