Depth Extraction from Video Using Non-parametric Sampling



Kevin Karsch, Ce Liu, Sing Bing Kang
Presented at ECCV 2012

[Code and dataset now available]




Several fames from two test videos (top) and our automatic depth estimate result (bottom).


Abstract

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large dataset containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.


Paper


[Download pdf]


Supplemental material

Supplemental PDF

[Download pdf]

Supplemental Video



Comparison to Youtube 2D-to-3D conversion
This comparison shows results converted from 2D into 3D (mono to stereo) using our automatic stereo synthesis technique (top) alongside Youtube's 2D-to-3D result (bottom).
Some 3D viewing mechanism is required for proper viewing (anaglyph glasses, polarized display/glasses, etc).

Our 2D-to-3D conversion (Depth Transfer + DIBR)


Youtube's 2D-to-3D conversion


IMPORTANT! You must do the following to view the Youtube result (bottom video) in 3D:
  1. Click the play button
  2. Click the gear near the bottom right of the video ("Change quality")
  3. Click the "3D" button that has now appeared near the gear
  4. To change the 3D visualization type (i.e. anaglyph to interleaved), click "3D" again


Source code

A MATLAB implementation of our Depth Transfer algorithm is available here:

[Download source code]



This code can be used in conjunction with our RGBD dataset (below) to compute depth for your own images and videos.



MSR-V3D (Microsoft Research Stereo Video + Depth) Dataset

Using side-by-side Microsoft Kinects, we collected a dataset of stereo videos for a variety of scenes. For one of the views, depth is available via the Kinect (the second view's depth was not recorded to avoid IR interference). This dataset can be used as training input for our Depth Transfer code, but might also be useful in other contexts (e.g. stereo matching in the wild). Our dataset primarily contains indoor scenes, but we have also captured some stereo videos outdoors for evaluation purposes (however, depth is unavailable for outdoor scenes due to limitations of the Kinect).

[Download MSR-V3D dataset]



Presentation slides


[Download pptx]
 [Download pdf without videos]


BibTeX

@inproceedings{Karsch:ECCV12,
 author     = {Kevin Karsch and Ce Liu and Sing Bing Kang},
 title      = {Depth Extraction from Video Using Non-parametric Sampling},
 booktitle  = {ECCV},
 year       = {2012}
}