Many seemingly static scenes contain subtle changes that are invisible to the naked human eye. However, it is possible to pull out these small changes from videos through the use of algorithms we have developed. We give a way to visualize these small changes by amplifying them and we present algorithms to pull out interesting signals from these videos, such as the human pulse, sound from vibrating objects and the motion of hot air.



Videos

New video technology that reveals an object's hidden properties

Abe Davis @ TED, March 2015 See invisible motion, hear silent sounds

Michael Rubinstein @ TEDxBeaconStreet, Nov 2014 Revealing Invisible Changes In The World

NSF Science and Engineering Visualization Challenge 2012 Visual Vibrometry: Estimating Material Properties from Small Motions in Video CVPR'15 Supplemental Video

Finding the Visible in the Invisible

Story in NY Times, Feb 2013 Eulerian Video Magnification

SIGGRAPH'12 Supplemental Video Detecting Pulse from Head Motions in Videos

CVPR'13 Supplemental Video Phase-based Video Motion Processing

SIGGRAPH'13 Supplemental Video The Visual Microphone: Passive Recovery of Sound from Video

SIGGRAPH'14 Supplemental Video Amplifying Tiny Movements to Visualize the Invisible

Reuters Video, Jan 2015

Software and Code

Eulerian Video Magnification code Matlab code and executables implementing Eulerian video processing for amplifying color and motion changes. Phase Based Video Motion Processing code Matlab code implementing the new and improved phase-based motion magnification pipeline. Learning-based Video Motion Magnification code Tensorflow implementation of the learning-based motion magnification pipeline. Videoscope Web interface for motion and color magnification. Upload your videos and have them magnified!

Publications (Magnifying Motion and Color Changes)

Tae-Hyun Oh*, Ronnachai Jaroensri*, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik

Learning-based Video Motion Magnification

European Conference on Computer Vision (ECCV), 2018

[Paper] [Webpage]

A learning-based approach to motion magnification with reduced artifact and better noise handling. Mohamed A. Elgharib, Mohamed Hefeeda, Frédo Durand, William T. Freeman

Video Magnification in Presence of Large Motions

IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015

[Paper] [Webpage]

A new video technique to motion magnify interesting small motions that are combined with large motions. Frédo Durand, William T. Freeman, Michael Rubinstein

A World of Movement

Scientific American, Volume 312, Number 1, January 2015

[Article in SciAm] [Videos]

An expository article describing several of the motion magnification techniques and applications we have worked on. Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman

Riesz Pyramids for Fast Phase-Based Video Magnification

Computational Photography (ICCP), 2014 IEEE International Conference on

[Paper] [Webpage]

Provides the quality of the previous phase-based technique with the real-time speed of the original linear technique. Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman

Phase-based Video Motion Processing

ACM Transactions on Graphics, Volume 32, Number 4 (Proc. SIGGRAPH), 2013.

[Paper] [Webpage] [BibTeX]

A new technique to amplify small motions that solves the noise amplification and intensity clipping artifacts of the previous linear method by manipulating the phase in sub-bands of videos. Michael Rubinstein, Neal Wadhwa, Frédo Durand, William T. Freeman

Revealing Invisible Changes In The World

Science Vol. 339 No. 6119, Feb 1 2013

[Article in Science] [Video] [NSF SciVis 2012] [BibTeX]

An expository video showcasing our results and explaining our technique. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman

Eulerian Video Magnification for Revealing Subtle Changes in the World

ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH), 2012

[Paper] [Webpage] [BibTeX]

The first Eulerian method to amplify small motions and color variations in videos. Ce Liu, Antonio Torralba, William T. Freeman, Frédo Durand, Edward H. Adelson

Motion Magnification

ACM Transactions on Graphics, Volume 24, Number 3 (Proc. SIGGRAPH), 2005

[Paper] [Webpage]

The original Lagrangian method to amplify small motions by explicitly estimating them and then warping the frame by amplified motion amounts.

Publications (Analysis of Small Motions)

Abe Davis, Katherine L. Bouman, Justin G. Chen, Michael Rubinstein, Frédo Durand, William T. Freeman

Visual Vibrometry: Estimating Material Properties from Small Motions in Video

IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015

[Paper] [Webpage]

A method to estimate the material properties of from small motions in videos. Tianfan Xue, Michael Rubinstein, Neal Wadhwa, Anat Levin, Frédo Durand, William T. Freeman

Refraction Wiggles for Measuring Fluid Depth and Velocity from Video

Proc. of European Conference on Computer Vision (ECCV), 2014.

[Paper] [Webpage]

A method to recover the velocity and depth of hot air from video. Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Frédo Durand, William T. Freeman

The Visual Microphone: Passive Recovery of Sound from Video

ACM Transactions on Graphics, Volume 33, Number 4 (Proc. SIGGRAPH), 2014.

[Paper] [Webpage]

A technique to recover sound from videos of objects subtly vibrating in response to sound. Justin G. Chen, Neal Wadhwa, Young-Jin Cha, Frédo Durand, William T. Freeman, Oral Buyukozturk

Structural Modal Identification through High Speed Camera Video: Motion Magnification

Proceedings of the 32nd International Modal Analysis Conference (2014)

[Paper]

A validation that the motion magnified motions are indeed real and a way to compute the mode shapes of a cantilevered beam from video. Guha Balakrishnan, Frédo Durand, John Guttag

Detecting Pulse from Head Motions in Video

Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on

[Paper] [Video] A method to measure pulse from the Newtonian motion of the head as blood flows into it.

People

Faculty:

Frédo Durand William T. Freeman

Students, Postdocs and Affiliates:

Guha Balakrishnan Katherine L. Bouman Justin G. Chen Abe Davis Hossein Mobahi Michael Rubinstein Neal Wadhwa Hao-Yu Wu Tianfan Xue Ronnachai 'Tiam' Jaroensri

Collaborators: Edward H. Adelson

Related Work and Other Applications

Visible Imaging of Global MHD on MAST

Ryan, D. A.

Plasma Science, IEEE Transactions on , vol.42, no.10, pp.2556,2557, Oct. 2014

[Paper] Video magnification applied to plasma physics. Auto Localization and Segmentation of Occluded Vessels in Robot-Assisted Partial Nephrectomy. Amir-Khalili, A., Peyrat, J. M., Abinahed, J., Al-Alao, O., Al-Ansari, A., Hamarneh, G., and Abugharbieh, R.

In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2014 (pp. 407-414). Springer International Publishing.

[Paper] The temporal frequency of subtle vibrations is used to segment blood vessels in medical imagery.

Talks