Eulerian Amplification

[Emerenciana Mcgreal]

Ourgoal is to reveal temporal variations in videos that are difficult or impossible to see with the naked eye and display them in an indicative manner. Our method, which we call Eulerian Video Magnification, takes a standard video sequence as input, and applies spatial decomposition, followed by temporal filtering to the frames. The resulting signal is then amplified to reveal hidden information. Using our method, we are able to visualize the flow of blood as it fills the face and also to amplify and reveal small motions. Our technique can run in real time to show phenomena occurring at temporal frequencies selected by the user.

- Executables for 64-bit Windows, 64-bit Linux and 64-bit Mac (v1.1, 2013-09-05)

This is a compiled version of the MATLAB code the can be run from the command line. It doesn't require any programming or for MATLAB to be installed. Instead, these binaries use the MATLAB Compiler Runtime (MCR), which is free and only takes a couple of minutes to install. See the README file for details.

The code and executables are provided for non-commercial research purposes only. By downloading and using the code, you are consenting to be bound by all terms of this software release agreement. Contact the authors if you wish to use the code commercially.

* This work is patent pending

At capture time:

- Minimize extraneous motion. Put the camera on a tripod. If appropriate, provide support for your subject (e.g. hand on a table, stable chair).

- Minimize image noise. Use a camera with a good sensor, make sure there is enough light.

- Record in the highest spatial resolution possible and have the subject occupy most of the frame. The more pixels covering the object of interest - the better the signal you would be able to extract.

- If possible, record/store your video uncompressed. Codecs that compress frames independently (e.g. Motion JPEG) are usually preferable over codecs exploiting inter-frame redundancy (e.g. H.264) that, under some settings, can introduce compression-related temporal signals to the video.

When Processing:

- To amplify motion, we recommend our new phase-based pipeline.

- To amplify color, use the linear pipeline (the paper and code in this page).

- Choose the correct time scale that you want to amplify. For example, heart beats tend to occur around once per second for adults, corresponding to 1Hz, and you can amplify content between 0.5Hz and 3Hz to be safe. The narrower the interval, the more focused the amplification is and the less noise gets amplified, but at the risk of missing physical phenomena.

- Don't forget to account for the video frame rate when specifying the temporal passband! See our code for examples.

We would like to thank Guha Balakrishnan, Steve Lewin-Berlin and Neal Wadhwa for their helpful feedback, and the SIGGRAPH reviewers for their comments. We thank Ce Liu and Deqing Sun for helpful discussions on the Eulerian vs. Lagrangian analysis. We also thank Dr. Donna Brezinski, Dr. Karen McAlmon, and the Winchester Hospital staff for helping us collect videos of newborn babies. This work was partially supported by DARPA SCENICC program, NSF CGV-1111415, and Quanta Computer. Michael Rubinstein was partially supported by an NVIDIA Graduate Fellowship.

Many seemingly static scenes contain subtle changes that are invisible to the naked human eye. However, it is possible to pull out these small changes from videos through the use of algorithms we have developed. We give a way to visualize these small changes by amplifying them and we present algorithms to pull out interesting signals from these videos, such as the human pulse, sound from vibrating objects and the motion of hot air.

Tae-Hyun Oh*, Ronnachai Jaroensri*, Changil Kim, Mohamed Elgharib, Frdo Durand, William T. Freeman, Wojciech Matusik

Learning-based Video Motion Magnification

European Conference on Computer Vision (ECCV), 2018

[Paper] [Webpage]

A learning-based approach to motion magnification with reduced artifact and better noise handling. Mohamed A. Elgharib, Mohamed Hefeeda, Frdo Durand, William T. Freeman

Video Magnification in Presence of Large Motions

IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015

[Paper] [Webpage]

A new video technique to motion magnify interesting small motions that are combined with large motions. Frdo Durand, William T. Freeman, Michael Rubinstein

A World of Movement

Scientific American, Volume 312, Number 1, January 2015

[Article in SciAm] [Videos]

An expository article describing several of the motion magnification techniques and applications we have worked on. Neal Wadhwa, Michael Rubinstein, Frdo Durand, William T. Freeman

Riesz Pyramids for Fast Phase-Based Video Magnification

Computational Photography (ICCP), 2014 IEEE International Conference on

[Paper] [Webpage]

Provides the quality of the previous phase-based technique with the real-time speed of the original linear technique. Neal Wadhwa, Michael Rubinstein, Frdo Durand, William T. Freeman

Phase-based Video Motion Processing

ACM Transactions on Graphics, Volume 32, Number 4 (Proc. SIGGRAPH), 2013.

[Paper] [Webpage] [BibTeX]

A new technique to amplify small motions that solves the noise amplification and intensity clipping artifacts of the previous linear method by manipulating the phase in sub-bands of videos. Michael Rubinstein, Neal Wadhwa, Frdo Durand, William T. Freeman

Revealing Invisible Changes In The World

Science Vol. 339 No. 6119, Feb 1 2013

[Article in Science] [Video] [NSF SciVis 2012] [BibTeX]

An expository video showcasing our results and explaining our technique. Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frdo Durand, William T. Freeman

Eulerian Video Magnification for Revealing Subtle Changes in the World

ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH), 2012

[Paper] [Webpage] [BibTeX]

The first Eulerian method to amplify small motions and color variations in videos. Ce Liu, Antonio Torralba, William T. Freeman, Frdo Durand, Edward H. Adelson

Motion Magnification

ACM Transactions on Graphics, Volume 24, Number 3 (Proc. SIGGRAPH), 2005

[Paper] [Webpage]

The original Lagrangian method to amplify small motions by explicitly estimating them and then warping the frame by amplified motion amounts.

Abe Davis, Katherine L. Bouman, Justin G. Chen, Michael Rubinstein, Frdo Durand, William T. Freeman

Visual Vibrometry: Estimating Material Properties from Small Motions in Video

IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015

[Paper] [Webpage]

A method to estimate the material properties of from small motions in videos. Tianfan Xue, Michael Rubinstein, Neal Wadhwa, Anat Levin, Frdo Durand, William T. Freeman

Refraction Wiggles for Measuring Fluid Depth and Velocity from Video

Proc. of European Conference on Computer Vision (ECCV), 2014.

[Paper] [Webpage]

A method to recover the velocity and depth of hot air from video. Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Frdo Durand, William T. Freeman

The Visual Microphone: Passive Recovery of Sound from Video

ACM Transactions on Graphics, Volume 33, Number 4 (Proc. SIGGRAPH), 2014.

[Paper] [Webpage]

A technique to recover sound from videos of objects subtly vibrating in response to sound. Justin G. Chen, Neal Wadhwa, Young-Jin Cha, Frdo Durand, William T. Freeman, Oral Buyukozturk

Structural Modal Identification through High Speed Camera Video: Motion Magnification

Proceedings of the 32nd International Modal Analysis Conference (2014)

[Paper]

A validation that the motion magnified motions are indeed real and a way to compute the mode shapes of a cantilevered beam from video. Guha Balakrishnan, Frdo Durand, John Guttag

Detecting Pulse from Head Motions in Video

Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on

[Paper] [Video]

A method to measure pulse from the Newtonian motion of the head as blood flows into it.

The algorithms that make this work possible are simple, efficient, and robust. Through the processing of local color or phase changes, we can isolate and amplify signals of interest. This is in contrast with earlier work to amplify small motions13 by computing per-pixel motion vectors and then displacing pixel values by magnified motion vectors. That technique yielded good results but it was computationally expensive, and errors in the motion analysis would generate artifacts in the motion magnified output. As we will show, the secret to the simpler processing described in this article lies in the properties of the small motions themselves.

To compare our new work to the previous motion-vector work, we borrow terminology from fluid mechanics. In a Lagrangian perspective, the motion of fluid particles is tracked over time from the reference frame of the particles themselves, similar to observing a river flow from the moving perspective of a boat. This is the approach taken by the earlier work, tracking points in the scene and advecting pixel colors across the frame. In contrast, an Eulerian perspective uses a fixed reference frame and characterizes fluid properties over time at each fixed location, akin to an observer watching the water from a bridge. The new techniques we describe follow this approach by looking at temporal signals at fixed image locations.

The most basic version of our processing looks at intensity variations over time at each pixel and amplifies them. This simple processing reveals both subtle color variations and small motions because, for small sub-pixel motions or large structures, motion is linearly related to intensity change through a first-order Taylor series expansion (Section 2). This approach to motion magnification breaks down when the amplification factor is large and the Taylor approximation is no longer accurate. Thus, for most motion magnification applications we develop a different approach, transforming the image into a complex steerable pyramid, in which position is explicitly represented by the phase of spatially localized sinusoids. We exaggerate the phase variations observed over time, modifying the coefficients of the pyramid representation. Then, the pyramid representation is collapsed to produce the frames of a new video sequence that shows amplified versions of the small motions (Section 3). Both Eulerian approaches lead to faster processing and fewer artifacts than the previous Lagrangian approach. However, the Eulerian approaches only work well for small motions, not arbitrary ones.

3a8082e126