Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.

First Order Motion Model for Image Animation / Siarohin, Aliaksandr; Lathuiliere, Stephane; Tulyakov, Sergey; Ricci, Elisa; Sebe, Nicu. - (2019), pp. 7135-7145. ((Intervento presentato al convegno Neural Information Processing Systems (NeurIPS'19) tenutosi a Vancouver nel December 2019.

First Order Motion Model for Image Animation

Siarohin, Aliaksandr;Lathuiliere, Stephane;Tulyakov, Sergey;Ricci, Elisa;Sebe, Nicu
2019

Abstract

Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.
Advances in Neural Information Processing Systems 32
New York
Curran Associates, Inc.
Siarohin, Aliaksandr; Lathuiliere, Stephane; Tulyakov, Sergey; Ricci, Elisa; Sebe, Nicu
First Order Motion Model for Image Animation / Siarohin, Aliaksandr; Lathuiliere, Stephane; Tulyakov, Sergey; Ricci, Elisa; Sebe, Nicu. - (2019), pp. 7135-7145. ((Intervento presentato al convegno Neural Information Processing Systems (NeurIPS'19) tenutosi a Vancouver nel December 2019.
File in questo prodotto:
File Dimensione Formato  
8935-first-order-motion-model-for-image-animation.pdf

accesso aperto

Tipologia: Versione editoriale (Publisher’s layout)
Licenza: Tutti i diritti riservati (All rights reserved)
Dimensione 9.21 MB
Formato Adobe PDF
9.21 MB Adobe PDF Visualizza/Apri

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione

Utilizza questo identificativo per citare o creare un link a questo documento: http://hdl.handle.net/11572/250831
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 140
  • ???jsp.display-item.citation.isi??? 84
social impact