Pose2Pose: Pose Selection and Transfer for 2D Character Animation

Nora S. Willett, Hijung Valentina Shin, Zeyu Jin, Wilmot Li, Adam Finkelstein

...

Abstract: An artist faces two challenges when creating a 2D animated character to mimic a specific human performance. First, the artist must design and draw a collection of artwork depicting portions of the character in a suitable set of poses, for example arm and hand poses that can be selected and combined to express the range of gestures typical for that person. Next, to depict a specific performance, the artist must select and position the appropriate set of artwork at each moment of the animation. This paper presents a system that addresses these challenges by leveraging video of the target human performer. Our system tracks arm and hand poses in an example video of the target. The UI displays clusters of these poses to help artists select representative poses that capture the actor’s style and personality. From this mapping of pose data to character artwork, our system can generate an animation from a new performance video. It relies on a dynamic programming algorithm to optimize for smooth animations that match the poses found in the video. Artists used our system to create four 2D characters and were pleased with the final automatically animated results. We also describe additional applications addressing audio-driven or text-based animations.

PDF


Nora S Willett, Hijung Valentina Shin, Zeyu Jin, Wilmot Li, and Adam Finkelstein. 2020. Pose2Pose: Pose Selection and Transfer for 2D Character Animation. In 25th International Conference on Intelligent User Interfaces (IUI ’20), March 17–20, 2020, Cagliari, Italy. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3377325.3377505

Video

Supplemental Material

For each character, more details are shown about the different pipeline stages. The "Processed data" contains statistics from the frame groups. The "Selected pose clusters" has cluster summary images for all the clusters as well as continuous motion diagrams and artist drawn poses for the selected clusters. "Animated results" contains the final animated characters from all performance videos clips using pose, audio and text to create the animations. It also contains some previz animations from the pose frame assignment algorithm. The "Parameter exploration" page has previz animations created from the parameter tuning study by varying the different constants in the frame assignment algorithm.

Summary of Animations

Contains a single performance video for each character demonstrating pose-driven, audio-driven, text-based and manual animations.


Anthony Scaramucci, Training (White House)

Anthony Scaramucci, Performance (Davos)


Sheryl Sandberg, Training and Performance


Donald Trump, Training (Arab Islamic Summit)

Donald Trump, Performance (Davos)


Hillary Clinton, Training (Nevada)

Hillary Clinton, Performance (Florida)