Leonardo Abstracts Service | Leonardo/ISASTwith Arizona State University

Leonardo Abstracts Service

  • Fox-Gieg, Nicholas "Lightning Artist Toolkit: A Hand-Drawn Volumetric Animation Pipeline." PhD , York University, 2024
    Keywords/Fields of Study : artificial intelligence, animation, augmented reality, drawing, film/cinema, machine learning, mixed reality, virtual reality

    Abstract: We propose a set of methods for freely integrating live-action volumetric video with hand-drawn volumetric animation. The Kinect, the first consumer depth camera, arrived in 2010; in 2016, the HTC Vive headset introduced the first mass-market 6DoF controllers. Combined, these two advances unlocked a new approach to creating frame-by-frame animation with 6DoF drawing tools, which our research develops as the Lightning Artist Toolkit (Latk)—a complete pipeline for hand-drawn volumetric animation, as far as we know the only open-source example of its kind. Our goal with this project is to make creation in 3D as expressive and intuitive as creation in 2D, retaining the human gesture from its origins in hand-drawn animation on paper. Importing and manipulating scanned photographic images alongside drawings in this way has been a core feature of 2D image editing and animation tools for almost fifty years. But initially, applying these raster editing capabilities to real-world animation production was impractical—so the earliest hand-drawn computer-animated short films used 2D vector strokes. Today, operating naïvely on 3D voxels similarly requires too much compute power to scale up for even a few minutes of high-resolution footage, while 3D vector graphics representations offer a promising workaround. At our project’s core is a collection of applied machine learning systems that transform live-action volumetric video into a sequence of volumetric brushstrokes. Integrated into a conventional animation workflow, this output should be suitable for the practical production of hand-drawn 3D animated short films in an XR drawing system.We propose a set of methods for freely integrating live-action volumetric video with hand-drawn volumetric animation, which our research develops as the Lightning Artist Toolkit (Latk)—a complete pipeline for hand-drawn volumetric animation, as far as we know the only open-source example of its kind. Our goal with this project is to make creation in 3D as expressive and intuitive as creation in 2D, retaining the human gesture from its origins in hand-drawn animation on paper. This effort is less a computer vision challenge with an objective goal, as with for example point cloud segmentation, than it is an attempt to approximate human vision—a drawing process that records only information from a scene that was subjectively important to an individual artist. In addition to supporting animation efforts in the near term, we believe the public TiltSet dataset assembled for this project will remain usable in new and unexpected ways.

    Department: Computational Arts , York University
    Advisor(s): Graham Wakefield