This blog theme is part of an emerging project that explores what image-oriented disciplines like Art History, Media and Film Studies contribute to the academic and public debate about AI, specifically what it means to conceptualize AI as a cultural phenomenon of image practices rather than a technological tool.[1] Building on this idea, we would like to use this blog post and theme to examine the unique role that animation – as a specific form of moving imagery – plays within such image-focused AI research. To do so, we propose to consider prime definitions of what animation is, in order to understand how it helps us engage with new (image) technologies such as AI. The claim we briefly want to explore is that animated imagery is the epitome of how AI (imagery) works. Animation aesthetics and theory are thus an essential lens through which to examine what AI is and does[2] and, consequently, a glimpse into the black box[3] that AI and machine vision technology appear to be.

Visions of AI in Animation (Theory). Following Alan Cholodenko, the two major features of animation are “the endowing with life and the endowing with motion” (Cholodenko 1991, 15), within which metamorphosis is a defining element (Cholodenko 2014, 101). These definitions show how straightforwardly the logics of animation can be related to the conceptual and structural foundations of AI technologies. Regarding the first aspect, one could argue that building an artificial intelligence comes close to the process of “endowing with life”, which is why the phenomenon of AI and its autonomous (unsupervised) workflows are embedded in the broader discourse about new forms of life, mind, and agency[4] in today’s technological culture (see e.g. Braidotti 2019; Biswasa Mellamphy 2021).

Secondly, when it comes to the visuality of AI as well as AI visuals[5], the aspects of movement and metamorphosis come into play. As Artificial Neural Networks and Machine Learning mechanisms in the field of Computer Vision[6] are trained based on huge sets of images, the transportation and transformation of visual information and data lie at the core of these technologies. Thus, successively ‘watching’/calculating multiple frames and anticipating/rendering what lies in between (or what connects them in different ways) is one of the central mechanisms of these AIs – as it is of animation.

The analogy of animation and AI regarding life and motion becomes even more evident when looking at Tom Gunning’s descriptions of animation as embedded into the broader realm of human-machine-relations[7]: “the very technical process of cinematic motion, the root of animation, conveys the possibility of enlivening – indeed, animating – the mechanical, of blurring the dichotomy between animate and inanimate” (Gunning 2014, 7). Thus, animation’s ability to ‘bring things to life’ is dependent on its ability to ‘set things in motion’ and vice versa – be it a filmstrip or the mind of a computer; the basic animation principle stays the same. And it holds true for the mechanical/technological as well as for the perceptual/sensual level: “Animation depends not simply on the transformation of still images into motion […] but rather on a transformation of perception, a melding of the human sensorium and the machine” (Gunning 2014, 7). Animation thus becomes the mode and moment in which ideas of (mechanical) artificiality and the living, mind and motion, image and intelligence converge.

That both aspects (life and motion) as the basic properties of animation are not meant in a merely metaphorical sense here but can be traced in concrete AI-related image materials becomes evident when looking at a specific example: Google’s Style Transfer mechanism.

Transferring Styles – Picturing the Similar Logics of AI and Animation. The recognition of patterns in (image) data analysis and their (re)implementation into image generation, as a form of prediction, is at the basis of Style Transfer, a method of automated image manipulation that modifies images in the style of famous paintings. The mechanism has existed for a while but has gained accuracy and complexity in recent years due to the development of AI. A posting on the Google AI blog from 2016 presents some of these improvements to Style Transfer by offering a description of the mechanism and gathering some illustrative images as well (see Fig. 1).

Figure 1. Illustration of the Style Transfer process.

Looking at this image, the similarity between the process of ‘machine thinking’ and a sequence of stills from an animated filmstrip strikes the eye – and it becomes even more apparent when looking at the additional short video demo that is embedded in the article and that shows the AI mechanism being applied as a real-time special effect to a video sequence of a dog (see video).

Real-time demo of Style Transfer applied to video.

Two aspects seem to be central in these visualizations of an ‘AI at work’ regarding the analogies to animation outlined above. First, it is remarkable that they use (and thus visualize) AI by letting it transfer something like ‘style’ which means in this case: to mimic handcrafted art as a manifestation of subjectivity and human expression whereby aesthetically endowing this pure calculative process ‘with life’. Second, the process of motion and metamorphosis becomes visible in this example in the presence of ‘the frame’ (be it in the image tiles in Fig. 1 or in the flickering special effects aesthetics of the video).[8]

In the image of Tübingen, upon which four different artistic visual styles have been applied (see Fig. 1), the seriality of images involved in the transformation at the base of Style Transfer is comparable to the seriality of images at the core of animation. Here we see a series of images whereby small changes create the illusion of movement and/or change through nuanced transformation. Whereas what a user often sees is only the starting and endpoint, i.e. the user’s chosen picture or video, which is then automatically rendered in the style of an artistic masterpiece, breaking down the process into single frames is enlightening. Style Transfer algorithmically refines the submitted image so that it sequentially and increasingly meets the stylistic criteria of an artwork or artist (or several), transforming one image into another, as seen above. 

Another famous and foundational definition of animation suggests itself here, Norman McLaren’s observation that: “What happens between each frame is more important than what exists on each frame. Animation is, therefore, the art of manipulating the invisible interstices that lie between frames” (McLaren 1986, as cited in Sifianos 1995). This applies to image-based AI mechanisms as well because they are all about calculating/learning what happens between frames (of their training data sets) in order to be able to manipulate this interstice of (in)visibility to generate their own frames or sequences. In other words, the seriality of images that underlies AI (including both the AI input – the massive data sets used to train AI systems, which consequently form a classification/’understanding’ of a certain term, logic, or pattern – as well as the AI output – based here on a transformation applied to an original image or a video sequence) can be traced back to the staples of animation theory and imagery. The aesthetics of AI discussed here thus promise to offer a glimpse into the artificial “brain” or black box of the AI mechanism through animation.

The Google Brain team exploring Style Transfer compares this to a form of artistic “pastiche”, where one work imitates the style of another (Dumoulin, Shlens and Kudlur 2016). The Style Transfer example, therefore, illustrates both aspects of animation’s definitions mentioned above: It is a useful image technology to reflect upon animation theory in the AI-based analysis and creation of imagery but also a wider trope for thinking about AI which essentially attempts to mimic (and in ways go beyond, as any new work imitating an older one) human creativity and intelligence.

Conclusion. As we become more dependent upon AI and machine vision to decipher the vast quantities of information around us, contemplating ways with which to understand the black boxes of current technologies becomes increasingly vital. We argue that animation, as a visual art form as well as a theoretical concept, plays an important role in these processes. By suggesting the topic of AI and Animation, we hope to expand these discussions and to ‘set some minds in motion’ – through this blog theme, upcoming SAS conferences, and future projects. We look forward to your input and comments.

References

Andersen, Christian Ulrik, et al. 2021. Aesthetics of New AI Interfaces. London: Creative AI Lab/Serpentine Gallery. https://serpentine-uploads.s3.amazonaws.com/uploads/2021/03/002_Aesthetics-of-New-AI-Interfaces.pdf.

Apprich, Clemens/Chun, Wendy Hui Kyong/Cramer, Florian/Steyerl, Hito. 2018. Pattern Discrimination. Minneapolis/London/Lüneburg: University of Minnesota Press/Meson Press.

Benjamin, Ruha. 2019. Race After Technology: Abolitionist Tools for the New Jim Code. Cambridge: Polity Press.

Biswasa Mellamphy, Nandita. 2021. “Re-thinking ‘Human-centric’ AI: An Introduction to Posthumanist Critique. EuropeNow, November 9, 2021. https://www.europenowjournal.org/2021/11/07/re-thinking-human-centric-ai-an-introduction-to-posthumanist-critique/.

BMVA. n.d. “What is Computer Vision?”. BMVA The British Machine Vision Association and Society for Pattern Recognition. Accessed March 5, 2019. http://www.bmva.org/visionoverview (retrievable via: https://web.archive.org/web/20170216180225/http://www.bmva.org/visionoverview).

Braidotti, Rosi. 2019. Posthuman Knowledge. Cambridge: Polity Press.  

Buolamwini, Joy .2017. Gender Shades. Intersectional Phenotypic and Demographic Evaluation of Face Datasets and Gender Classifiers. MIT Master’s Thesis.

Bunz, Mercedes, et al. 2020. Aesthetics of New AI. London: Creative AI Lab/Serpentine Gallery. https://serpentine-uploads.s3.amazonaws.com/uploads/2020/09/001_Aesthetics-of-New-AI.pdf.

Cholodenko, Alan. 1991. The Illusion of Life. Essays on Animation. Sydney: Power Publications.

Cholodenko, Alan. 2014. “‘First principles’ of Animation”. In Animating Film Theory, edited by Karen Beckman, 98–110.  Durham/London: Duke University Press.

Crawford, Kate/Paglen, Trevor. 2019. Excavating AI: The Politics of Training Sets for Machine Learning (September 19, 2019). https://excavating.ai.

Demush, Rostyslav. 2019. “A Brief History of Computer Vision (and Convolutional Neural Networks)”. Hackernoon. February 26, 2019. https://hackernoon.com/a-brief-history-of-computer-vision-and-convolutional-neural-networks-8fe8aacc79f3.

Dumoulin, Vincent, Jonathan Shlens, and Manjunath Kudlur. 2016. “Supercharging Style Transfer.” Google AI Blog. October 26, 2016. https://ai.googleblog.com/2016/10/supercharging-style-transfer.html

Gunning, Tom. 2014. “ANIMATION AND ALIENATION: Bergson’s Critique of the Cinématographe and the Paradox of Mechanical Motion”. The Moving Image: The Journal of the Association of Moving Image Archivists, Vol. 14, No. 1 (Spring 2014): 1-9.

IBM. n.d. “What is Computer Vision?” Accessed April 16, 2022. https://www.ibm.com/topics/computer-vision

IBM Cloud Education. 2020. “Artificial Intelligence (AI)”. June 3, 2020 https://www.ibm.com/cloud/learn/what-is-artificial-intelligence?mhsrc=ibmsearch_a&mhq=what%20is%20AI%3F.

Manovich, Lev. 2018. AI Aesthetics. Moscow: Strelka Press.

Salemy, Mohammad. 2016. For Machine Use Only: Contemplations on Algorithmic Epistemology. Gwanju: &&& Publishing / The New Centre for Research and Practice.

Sifianos, Georges. 1995. „The definition of animation. With a letter from Norman McLaren.” Animation Journal 3, no. 2 (Spring 1995): 62–66.

Virilio, Paul. 1994. The Vision Machine. Bloomington: Indiana University Press.

Zylinska, Joanna. 2020. AI Art: Machine Visions and Warped Dreams. London: Open Humanities Press.

Jun.-Prof. Dr. Julia Eckel is Junior Professor for Film Studies at the Department of Media Studies at Paderborn University, Germany. Currently, she is working on a book on the nexus of animation, documentation, and demonstration. Additional research interests are: audiovisual anthropomorphism, synthespians, animation and agency, animation and AI, screencasting, and selfies. Recent publications include Das Audioviduum. Eine Theoriegeschichte des Menschenmotivs in audiovisuellen Medien (2021) and the three co-edited volumes Ästhetik des Gemachten. Interdisziplinäre Beiträge zur Animations- und Comicforschung (2018; edited with H.-J. Backe, E. Feyersinger, V. Sina, and  J.-N. Thon), Exploring the Selfie – Historical, Theoretical, and Analytical Approaches to Digital Self-Photography (2018; edited with J. Ruchatz and S. Wirth), and Im Wandel … Metamorphosen der Animation (2017; edited with E. Feyersinger, M. Uhrig). For more info see: www.juliaeckel.de.

Dr. Nea Ehrlich is a Senior Lecturer in the Department of Arts at Ben-Gurion University of the Negev in Israel. She completed her Ph.D. in the Department of Art History at the University of Edinburgh and was a Polonsky postdoctoral fellow at The Van Leer Jerusalem Institute. She is the author of articles on realism, serious games, animation and documentary. Her research appears in animation: an Interdisciplinary Journal, Studies in Documentary Film and Visual Resources, she is co-editor of Drawn from Life, the 2018 anthology about animated documentaries published by Edinburgh University Press. Her book, Animating Truth, on animated documentary and the virtualization of culture in the 21st century was published in 2021 by Edinburgh University Press. Her work lies at the intersection of Art History, Film Studies, Animation, Digital Media Theory, Gaming, and Epistemology. She is currently working on a project on art and robotics, focusing on AI and machine vision.

[1] First attempts in the direction of image-focused AI research have been made recently regarding data biases and the discriminating and marginalizing effects inherent in AI datasets (e.g. Buolamwini 2017, Benjamin 2019, Crawford/Paglen 2019, Apprich et al. 2018), contributing cultural and societal aspects of images and image data to AI research. Simultaneously, artists and AI-based artworks explore these dimensions on a practical and aesthetical level, producing images that are meant for human vision rather than the machine’s ‘eyes’ only (e.g. works by Tom White, Anna Ridler, or Mario Klingman; see also Salemy 2016, Manovich 2018, Bunz 2020, Zylinska 2020, Andersen 2021). What is missing yet is an additional image and film theoretical perspective which the authors aim to develop in a respective project.

[2] See Julia Eckel’s additional contribution to this blog theme that focuses on “Intelligence In Between – Documenting AI in Animation”.

[3] See the contribution by Deborah Levitt on “Animation vs. Black Boxes” as part of this blog theme.

[4] See the contribution about digital and artificial humans by Gina Moore titled “3D Animation, Automation and Cliché” on the blog.

[5] Meaning: contexts in which AI uses images (e.g. as training data) is perceivable as something visible and/or producing visuals itself (e.g. in AI artworks).

[6] Machine vision is a phenomenon defined by Paul Virilio (The Vision Machine, 1994) which has developed into the entire field of computer vision (for a historical overview see Demush 2019). The British Machine Vision Association and Society for Pattern Recognition (BMVA) defines computer vision as concerned with “the automatic extraction, analysis and understanding of useful information from a single image or a sequence of images. It involves the development of a theoretical and algorithmic basis to achieve automatic visual understanding” and is used in varied fields such as forensics, biometrics, robotics, medical image analysis, face recognition, augmented reality, and many more (BMVA n.d.). Machine Vision occurs when machines, independently of human operators, ‘see’, record, and share data with other machines. Machine Learning is a subset of AI, which uses algorithms to automatically recognize patterns from data. Deep learning, a higher mode of machine learning, uses convolutional and recurrent neural networks to process information in a way that reflects the behavior of the human brain (mimicking the way that biological neurons signal to one another), in order to learn, recognize, classify, and predict complex patterns, free of human input (see IBM n.d. and IBM Cloud Education 2020).

[7] See Joel McKim’s contribution to this blog theme as well, which focuses on another level of this intersection of human and machine titled “Animation Without Animators: From Motion Capture to MetaHumans”.

[8] For an additional example, see again Julia Eckel’s contribution to this blog theme.