ArtEmis & Painting Style Transfer — Portfolio Project by Muhammad Daffa Ashdaqfillah

An experimental lab exploring painting-to-video animation, featuring an iOS app for emotion and style classification, alongside web prototypes for real-time portrait and pose animation.

ArtEmis & Painting Style Transfer — Portfolio Project by Muhammad Daffa Ashdaqfillah

Published on May 21, 2026

Project Details

Image

Description

This project is an experimental laboratory focused on bridging classical art with real-time motion and artificial intelligence. The goal is to explore "Painting-to-Video Animation"—the concept of animating static paintings using a driving video or webcam input, allowing subjects in the artwork to mimic the user's movements while preserving the original artistic style, composition, and brushstrokes.

Background

The core idea is inspired by the desire to make static artwork interactive. Instead of being frozen in time, painted characters and environments come alive on screen. By combining scene-aware video-to-video animation and style transfer techniques, the project investigates how AI can seamlessly merge art and motion.

Key Features

  • ArtEmis iOS App: A native iOS application that uses CoreML and Vision to classify the emotion and art style of paintings in real-time. It can detect a wide range of emotions and classify artwork into eras such as Impressionism, Baroque, and Abstract Expressionism.
  • Real-Time Painting Animation (Web MVP): A browser-based prototype that captures a live subject via webcam, estimates pose landmarks using MediaPipe, and renders a mirrored painting-style subject on canvas in real-time.
  • LivePortrait Web PoC: A proof-of-concept integrating the open-source LivePortrait model to animate specific portrait paintings (like the Mona Lisa or Girl with a Pearl Earring) based on facial movements and head poses from a driving video.

Implementation

The project relies on multiple decoupled prototypes to explore different facets of the problem:

  • CoreML & Vision: The iOS application leverages Apple's on-device machine learning frameworks (ArtEmisClassifierV3.mlpackage) to perform emotion and style inference directly on the device.
  • MediaPipe Pose Tracking: For the real-time web MVP, MediaPipe is used to extract pose landmarks. Temporal landmark smoothing is applied to reduce jitter, maintaining responsiveness while rendering painterly brush effects on an HTML5 canvas.
  • LivePortrait Integration: The web PoC utilizes the LivePortrait model's inference.py backend to transfer head pose and facial motion to a source painting. A local Python server orchestrates video capture, format conversion using FFmpeg, and inference processing.

Architecture

mermaidflowchart TD
  A[User Webcam/Video] --> B{Application Mode}
  B -->|Web MVP| C[MediaPipe Pose Tracking]
  B -->|LivePortrait PoC| D[LivePortrait Inference Backend]
  B -->|iOS App| E[Vision & CoreML Classifier]
  C --> F[Temporal Smoothing & Canvas Rendering]
  D --> G[Facial Motion Transfer]
  E --> H[Emotion & Style Analysis]
  F --> I[Animated Artwork Output]
  G --> I

Result

The project successfully demonstrates the feasibility of real-time pose retargeting and portrait animation in both browser and native environments. The decoupled prototypes serve as a foundation for a future unified system where optical-flow stabilization and advanced frame blending can be introduced for cinematic, temporally consistent animation.

Slides

Image
Image